
In Self-Compacting & Bi-Temporal Agent Memory and Zero-Trust Signal Gateways for Local AI Agents, we built an autonomous agent runtime that verifies live infrastructure state (~/scripts/antigravity_verify.py) and communicates exclusively through a hardened Signal “Note to Self” gateway (~/scripts/signal_bot.py).
Until now, antigravity_verify.py was strictly read-only. It told you what was broken—a stalled NUT UPS daemon, an unmounted NFS share, or a dropped Tailscale policy route—but left you to SSH in and fix it manually.
Turning a diagnostic probe into an automated remediation engine is a classic engineering trap: unconstrained self-healing is just automated outage amplification. If a disk is full or hardware is failing, a naive auto-restarter will thrash systemd units in an infinite loop.
Here is how we upgraded our read-only probe into a Supervised Self-Healing Sentinel with strict guardrails, circuit breakers, and one-tap human approval over Signal.
Architecture Overview

The pipeline separates detection, gating, approval, and execution into four decoupled stages:
- Functional Probes (
systemd --usertimer every 10 minutes): Checks not just TCP ports, but actual daemon behavior. - Guardrail Gate (
Meta/states/sentinel-proposals.json): Enforces cooldowns, TTLs, and a 24-hour circuit breaker before creating a short 4-character proposal ID. - Lock-Free Signal Outbox (
Meta/states/sentinel-outbox.jsonl): Avoidssignal-clilock contention by handing alerts to the running JSON-RPC daemon. - Idempotent Execution (
--approve <id>): Re-verifies the failure before running the runbook, executes the bounded command, and runs a post-check probe.
1. Functional Probes Over Shallow TCP Checks
An open TCP port does not mean a service is healthy. A NUT UPS server can listen on port 3493 while its USB driver is wedged, and an NFS mountpoint can exist in /etc/mtab while returning stale handle hangs.
We extended antigravity_verify.py with three functional probes alongside standard TCP socket checks:
def check_nut_telemetry() -> dict:
"""Verify NUT daemon actually serves UPS telemetry, not just an open port."""
proc = subprocess.run(
["upsc", "volt_ups@nas-local", "ups.status"],
capture_output=True, text=True, timeout=3.0,
)
val = proc.stdout.strip()
if proc.returncode == 0 and val:
return {"ok": True, "detail": f"status={val}"}
return {"ok": False, "detail": (proc.stderr or "").strip() or f"rc={proc.returncode}"}
def check_nfs_mount() -> dict:
"""Verify NFS mount is both mounted and responsive (no stale handle)."""
mp = subprocess.run(["mountpoint", "-q", "/mnt/nas_storage"], timeout=2.0)
if mp.returncode != 0:
return {"ok": False, "detail": "not mounted"}
st = subprocess.run(["stat", "-t", "/mnt/nas_storage"], capture_output=True, timeout=2.5)
return {"ok": st.returncode == 0, "detail": "mounted & responsive" if st.returncode == 0 else "stale mount"}
Similarly, the Tailscale probe inspects tailscale status --json (BackendState == "Running") and verifies that policy routing table 52 contains active routes (ip route show table 52).
2. Declarative Runbooks & Non-Negotiable Guardrails
Every remediable failure maps to an explicit entry in RUNBOOKS, tagged with its blast radius (low or medium), an idempotent command, and a post-verification function:
RUNBOOKS = {
"nut_ups_nas": {
"title": "Restart NUT UPS Daemon",
"blast_radius": "low",
"command": [
"ssh", "-o", "ConnectTimeout=5", "-o", "BatchMode=yes",
"user@nas-local",
"sudo -n systemctl restart nut-driver-enumerator.service nut-server.service",
],
"verify_fn": lambda: check_nut_telemetry()["ok"],
},
"nfs_local_mount": {
"title": "Remount Primary NFS Share Locally",
"blast_radius": "low",
"command": ["bash", "~/scripts/mount_nas_nfs.sh", "mount"],
"verify_fn": lambda: check_nfs_mount()["ok"],
},
}
Four Guardrails That Prevent Runaway Loops
- 60-Minute Proposal TTL: Approvals are time-bounded. If you tap
approve a1b2three hours later when network topology has changed, the proposal is already markedexpired. - 15-Minute Cooldown: Rejected or recently triggered runbooks enter a cooldown window so a flaky sensor cannot spam your phone every 10 minutes.
- 24-Hour Circuit Breaker (
MAX_ATTEMPTS_PER_DAY = 3): If a runbook runs 3 times in 24 hours and the service keeps failing, the Sentinel halts proposals for that runbook and escalates with a⛔ Circuit Breaker Activealert. - Self-Protected Daemon Boundary: A child subprocess spawned by
signal-bot.servicemust never restartsignal-bot.serviceitself—doing so severs the parent process mid-reply. Self-protected units are flagged for external/manual intervention only.
3. Idempotent Pre-Check & Post-Verification
When an operator replies approve <id> over Signal, the Sentinel does not blindly execute the shell command. It runs the probe again first:
# Idempotent pre-check: did the issue already self-resolve?
if rb["verify_fn"]():
p["status"] = "executed_ok"
return f"✅ [{p['id']}] {p['title']}: Already healthy (self-resolved). No action needed."
If the failure is still present, the Sentinel executes the runbook, waits 1.5 seconds for daemon stabilization, runs rb["verify_fn"]() again, and logs the final outcome to Meta/agent-log.md.
4. Solving signal-cli Lock Contention with an Atomic Outbox
Because signal_bot.py holds a persistent signal-cli jsonRpc child process open, running signal-cli send from a separate systemd timer fails on the SQLite/config lock.
Instead of spawning a second client, antigravity-sentinel.timer writes JSON lines to Meta/states/sentinel-outbox.jsonl. Inside signal_bot.py, a lightweight background thread polls the outbox every 5 seconds, atomically renames it to .processing via os.replace(), and dispatches each alert through the existing send_signal_reply(proc, msg) pipe:
def sentinel_outbox_watcher():
outbox = Path("~/Vault/Meta/states/sentinel-outbox.jsonl").expanduser()
while proc.poll() is None:
if outbox.exists() and outbox.stat().st_size > 0:
processing = outbox.with_suffix(".processing")
outbox.replace(processing)
for raw in processing.read_text(encoding="utf-8").splitlines():
if raw.strip():
msg_text = json.loads(raw).get("message", "").strip()
if msg_text:
send_signal_reply(proc, msg_text)
processing.unlink(missing_ok=True)
time.sleep(5)
From your phone, the entire workflow takes two words:
heal— Triggers an on-demand Sentinel scan and lists any pending proposal IDs.approve <id>/reject <id>— Executes (with pre/post verification) or rejects a proposal in seconds.
Key Takeaways
- Read-only probes are the foundation, not the ceiling—reusing the exact same probe functions as pre-checks and post-checks guarantees runbook idempotency.
- Short-lived tokens + circuit breakers make remote mobile approvals safe enough for real infrastructure.
- Decoupled outbox files keep single-writer daemons like
signal-clilock-free without adding a message broker.