Heartbeats, issues, incidents: what agent health tells you

The Silent Failure Mode

When an AI agent stops responding, most teams assume the process crashed. They check logs, restart the container, and move on. But in production autonomous systems, silence is the most dangerous signal of all. An agent that stops reporting is not necessarily dead. It may still be running, still making tool calls, still modifying state — just no longer under observation.

This is the rogue agent problem. It is not about agents that produce bad outputs. It is about agents that slip outside the boundary of your control plane entirely. No heartbeat means no visibility. No visibility means no governance. And in agentic systems where a single tool call can trigger an API request, a database write, or an external transaction, ungoverned execution is an operational risk you cannot afford.

GovernorAI treats rogue agent detection as a first-class infrastructure concern, not an afterthought bolted onto application logging. This post walks through what the control plane observes, how a missed heartbeat becomes an issue and then an incident, and how that connects to the kill switch — which is an explicit action, not an automatic one.

What the Control Plane Observes

Heartbeat monitoring

Every registered agent sends periodic heartbeats to the control plane at POST /api/v1/agents/{id}/heartbeat. The heartbeat confirms the agent is alive, connected, and still operating within its declared session. The control plane records the timestamp of the last one; the default interval is 10 seconds.

Heartbeat monitoring catches the most common failure mode: an agent process that hangs, loses network connectivity, or enters a loop that stops it reporting back. The design decision that matters is that the control plane does not wait for an agent to self-report failure. It treats the absence of evidence as the signal.

Health status

Each agent carries a health status — healthy, degraded, unhealthy, offline or unknown — alongside the operational signals the control plane keeps per agent: uptime percentage, heartbeat interval, error rate, request count and resource usage.

Issues

Observations that need attention become issues attached to the agent, each with a type and a severity. A run of missed heartbeats, for example, is recorded as an issue of type missed_heartbeat at critical severity with the detection timestamp and a human-readable message.

From Issue to Incident

Issues are observations. Incidents are the thing an operator owns. An incident carries the agent it belongs to, the SLO it breached, when it started, when it was detected, a severity and a description.

Incidents have a workflow rather than a timer. Through the API an incident can be acknowledged, assigned to a responder, escalated, and closed out with a postmortem. That is the path a missed-heartbeat issue takes to reach a human: it is an operational queue your SOC works, not a countdown.

The Kill Switch Is an Explicit Action

This is the part worth being precise about, because it is where most descriptions of agent governance overreach.

GovernorAI does not automatically kill an agent because its heartbeat stopped. Silence is ambiguous — a network partition, a slow garbage collection pause and a genuinely rogue process all look identical from the control plane — and killing a healthy agent because a switch flapped is its own production incident.

What exists instead are two explicit controls:

• Suspend — POST /api/v1/agents/{id}/suspend moves an agent to suspended status.
• Kill switch — POST /api/v1/killswitch activates a kill at a chosen scope, and the response reports the measured propagation time.

Both are deliberate actions. They are exposed as plain REST endpoints precisely so that your SIEM, your SOAR playbook, or an on-call engineer following a runbook is the thing that decides — on your thresholds, with your risk tolerance, and with the decision recorded against an authenticated identity rather than a client-supplied one.

That is a narrower claim than “the platform kills rogue agents for you,” and it is the one the product actually supports.

Querying Agent Status

The GovernorAI API exposes agent health status for integration with your existing monitoring stack. SOC teams can poll agent status or subscribe to webhook notifications for severity changes.

# Check health status of a specific agent
curl -s https://api.governorai.ai/api/v1/agent-health/agent-4f9a \
  -H "Authorization: Bearer $GOV_API_KEY" | jq .

# Response
# {
#   "agent_id": "agent-4f9a",
#   "agent_name": "support-refunds",
#   "status": "degraded",
#   "last_heartbeat": "2025-02-14T09:41:12Z",
#   "heartbeat_interval_ms": 5000,
#   "uptime_percent": 94.2,
#   "error_rate": 0.11,
#   "issues": [ ... ]
# }

# List all agents currently flagged as rogue
curl -s https://api.governorai.ai/api/v1/agent-health \
  -H "Authorization: Bearer $GOV_API_KEY" \
  | jq '[.agents[] | select(.status != "healthy")]
        | map({agent_id, status, last_heartbeat})'

Integration with the Kill Switch

The kill switch is the control you reach for when an agent has to stop now. It is reached the same way whoever is calling it: an operator at a console, a SOAR playbook, or a SIEM rule all hit POST /api/v1/killswitch with a scope and a target.

Because the path is the same, the behaviour is the same. There is one mental model for kill switch behaviour, one blast radius, one recovery process and one audit trail, regardless of what prompted the call. The response reports the measured propagation time for that activation, so the number in your incident timeline is observed rather than assumed.

The recorded actor is the authenticated one. The server derives it from the JWT subject and ignores any identity supplied in the request body, so a caller cannot attribute a kill switch to someone else.

Designing for False Positives

Every detection system produces false positives, and the honest answer to “what happens when it does” is the reason this design keeps a human in the path.

A missed heartbeat is ambiguous. A network partition, a long garbage-collection pause and a genuinely rogue process are indistinguishable from the control plane’s point of view — all three look like silence. A system that killed agents on that signal alone would trade an uncertain risk for a certain outage.

So the escalation stops at an incident. Issues and incidents are observational: they surface in the queue, carry a severity, and can be acknowledged, assigned and escalated. Stopping the agent — suspending it, or activating the kill switch — stays an explicit action, taken by someone with the context to judge whether silence means danger or means the network blipped.

If your agents handle high-value operations where even a brief period of ungoverned execution is unacceptable, the right answer is a tighter runbook and an automated playbook that calls the API on your conditions, not a threshold inside the platform guessing at your risk tolerance.

What Comes Next

Rogue agent detection is one layer of a broader governance stack. It works alongside policy enforcement, audit logging, and the kill switch to provide defense in depth. No single mechanism is sufficient on its own. But together, they ensure that autonomous agents remain observable, controllable, and accountable — even when they fail in unexpected ways.