Heartbeat Watches: turn a silent job into a durable Action

Most scheduled jobs tell you when they fail loudly. Fewer tell you when they stop running altogether. A backup process can exit before it sends its status, a CI runner can disappear between jobs, and an agent can hang without producing a useful log line. That is the gap a heartbeat Watch closes: the monitored system reports a small liveness signal, and silence becomes a durable Action for a human to inspect.

The useful boundary: expectation, not synthetic uptime

A Watch is not a browser check, a log index, or a second alert inbox. It stores one expectation for one Source: an interval or five-field cron schedule, an IANA timezone, a grace period, and optionally a maximum runtime. When the expectation is violated, Actionbox uses the existing Action lifecycle so the incident appears in the same web and mobile inbox, has the same audit trail, and can be system-resolved when the job recovers.

This is intentionally boring infrastructure. One outage episode produces at most one open Action; repeated evaluator passes do not page repeatedly.

Create one for a backup job

Create a Watch with a Source credential. The raw capability URL is returned only on this response, so copy it directly into a secret manager or the job's protected environment.

actionbox watch create "Nightly backup" \
  --source-id src_123 \
  --interval-seconds 3600 \
  --grace 15m \
  --max-runtime 1h \
  --priority high \
  --runbook-url https://runbooks.example.com/backups/nightly

The output contains a URL like https://api.actionbox.cloud/hb/hb_<capability>. Do not put it in shell history, application logs, analytics, screenshots, or an Action payload. List output and Watch detail responses contain only the safe token prefix.

Wrap the command, preserve its exit code

The CLI wrapper sends start, runs the child with inherited stdin/stdout/stderr, then sends success or fail. Heartbeat delivery is bounded and best-effort; an unavailable monitor never hides the command's own exit status.

export ACTIONBOX_WATCH_URL='https://api.actionbox.cloud/hb/hb_<capability>'
actionbox run --watch-url "$ACTIONBOX_WATCH_URL" -- /opt/jobs/backup.sh

For a scheduler that cannot install the CLI, the HTTP contract is just as small:

curl -fsS -X POST "$ACTIONBOX_WATCH_URL/start"
if /opt/jobs/backup.sh; then
  curl -fsS -X POST "$ACTIONBOX_WATCH_URL/success"
else
  curl -fsS -X POST "$ACTIONBOX_WATCH_URL/fail"
  exit 1
fi

The bare URL is a ping and is equivalent to a successful liveness signal. Paused Watches timestamp signals but do not evaluate or create Actions, which makes planned maintenance safe.

Cron schedules and daylight saving time

For a job that runs at a wall-clock time, use a five-field cron expression and an IANA timezone:

actionbox watch create "Toronto settlement" \
  --source-id src_123 \
  --schedule cron \
  --cron "0 2 * * *" \
  --timezone America/Toronto \
  --grace 20m

Occurrences are calculated in the configured timezone and persisted as UTC. During spring-forward, a nonexistent local time is advanced to the next valid cron occurrence; during fall-back, the timezone-aware occurrence is retained without creating duplicate outage Actions. The evaluator advances from the schedule rather than from the time a worker happens to wake up.

What the human sees

When silence, an explicit fail, or a maximum-runtime deadline is reached, Actionbox creates an ordinary high-priority (or configured priority) Action with a stable watch:<id>:incident dedupe key. Its context includes the expected time, last success, condition, and optional runbook. A later ping or success records a recovered Watch event and system-resolves the open Action.

If a human already resolved that Action, recovery preserves the terminal event; it never reopens or mutates history. A new Action can appear only after the Watch is healthy again and a later outage begins.

Use Watches where silence is the failure mode

• Backups: detect a backup that never started, not only a process that exits non-zero.
• CI runners: catch a runner that disappears before it can report a failed job.
• Data pipelines: pair a cron schedule with a runbook for the first missed partition.
• Agent tasks: use the same contract later for long-running agents without inventing a new model.
• Maintenance windows: pause and resume from the dashboard while preserving signal history.

Watches complement an independent monitor for Actionbox itself. A stopped Watch evaluator cannot create its own missing-heartbeat Action, so keep the external production health monitor enabled.

Try it safely

Start with one internal, non-critical job, keep your existing external monitor in parallel, and compare expected, down, and recovery times for at least one schedule cycle. Rotate the capability URL after any possible disclosure and archive the Watch when the integration retires.

Watch API reference · CLI guide · Cron job approval workflow · Create a free Source