Job Monitor Operations
Operate Watches-backed job monitors, understand incidents, and protect heartbeat capabilities.
Watch availability
Watch support may not be available in every hosted environment yet. If the API
returns WATCHES_DISABLED or WATCH_INGRESS_DISABLED, there is no client
setting to change; contact ActionBox support before relying on Watch
evaluation or heartbeat signals.
Use a Watch when a worker, cron job, or long-running process must prove that it is alive. ActionBox evaluates the expected schedule and grace period; a missed signal, failed run, or exceeded runtime creates one ordinary Action for human review.
Lifecycle
| Status | Meaning | Next step |
|---|---|---|
new | Created, but not armed by a signal yet | Send ping, start, or success, or resume it. |
healthy | A signal has arrived and the Watch is not currently down | Keep the job emitting its configured signal; inspect timestamps for proof of a recent run. |
down | A failure, missed schedule, or runtime deadline was observed | Inspect the linked incident Action and recover the job. |
paused | Evaluation is suspended | Resume only when the job should be monitored again. |
The API also returns derived running and late values. Treat status and
the timestamps in the Watch response as the source of truth, not a dashboard
badge or push notification.
Create a Watch
Watch management uses a user session and requires an owner or admin role:
curl -fsS -X POST https://api.actionbox.cloud/v1/watches \
-H "Authorization: Bearer $ACTIONBOX_USER_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"source_id": "src_<source_id>",
"name": "Nightly backup",
"schedule_type": "interval",
"interval_seconds": 3600,
"grace_seconds": 300,
"timezone": "UTC",
"max_runtime_seconds": 7200,
"priority": "high",
"signal_method": "post"
}'An interval is between 60 seconds and 30 days. A cron schedule uses exactly
five valid cron fields and an IANA timezone. Grace and maximum runtime are
bounded to 30 days. The response returns a raw heartbeat_url once; store it
in the job's secret manager before closing the response.
Send signals
The returned capability URL supports these paths:
| Signal | Use it when | Effect |
|---|---|---|
/hb/<token> | A periodic liveness pulse arrived | Records heartbeat and success; the Watch is healthy. |
/hb/<token>/start | A scheduled run began | Records the run and starts the optional runtime deadline. |
/hb/<token>/success | A run completed successfully | Records success, clears the active run, and can resolve an open Watch incident. |
/hb/<token>/fail | A run failed | Marks the Watch down and creates or updates its incident Action. |
For a POST-only Watch, use the CLI wrapper or Python SDK. Both read the capability from your environment rather than putting it in a shell argument:
actionbox run -- /opt/jobs/nightly-backup.shFor a long-running process, the Python SDK can call send_heartbeat(os.environ["ACTIONBOX_WATCH_URL"], "ping"). Keep transport errors separate from the job's exit status and do not print the URL.
Successful heartbeat requests return 204 with no credential-bearing response.
GET is accepted only when signal_method is any; a POST-only Watch returns
405 WATCH_POST_REQUIRED. Do not add arbitrary metadata to a heartbeat. The
schedule, timestamps, and signal type are the durable health record.
Incidents and recovery
The evaluator considers a Watch late after the expected occurrence plus
grace_seconds, or when max_runtime_seconds is exceeded for an active run.
It creates one deduplicated ordinary Action and links it through
incident_action_id. Repeated evaluator passes do not create a second Action.
When a healthy signal arrives, ActionBox system-resolves the open incident. A
human-resolved incident is not reopened while the Watch remains in the same
down episode, including when another failure signal arrives. A later
ping or success records recovery and returns the Watch to healthy; the
recovered Watch clears incident_action_id, and the next failure starts a new
incident Action. This prevents one continuing
outage from producing an alert storm. If a failure is expected, pause the Watch
before maintenance; paused signals are recorded but do not evaluate the
schedule or create Actions.
Where configured for a Business or Enterprise workspace, a monitor can add one Retry job control to future incidents. A reviewer requests the retry; your HTTPS receiver verifies the signed callback with that Source's webhook signing secret, deduplicates its request ID, and decides how to run the job. A Source created in the monitor setup does not reveal its signing secret there; rotate and save it from the Source screen before deploying a receiver. The control is not proof that the retry succeeded. Report a genuine success signal after validating the job's result. Changing the profile does not rewrite an incident that is already open.
Management endpoints
| Operation | Endpoint | Credential |
|---|---|---|
| List or inspect | GET /v1/watches, GET /v1/watches/{id} | User session |
| Create or edit | POST /v1/watches, PATCH /v1/watches/{id} | Owner/admin user session |
| Read event history | GET /v1/watches/{id}/events | User session |
| Pause/resume | POST /v1/watches/{id}/pause or /resume | Owner/admin user session |
| Rotate capability | POST /v1/watches/{id}/token/rotate | Owner/admin user session |
| Archive | DELETE /v1/watches/{id} | Owner/admin user session |
A Source key can manage only its own active Watches through
/v1/source/watches and the corresponding pause, resume, rotate, and archive
routes. Source-scoped creation keeps the source_id equal to the authenticated
Source. A Watch capability URL is not a user session and cannot list or edit a
Watch.
Capability safety
- Keep the heartbeat URL in a secret manager; never put it in logs, source control, issue attachments, browser code, or a referrer.
- Prefer
signal_method: "post"so link scanners and previewers cannot record a GET signal. - Rotate the capability immediately after a suspected disclosure and update the job. The old URL stops working after rotation.
- Archive a Watch that is no longer expected to run. Revoking its Source also pauses the Watch and invalidates its heartbeat capability.
- Use the Watch event history for diagnosis; do not infer health from a missing notification.
Common issues
| Symptom | Likely cause | Fix |
|---|---|---|
Watch stays new | No first signal or resume yet | Send ping, start, or success, or resume the Watch. |
404 WATCH_NOT_FOUND | URL was rotated/archived, Source revoked, or token is wrong | Fetch the Watch metadata with a user session and update the job's secret. |
405 WATCH_POST_REQUIRED | GET used for a POST-only Watch | Send the signal with POST and no token in query parameters. |
Watch is down after a healthy job run | Wrong timezone/cron, insufficient grace, missing signal, or runtime exceeded | Compare next_expected_at, timezone, grace_seconds, run_deadline_at, and the job's emitted signal. |
| Multiple alerts for one outage | Separate Watches or a new incident after recovery | Check watch_id and incident_action_id; one Watch deduplicates its open incident. |
| Heartbeat returns success but no inbox alert | Signals make a Watch healthy; alerts are for missed/failed health | Inspect Watch state and event history rather than expecting an Action for every pulse. |
Escalate with redacted evidence
Include the Watch ID, Source ID, signal time with timezone, status, relevant
timestamps, HTTP status, error code, and X-Request-ID. Redact the raw
heartbeat URL, Source key, user token, callback secret, Action content, and
customer data.