When the dashboard is red and the on-call engineer is in the shower, your runbook has two options: act blindly, or wait forever. The right answer is a decision with a countdown — the automation escalates to a human, shows the SLA ticking down, and executes the pre-approved fallback if nobody answers.
This is the pattern: detect → decide → act-on-approval-or-fallback.
Incident response approval workflow
# Alert fired: error rate > 5% for 5 minutes
actionbox ask "Error rate 12% — roll back canary to stable v2.14.1?" \
--option rollback="Roll back now" \
--option investigate="Keep canary, investigate" \
--context-json '[{"type":"key_value","items":{"service":"checkout","error_rate":"12%","canary":"v2.15.0","stable":"v2.14.1","incident":"INC-4821"}}]' \
--expires "$(date -u -d '+15 minutes' '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -v+15M '+%Y-%m-%dT%H:%M:%SZ')" \
--callback-url https://ops.acme.com/actionbox/rollback-callbackThe countdown is the SLA: the dashboard and push notifications show how long the approver has before the fallback executes.
Step 1: fail closed with an explicit fallback
Decide what "nobody answered" means before the incident. Two sane defaults:
- Rollback gate: timeout → return to stable (deploy is reversible, act)
- Kill switch: timeout → keep the system running and page louder (can't un-send a payment email)
# Reversible action: fall back to the safe state
actionbox ask "Roll back to stable?" \
--option rollback="Roll back now" \
--option investigate="Keep canary" \
--on-expire-json '{"type":"resolve","response":{"type":"single_choice","value":"rollback"},"reason":"SLA elapsed, auto-rollback"}' \
--expires "$(date -u -d '+15 minutes' '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -v+15M '+%Y-%m-%dT%H:%M:%SZ')"
# Irreversible action: fail closed, keep paging
actionbox ask "Delete the corrupted queue partition?" \
--option delete="Delete now" \
--option hold="Hold, investigate" \
--on-expire-json '{"type":"return_expired"}' \
--expires "$(date -u -d '+10 minutes' '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -v+10M '+%Y-%m-%dT%H:%M:%SZ')"return_expired means "the Action expires unresolved" — your callback receives the expired event and can trigger the next escalation level (page the manager, post to the war room).
Step 2: the callback does the work, idempotently
Every decision lands in your callback signed with HMAC-SHA256 — verify before acting:
#!/usr/bin/env bash
# rollback-callback: verify X-Actionbox-Timestamp and X-Actionbox-Signature
# over "timestamp.raw_body" with the Source webhook secret before parsing.
DECISION=$(jq -r '.data.response.value // .data.decision // empty' "$PAYLOAD")
if [[ "$DECISION" == "rollback" ]]; then
kubectl rollout undo deployment/checkout
./scripts/notify-war-room.sh "Rolled back per INC-4821"
fiBecause the decision is a webhook, the rollback runs even if the requesting cron/alert process died. Idempotency keys make retries safe.
Step 3: wire it to your alerting
PagerDuty, Grafana, Datadog, Prometheus — any alert hook can become a decision:
# Prometheus alert -> webhook -> Actionbox
groups:
- name: checkout
rules:
- alert: CheckoutErrorRateHigh
expr: sum(rate(http_requests_total{service="checkout",status="5xx"}[5m])) > 0.05
for: 5m
labels:
severity: page
annotations:
summary: "Error rate high — decide: rollback or investigate?"
runbook: /opt/actionbox/runbooks/checkout-rollback.sh# Grafana contact point -> webhook
- name: actionbox-gate
type: webhook
settings:
url: https://api.actionbox.cloud/v1/actions
http_method: POST
authorization: Bearer $ACTIONBOX_TOKENWhy a decision beats a page
| Plain page | Actionbox decision | |
|---|---|---|
| On-call sees | "error rate high" | "roll back? approve / investigate" |
| Context | pull logs separately | attached (rate, versions, incident) |
| No answer | alarm fatigue, re-page | explicit fallback on record |
| Audit | pager history | who decided, when, why, what ran |
Sources & Research
- Google Site Reliability Engineering (SRE) Book — Incident management and emergency response principles.
- PagerDuty Incident Response Best Practices — On-call triage, escalation SLAs, and failover workflows.
- Prometheus Alertmanager Architecture — High-urgency alert routing and webhook receivers.
Try it
- Create a Source and paste its token into your alert webhook
- Copy the
askcommand above into your runbook script - Trigger a fake alert and watch the countdown in the dashboard