ActionBoxBlog

SREDevOpsPlatform

Incident response approval workflow with an SLA countdown

A copy-paste incident response approval workflow for rollbacks, scaling, and kill switches with a visible SLA countdown and fail-closed timeout.

Suson Sapkota 3 min read Incident response

When the dashboard is red and the on-call engineer is in the shower, your runbook has two options: act blindly, or wait forever. The right answer is a decision with a countdown — the automation escalates to a human, shows the SLA ticking down, and executes the pre-approved fallback if nobody answers.

This is the pattern: detect → decide → act-on-approval-or-fallback.

Incident response approval workflow

bash
# Alert fired: error rate > 5% for 5 minutes
actionbox ask "Error rate 12% — roll back canary to stable v2.14.1?" \
  --option rollback="Roll back now" \
  --option investigate="Keep canary, investigate" \
  --context-json '[{"type":"key_value","items":{"service":"checkout","error_rate":"12%","canary":"v2.15.0","stable":"v2.14.1","incident":"INC-4821"}}]' \
  --expires "$(date -u -d '+15 minutes' '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -v+15M '+%Y-%m-%dT%H:%M:%SZ')" \
  --callback-url https://ops.acme.com/actionbox/rollback-callback

The countdown is the SLA: the dashboard and push notifications show how long the approver has before the fallback executes.

Step 1: fail closed with an explicit fallback

Decide what "nobody answered" means before the incident. Two sane defaults:

  • Rollback gate: timeout → return to stable (deploy is reversible, act)
  • Kill switch: timeout → keep the system running and page louder (can't un-send a payment email)
bash
# Reversible action: fall back to the safe state
actionbox ask "Roll back to stable?" \
  --option rollback="Roll back now" \
  --option investigate="Keep canary" \
  --on-expire-json '{"type":"resolve","response":{"type":"single_choice","value":"rollback"},"reason":"SLA elapsed, auto-rollback"}' \
  --expires "$(date -u -d '+15 minutes' '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -v+15M '+%Y-%m-%dT%H:%M:%SZ')"

# Irreversible action: fail closed, keep paging
actionbox ask "Delete the corrupted queue partition?" \
  --option delete="Delete now" \
  --option hold="Hold, investigate" \
  --on-expire-json '{"type":"return_expired"}' \
  --expires "$(date -u -d '+10 minutes' '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || date -u -v+10M '+%Y-%m-%dT%H:%M:%SZ')"

return_expired means "the Action expires unresolved" — your callback receives the expired event and can trigger the next escalation level (page the manager, post to the war room).

Step 2: the callback does the work, idempotently

Every decision lands in your callback signed with HMAC-SHA256 — verify before acting:

bash
#!/usr/bin/env bash
# rollback-callback: verify X-Actionbox-Timestamp and X-Actionbox-Signature
# over "timestamp.raw_body" with the Source webhook secret before parsing.
DECISION=$(jq -r '.data.response.value // .data.decision // empty' "$PAYLOAD")

if [[ "$DECISION" == "rollback" ]]; then
  kubectl rollout undo deployment/checkout
  ./scripts/notify-war-room.sh "Rolled back per INC-4821"
fi

Because the decision is a webhook, the rollback runs even if the requesting cron/alert process died. Idempotency keys make retries safe.

Step 3: wire it to your alerting

PagerDuty, Grafana, Datadog, Prometheus — any alert hook can become a decision:

yaml
# Prometheus alert -> webhook -> Actionbox
groups:
  - name: checkout
    rules:
      - alert: CheckoutErrorRateHigh
        expr: sum(rate(http_requests_total{service="checkout",status="5xx"}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "Error rate high — decide: rollback or investigate?"
          runbook: /opt/actionbox/runbooks/checkout-rollback.sh
yaml
# Grafana contact point -> webhook
- name: actionbox-gate
  type: webhook
  settings:
    url: https://api.actionbox.cloud/v1/actions
    http_method: POST
    authorization: Bearer $ACTIONBOX_TOKEN

Why a decision beats a page

Plain pageActionbox decision
On-call sees"error rate high""roll back? approve / investigate"
Contextpull logs separatelyattached (rate, versions, incident)
No answeralarm fatigue, re-pageexplicit fallback on record
Auditpager historywho decided, when, why, what ran

Try it

  1. Create a Source and paste its token into your alert webhook
  2. Copy the ask command above into your runbook script
  3. Trigger a fake alert and watch the countdown in the dashboard

Create a free Source · Webhook reference · Gated cron jobs

#incident-response#runbooks#sla#on-call

Try this workflow in minutes

Create a free Source, then run the exact commands from this post against the live API — no approval infrastructure to build.

Create a free SourceRead the docs
S
Suson Sapkota

Founder, Actionbox

Related workflows