CI turned green after three retries, and nobody knows the test failed once. Next week it fails four times, then five — and the suite is now a slot machine. The retry button is the symptom; the silent retry is the disease. A flaky test retry approval makes each retry a human decision on record: the failure, the log, the rerun — and why we're allowed to ship anyway.
The pattern: retries are requests, not reflexes
| Situation | What happens | Decision needed |
|---|---|---|
| Test fails once, unknown flake | Ask a human: retry or investigate | Yes — with the log attached |
| Retry passes, flake confirmed | Record it, count it against a budget | No — but the record is permanent |
| Retry fails again | Block the merge — real failure | No |
| Infra failure (timeout, network) | Auto-retry once | No — it's not a test failure |
| Flaky budget exceeded | Block until a human investigates | Yes |
Step 1: fail loud, then ask
The first failure must not auto-retry. It stops the pipeline and creates a decision with the evidence:
name: test-gate
on:
pull_request:
permissions:
contents: read
checks: write
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests once
id: test
run: npm test -- --reporter=json --outputFile=results.json
continue-on-error: true
- name: Parse failures
id: parse
if: steps.test.outcome == 'failure'
run: |
node -e "
const r = require('./results.json');
const failed = r.testResults
.flatMap(s => s.assertionResults.filter(a => a.status === 'failed'));
console.log(failed.map(f => f.fullName).join('\n'));
require('fs').writeFileSync('failures.txt', failed.map(f => f.fullName).join('\n'));
"
echo "failed=$(wc -l < failures.txt)" >> "$GITHUB_OUTPUT"
- name: Install Actionbox CLI
if: steps.parse.outputs.failed > 0
run: python -m pip install actionbox
- name: Ask before retry
if: steps.parse.outputs.failed > 0
id: approval
env:
ACTIONBOX_TOKEN: ${{ secrets.ACTIONBOX_TOKEN }}
run: |
result=$(actionbox ask "Retry ${{ steps.parse.outputs.failed }} failed test(s) as flaky?" \
--option approve="Retry" \
--option reject="Fail the job" \
--context-json "[{\"type\":\"key_value\",\"items\":{\"failed_tests\":\"${{ steps.parse.outputs.failed }}\",\"suite\":\"unit\",\"attempt\":\"first failure\",\"log\":\"${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\"}}]" \
--wait --timeout 15m --json)
echo "decision=$(jq -r '.decision // empty' <<< "$result")" >> "$GITHUB_OUTPUT"
- name: Rerun on approval
if: steps.approval.outputs.decision == 'approve'
run: npm testThe approver sees the count and the exact log link — and their "retry" is on the record, so a green build is never a guess about what passed.
Step 2: count flakes against a budget
A retry here and there is fine. A retry on every PR is a broken test hiding in the suite. Track them mechanically:
Keep the retry count in your CI or test-results store and compare it with the
budget before creating the next Action. The current Actionbox CLI does not
provide an actionbox list command; use the documented Actions API or History
view when you need to reconcile the audit record.
The budget makes flakiness visible as a trend instead of a vibe — and gives the test owner a number to be responsible for.
Step 3: skips are decisions too
"Skip this test, we'll fix it later" is a policy decision, and it deserves a signature — not a test.skip in a drive-by commit:
- name: Skip requires an on-record approval
if: contains(github.event.pull_request.labels.*.name, 'flake-skip-approved')
run: |
echo "Skip approved with the linked decision on the PR."
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}A skipped test with an attached decision is a tracked debt. A skipped test with no decision is a hole in your coverage that nobody owns.
Full example: quarantine flow with a reopen date
#!/usr/bin/env bash
# quarantine.sh — move a flaky test to quarantine with a decision + expiry
set -euo pipefail
TEST_NAME="${1:?test name}"
actionbox ask \
"Quarantine ${TEST_NAME} and reopen in 14 days?" \
--option approve="Quarantine (auto-reopens 14d)" \
--option deny="Keep it in the suite" \
--context-json "[{\"type\":\"key_value\",\"items\":{\"test\":\"${TEST_NAME}\",\"failure_rate\":\"4/10 runs\",\"owner\":\"team-ci\",\"reopen\":\"2026-08-30\"}}]" \
--callback-url https://ops.acme.com/actionbox/quarantine-callbackQuarantine with an expiry is honest debt: the test comes back with a date, and the decision that moved it is findable forever.
Why this beats "add a retry plugin"
- Green builds mean something — every retry that produced the green is named
- Flakiness is measurable — a budget turns "the suite feels flaky" into a number
- Skips have owners — no test disappears without a recorded decision
- Real failures still block — a second failure is a merge stop, no questions asked
Sources & Research
- Playwright Test Runner Documentation — Test isolation, retries, and failure diagnostics.
- Google Testing Blog: Flaky Tests at Scale — Root cause analysis and quarantine strategies.
- Playwright Project — Playwright is maintained by Microsoft under Apache 2.0.
Try it
- Install the CLI on your runner
- Create a Source →
ACTIONBOX_TOKEN - Replace your retry plugin with the ask pattern
Create a free Source · GitHub Actions reference · Deploy approval gates in GitHub Actions · Dependency PR approvals without rubber-stamping