AI Agent Orchestration: A Production Control Loop for Tools, Approvals, and Recovery

AI agent orchestration is the control system that turns an agent's changing plan into a reliable sequence of model calls, tool calls, policy checks, human decisions, and recorded outcomes. It owns execution state and transitions. The model may propose what happens next, but the orchestrator decides whether that transition is valid and how it survives failure.

That distinction matters in production. A convincing plan is not a checkpoint. A tool result is not proof that the intended operation ran exactly once. An approval is not permission to use any credential the reviewer happens to possess.

The most useful mental model is two connected loops:

1. A reasoning loop interprets the goal, selects a step, and learns from the result.
2. An execution control loop persists state, enforces policy, pauses for decisions, runs side effects, and records what actually happened.

The reasoning loop can be probabilistic. The execution control loop should be boring, explicit, and testable.

Orchestration is more than routing between agents

Some systems use "orchestration" to mean choosing which agent handles a task. Routing is part of the job, but production orchestration also answers harder questions:

• Which run owns this tool call?
• Has the same logical step already executed?
• Is the tool allowed for this workload and resource?
• Did a person approve these exact arguments or an older version?
• What happens if the worker stops while waiting?
• Which failures can be retried without repeating a side effect?
• Did the approved operation succeed, fail, or roll back?

Anthropic's guide to building effective agents separates predefined workflows from agents that dynamically direct their own tool use. AWS Prescriptive Guidance for workflow orchestration agents includes state tracking, retries, conditional execution, and coordination with humans. Both point to the same practical boundary: model-driven planning still needs ordinary software to own control flow.

The two-loop reference architecture

The model never writes directly to the protected system in this design. It produces a proposal. A deterministic boundary validates the proposal, computes policy, and decides whether to deny, execute, or request review.

Google Cloud's agentic architecture component guide describes models, tools, memory, runtimes, frameworks, and design patterns as separate choices. The diagram above adds the production control path between those components. Memory helps the agent reason. Durable run state tells the system what has already happened.

Make the run a state machine

If run state exists only inside a prompt or worker process, recovery is guesswork. Persist an explicit state machine and allow only named transitions.

stateDiagram-v2
  [*] --> queued
  queued --> planning
  planning --> tool_proposed
  tool_proposed --> denied
  tool_proposed --> waiting_for_approval
  tool_proposed --> executing
  waiting_for_approval --> executing: approved snapshot
  waiting_for_approval --> rejected
  waiting_for_approval --> expired
  executing --> completed
  executing --> failed
  failed --> executing: safe retry
  queued --> cancelled
  planning --> cancelled
  waiting_for_approval --> cancelled

The model can recommend a transition. Application code validates and commits it. A transition record should contain enough evidence to reject stale events and recover after a crash:

{
  "run_id": "run_release_842",
  "run_version": 4,
  "state": "waiting_for_approval",
  "step": {
    "id": "promote-checkout",
    "attempt": 1,
    "tool": "deployment.promote",
    "arguments_sha256": "sha256:<canonical-argument-hash>",
    "idempotency_key": "run_release_842:promote-checkout"
  },
  "policy": {
    "decision": "require_human",
    "policy_version": "release-policy-v4"
  },
  "approval": {
    "action_id": "act_<id>",
    "action_version": 1,
    "fingerprint": "sha256:<reviewed-action-fingerprint>"
  },
  "updated_at": "2026-09-01T18:42:11Z"
}

This is a vendor-neutral run record, not an ActionBox API payload. Store the canonical operation in a system authorized to hold it. The hash lets the orchestrator compare snapshots without copying sensitive arguments through logs and approval messages.

Five invariants that keep the control loop safe

1. Give every logical step a stable identity

Network failures create ambiguity. A worker can send a request, lose the response, and have no idea whether the server accepted it. Retrying with a fresh identifier creates duplicate work. Reusing a stable idempotency key lets the receiver return the original result or reject changed content.

Derive the key from business identity, not process identity:

run_id + step_id + operation_version

A random value generated on every attempt cannot connect those attempts to the same logical operation.

2. Bind decisions to immutable operation evidence

"Approve the deployment" is too vague when the agent can change the version, environment, or arguments after review. Persist a canonical argument hash or version before requesting approval. Recheck it immediately before execution.

If material input changes, create a new review. Do not carry approval forward because the title still looks similar.

3. Classify retry semantics before running the tool

A timeout does not tell you whether a remote side effect happened. The AWS durable execution guidance on idempotency and retries distinguishes at-least-once work, which must tolerate repetition, from at-most-once work, which stops rather than risk replay after an ambiguous interruption. Neither label magically provides exactly-once execution.

4. Treat waiting as durable state

A person may answer after the original worker has stopped. Waiting for approval must not require an open browser tab, a held HTTP connection, or one specific process. Store the approval identity, return the worker to the queue, and let any authorized worker reconcile the result later.

Microsoft's durable agent patterns model human input as an external event in a durable orchestration. The same separation works without that framework: the orchestrator owns the checkpoint, while the decision system owns the request and response.

5. Record execution separately from authorization

An approval says the operation may proceed. It does not say the tool call succeeded. Store a separate outcome with the real status, duration, error class, and rollback state. This distinction prevents dashboards from reporting approved failures as successful automation.

A small ActionBox orchestration worker

The following Python worker demonstrates the control boundary with only the standard library. It creates one idempotent approval request, reconciles the durable Action until it becomes terminal, writes a harmless local release marker as the protected operation, and reports the execution outcome.

The local marker uses exclusive creation, so restarting the example does not write the same logical release twice. Replace that function with a downstream API that accepts its own idempotency key or supports authoritative state reconciliation.

In the inbox, the reviewer sees the proposed effect, risk, rollback plan, affected scope, and source evidence before choosing. The orchestration worker remains paused outside the inbox.

from __future__ import annotations

import json
import os
import time
import urllib.request
from datetime import datetime, timedelta, timezone
from pathlib import Path

API = "https://api.actionbox.cloud"
SOURCE_KEY = os.environ["ACTIONBOX_SOURCE_KEY"]
RUN_ID = os.environ.get("RUN_ID", "release-842")


def request(method: str, path: str, body=None, idempotency_key=None) -> dict:
    data = None if body is None else json.dumps(body).encode()
    headers = {"Authorization": f"Bearer {SOURCE_KEY}"}
    if body is not None:
        headers["Content-Type"] = "application/json"
    if idempotency_key:
        headers["Idempotency-Key"] = idempotency_key

    req = urllib.request.Request(
        f"{API}{path}", data=data, headers=headers, method=method
    )
    with urllib.request.urlopen(req, timeout=35) as response:
        return json.load(response)


def create_review() -> dict:
    expires_at = datetime.now(timezone.utc) + timedelta(minutes=30)
    payload = {
        "title": "Publish release 842?",
        "description": "The release worker is paused before publishing its marker.",
        "priority": "high",
        "options": [
            {"id": "approve", "label": "Approve publish", "style": "primary"},
            {"id": "reject", "label": "Reject", "style": "destructive"},
        ],
        "expires_at": expires_at.isoformat().replace("+00:00", "Z"),
        "metadata": {"run_id": RUN_ID, "tool": "release.publish"},
    }
    return request(
        "POST",
        "/v1/actions",
        payload,
        idempotency_key=f"{RUN_ID}:publish:approval-v1",
    )["data"]


def wait_for_terminal(action_id: str, deadline_seconds: int = 1800) -> dict:
    deadline = time.monotonic() + deadline_seconds
    while time.monotonic() < deadline:
        action = request(
            "GET", f"/v1/source/actions/{action_id}?wait_seconds=30"
        )["data"]
        if action["status"] != "open":
            return action
    raise TimeoutError("Local wait ended without a terminal Action")


def publish_once() -> str:
    output = Path("release-output")
    output.mkdir(exist_ok=True)
    marker = output / f"{RUN_ID}.json"
    content = json.dumps({"run_id": RUN_ID, "published": True}) + "\n"
    try:
        fd = os.open(marker, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
    except FileExistsError:
        return "already_published"
    with os.fdopen(fd, "w") as handle:
        handle.write(content)
    return "published"


def report_outcome(action: dict, status: str) -> None:
    request(
        "POST",
        f"/v1/actions/{action['id']}/outcome",
        {
            "status": status,
            "action_version": action["action_version"],
            "fingerprint": action["fingerprint"],
            "rollback": False,
        },
        idempotency_key=f"{RUN_ID}:publish:outcome-v1",
    )


action = wait_for_terminal(create_review()["id"])

if action["status"] != "resolved" or action["resolution_option_id"] != "approve":
    raise SystemExit(f"Stopped safely: {action['status']}")

try:
    print(publish_once())
except Exception:
    report_outcome(action, "failed")
    raise
else:
    report_outcome(action, "success")

Run it with a Source key kept in your server-side secret store:

export ACTIONBOX_SOURCE_KEY='axb_live_…'
export RUN_ID='release-842'
python3 orchestrate_release.py

Do not put the Source key in browser code, source control, screenshots, or the Action metadata. The worker remains the executor. A signed-in workspace member only decides the bounded Action.

The ActionBox API can return while the Action is still open, so the worker reconciles in a loop. resolved with the approve option is the only branch that executes. Rejection, cancellation, expiry, malformed data, or an API error stops the operation. The outcome report is bound to the resolved action_version and fingerprint.

After execution, History keeps the human decision and the reported software outcome as separate evidence. The event trail also shows how the Action reached its terminal state.

For the complete contract, see API authentication, idempotency and dedupe, and decisions and outcomes.

Instrument the transitions, not the model's private reasoning

Operational telemetry should answer where a run is stuck and what crossed a control boundary. Useful fields include:

• run ID, step ID, attempt, and state transition;
• agent and integration identity;
• model and tool name;
• policy result and policy version;
• approval ID, response type, and waiting duration;
• tool duration, error class, and outcome;
• token usage and cost when the provider returns them.

Avoid recording raw prompts, tool arguments, or results by default. They can contain credentials, personal data, and proprietary context. OpenTelemetry's generative AI semantic conventions define operations such as invoke_agent, invoke_workflow, and execute_tool, while warning that tool arguments and results may contain sensitive information.

The trace and the durable run record solve different problems. A trace explains timing and causality across services. The run record decides what may happen next after every process involved in that trace has disappeared.

Test the seams where ownership changes

Happy-path agent demos test whether the model can select a tool. Production tests should target the boundaries between planner, state store, policy, reviewer, and executor.

When to add more agents

More agents are useful when subtasks need distinct context, tools, or evaluation. They are not a substitute for run state or policy.

The simplest design that meets the reliability requirement is usually the easiest to secure and operate. Add a router when routing helps. Add a second agent when specialization helps. Keep execution authority in the same explicit control plane either way.

Production review checklist

• Is each run and logical tool step durably identified?
• Are legal state transitions enforced outside the model?
• Can a worker restart without recreating reviews or repeating side effects?
• Are tool names, resources, and arguments validated before execution?
• Is human approval bound to the exact operation snapshot?
• Are retry semantics documented for every side-effecting tool?
• Does rejection, expiry, cancellation, or uncertainty stop safely?
• Is execution outcome stored separately from approval?
• Can operators find a run by its business ID without reading raw prompts?
• Do traces redact or omit sensitive tool content by default?

Sources and further reading

• Anthropic: Building effective agents
• Google Cloud: Choose your agentic AI architecture components
• AWS: Workflow orchestration agents
• AWS: Idempotency and retries
• Microsoft: Durable agent application patterns
• OpenTelemetry: Generative AI semantic conventions
• AI agent identity and access control: separate planner, integration, reviewer, and executor identities.
• Durable execution explained: checkpoint and resume patterns for long-running agent work.
• AI agent observability: trace decisions, tool calls, and outcomes without turning logs into a data leak.

Create a free Source · Read the orchestration API contract · Add a human approval gate