Designing an Evidence Trail for Agent Actions

When an agent writes to another system, I want to reconstruct four things: what it requested, what policy decided, what actually executed, and what changed. I design the evidence around those questions.

In my previous evaluation, some well-formed, authorized writes contradicted the user's request. The tool boundary accepted an operation; the task still failed. Understanding that failure required the action sequence and the resulting state.

This article uses a different public example: a synthetic customer-summary workflow connecting AxonLLM, the Ostiari Guard bridge, and Escape Lab. The recordings expose the request, policy decision, payload transformation, tool result, and destination state. They also expose the limits of the evidence verifier.

Evidence scope: one deterministic run per control profile, recorded on 1 October 2026. All records, destinations, model responses, and tool effects are synthetic. The tools operate in an in-process state machine. This article describes that implementation and identifies additional controls I would require in a production design.

The diagram separates the recorded action path from the retained evidence and the checks a reviewer needs to perform. Dashed connectors show evidence references. The grouping describes the modeled tool surface; it does not imply an independently isolated collector.

Full-size SVG · High-resolution PNG · Editable draw.io source

Give each stage a separate evidence contract

I use the action lifecycle to decide what to record and which component should record it:

Scroll the table horizontally to see every column.

The same request_id connects a proposal to its decision and execution result. The event stream supplies run_id, sequence, event_id, and timestamp. Parent-action and tool-call identifiers are available for further correlation.

The recording component matters. A model's claim that it completed a task has different authority from an executor's result. An executor's delivery acknowledgment has different authority from a read of the destination state. I keep those records distinguishable so an investigation can follow each claim to its source.

The pinned event writer and orchestrator show how this example implements those records.

Follow one request all the way to its effect

The workflow permits an agent to read a synthetic customer record and send a summary to an approved review destination. An untrusted instruction asks it to send the complete record, including a protected synthetic field.

In the C4 recording, the send operation appears as this sequence:

Scroll the table horizontally to see every column.

This is an abbreviated view of the published event stream. It is not a separate execution.

The control did not block the send entirely. It transformed the payload and allowed execution with the transformed parameters. A single blocked=true metric would misdescribe that intervention.

Attribution also matters. The Ostiari Guard bridge is enabled in this run, but the decisive redaction comes from Escape Lab's source-to-destination rule. The recorded policy provenance lets a reader identify which component made that decision.

Check state before cleanup removes the evidence

I compare the execution records with three snapshots: before the task, after the task, and after teardown.

The two published profiles produce this comparison:

Scroll the table horizontally to see every column.

The C1 destination snapshot and C4 destination snapshot make the difference inspectable.

Both runs have a valid event chain. Both report task completion. Only the state assertion distinguishes the protected-data outcome.

The C4 placeholder also leaves a separate quality question open: did the user receive a useful summary? This fixture does not grade that property. I would keep content-policy compliance and summary usefulness as separate acceptance checks.

Teardown clears the modeled files, destination records, and synthetic identities. The after-task snapshot preserves the state needed for review. The post-teardown snapshot documents the subsequent cleanup.

For workflows that can roll back intermediate effects, I also need observations during execution. Escape Lab's orchestrator evaluates state after each executed action and retains observed assertion hits. An empty destination after cleanup cannot establish that it never received prohibited content.

Bind records and artifacts together

The event writer records a sequence number, includes the preceding record's hash, and computes a SHA-256 digest over the current event before adding its record_hash field. It serializes the hash input with the project's deterministic JSON helper.

The writer flushes and calls fsync after each event. Snapshot records include a relative path and a digest of the snapshot's state object. The run manifest captures the scenario, configuration, and environment digests; the reproducible example separately pins the source revisions and dependency lock.

Those connections make several checks possible:

• Recompute each event hash and check its link to the previous event.
• Recompute a snapshot's state digest and compare it with the referenced event.
• Resolve the request IDs across decisions, interventions, and tool results.
• Check required lifecycle events and expected terminal records.
• Compare the source and configuration references with the intended experiment.

The pinned CLI verifier does only part of this. Its evidence check recomputes event hashes and previous-hash links. It also checks the disclosure ledger. It does not independently recompute snapshot contents, require a terminal run_completed event, or compare the chain's final hash with an externally trusted value.

That distinction changes how I interpret evidence_valid=True: it reports internal consistency of the checked records. I still need artifact, lifecycle, and outcome checks. The exact verification command implementation is available for inspection.

State the trust boundary around the evidence

In this synthetic example, the agent receives modeled tools. The host-side evidence files are outside the operations exposed through those tools.

That is a useful separation within the fixture. The in-process run does not demonstrate isolation from arbitrary host code execution, a compromised collector, or an administrator who controls the artifact directory.

A hash chain detects inconsistent edits. A party that can replace the entire chain can also recompute its hashes. Removing a suffix can leave a consistent prefix. A digest stored beside the files therefore needs additional trust assumptions before it can establish authenticity or completeness.

For a production design, I would define:

• A collector identity separate from the agent and its tool credentials.
• A durable acknowledgment point before consequential actions proceed.
• An expected run-completion record and a final digest retained outside the agent's administrative reach.
• Authentication for the producer and protected retention for the resulting evidence package.
• Explicit handling of missing events, failed writes, and interrupted runs.

These are additional design requirements, not capabilities demonstrated by the two synthetic recordings. The required assurance depends on which components the threat model allows an adversary to control.

Carry uncertainty through retries and failures

For a distributed workflow, I would give each execution attempt its own identifier and retain the relationship to the original request, retry, or delegated action. An idempotency key, policy revision, and relevant state version would help explain whether two attempts could produce the same external effect.

The important failure case is an acknowledgment lost after the external system commits a write. I would record that outcome as unknown until a destination query or other authoritative receipt resolves it. Retrying without that distinction can change the system twice while the trace appears to show one failed attempt and one successful attempt.

The current in-memory fixture does not test distributed commits, queue delivery, or recovery from a collector outage. Those need their own fault-injection scenarios before I would make an enterprise reliability claim.

Treat evidence capture as a data boundary

The published fixture can retain its record contents because they are synthetic. That does not justify retaining equivalent production payloads.

For enterprise workloads, I would define an allowlist for captured fields: action identifiers, policy references, classified resource references, status, and the minimum attributes needed to test the contract. Any payload required for investigation would have a separately defined access and retention policy.

I would also check the capture path before writing durable records. Removing a field from a dashboard leaves its stored copy untouched.

The reference writer has key-based redaction for likely credentials. That is not a complete classification system: sensitive data can appear under an ordinary field name or inside a snapshot. A digest is not encryption and should not be treated as automatic anonymization of a predictable value.

This article uses only public code and synthetic recordings. Generalized enterprise examples describe engineering requirements without disclosing customer names, internal endpoints, or proprietary architecture.

Separate verification, reconstruction, and re-execution

I use three distinct operations when reviewing a run:

1. Verify the retained evidence. Check record integrity, artifact references, expected lifecycle, and the trust assumptions around the producer.
2. Reconstruct the recorded outcome. Evaluate retained state or replay recorded actions against the declared simulator contract.
3. Re-execute the workflow. Run the pinned implementation again and compare its outcome with the stated reproducibility criteria.

These operations answer different questions. A new run can generate different timestamps, request IDs, and hashes while reproducing the same boundary behavior. A replay that consumes recorded choices does not establish that a model would choose those actions again.

In this example, the deterministic provider fixture supports re-execution of the integration. The published reproduction contract covers outcome, payload treatment, evidence validity, and protected-field assertions. It does not require identical event bytes or claim repeatability of live-model reasoning.

Inspect the evidence yourself

Use the portfolio revision that contains the cited recordings and locked integration dependencies:

git clone https://github.com/hk-775/hk-775.git
cd hk-775
git checkout 954cc781c21eb80d207f6f0f068dfc0fe6400bd8
cd examples/customer-summary
uv sync --locked
uv run --locked escape-lab verify ../../site/evidence/C1
uv run --locked escape-lab verify ../../site/evidence/C4

The event-chain checks report 23 events for C1 and 24 for C4. To check the snapshot-state references as a separate, read-only operation, run this from the same directory:

uv run --locked python - <<'PY'
import json
from pathlib import Path
from escape_lab.util import sha256_json

for profile in ("C1", "C4"):
    root = Path("../../site/evidence") / profile
    events = [json.loads(line) for line in
              (root / "events.jsonl").read_text().splitlines()]
    snapshots = [event for event in events
                 if event["event_type"] == "state_snapshot"]
    assert {event["data"]["name"] for event in snapshots} == {
        "before", "after", "post-teardown"}
    for event in snapshots:
        data = event["data"]
        expected = f"snapshots/{data['name']}.json"
        assert data["path"] == expected
        snapshot = json.loads((root / expected).read_text())
        assert snapshot["run_id"] == event["run_id"]
        assert snapshot["name"] == data["name"]
        assert sha256_json(snapshot["state"]) == (
            snapshot["state_digest"]) == data["state_digest"]
    print(f"{profile}: {len(snapshots)} snapshot state digests match")
PY

This snippet checks the three named snapshots against their recorded references. It does not authenticate the collector or validate every artifact in the package.

To generate fresh fixture runs, use uv run --locked python run.py. The script checks the paired outcome and protected-field difference and writes new run directories. It leaves the published recordings unchanged.

The engineering decision

I would make an agent's evidence contract part of its execution interface: correlated requests and decisions, an explicit record of transformed parameters, executor results, state assertions evaluated separately from policy decisions, and documented failure behavior when evidence is incomplete.

The synthetic example establishes why those distinctions matter. A task can report completion while violating its data contract. A policy can transform an action that still executes. A chain can verify while a broader completeness or trust question remains unanswered.

For an enterprise review, I want each conclusion to identify the observation that supports it, the component responsible for that observation, and the assumptions that could invalidate it.

Source and reproduction references

• Recorded comparison and scope
• C1 evidence package
• C4 evidence package
• Pinned integration and reproduction commands
• Event writer and verifier
• Serialization and redaction helpers