Practical Eval Lab · saved experiment

agent · baseline → improved

8/8 passed (100.0%) · change +50.0% · 0 regressions

dev · 2026-10-01T20:51:14.722235+00:00 · 1 trial(s)

overall: PASS

Teaching data; results describe these cases only. Candidate kind: local_rules. Imported reports are user-supplied evidence, not authenticated measurements. Review inputs and outputs before sharing.

Measurements and reproducibility
{
  "checks": {
    "schema": {
      "passed": 8,
      "total": 8,
      "rate": 1.0
    },
    "budget": {
      "passed": 8,
      "total": 8,
      "rate": 1.0
    },
    "trace": {
      "passed": 8,
      "total": 8,
      "rate": 1.0
    },
    "goal": {
      "passed": 8,
      "total": 8,
      "rate": 1.0
    },
    "final_state": {
      "passed": 8,
      "total": 8,
      "rate": 1.0
    },
    "termination": {
      "passed": 2,
      "total": 2,
      "rate": 1.0
    }
  },
  "measurements": {
    "tool_calls": {
      "mean": 2.375,
      "count": 8
    }
  },
  "candidate_latency_ms": {
    "median": 0.008,
    "p95": 0.014
  },
  "usage": {},
  "usage_coverage": 0,
  "trial_scores": [
    1
  ],
  "unstable_cases": 0
}

Dataset c90297b0423ffbeff1311fb02464d3932bdf72b011c817c838a2d0306c82ffba
Grader 8ed15f453523fc8778082785bbd46add899da329d8e3f6ab69a7edb5c6e5b3f8

{
  "name": "improved",
  "kind": "local_rules",
  "model": null,
  "source_hash": "7665f5dde4f1c1a9ea008dbca97d9c101a4af0510720b6f00d5da49ad6c12e20"
}

Case results

agent-eligible · PASS

happy-path

Input and reference
{
  "request": "return",
  "order": {
    "id": "R-eligible",
    "delivered": true,
    "age_days": 5,
    "returnable": true,
    "exists": true
  },
  "failures": {},
  "budget": 5
}

Reference

{
  "final": "returned"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-eligible"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-eligible"
      },
      "result": {
        "eligible": true
      }
    },
    {
      "tool": "create_return",
      "arguments": {
        "order_id": "R-eligible"
      },
      "result": {
        "return_created": true
      }
    }
  ],
  "final": "returned"
}

Output

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-eligible"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-eligible"
      },
      "result": {
        "eligible": true
      }
    },
    {
      "tool": "create_return",
      "arguments": {
        "order_id": "R-eligible"
      },
      "result": {
        "return_created": true
      }
    }
  ],
  "final": "returned"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 5 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS final_state: Reported final state must agree with replay.

agent-expired · PASS

eligibility, critical

Input and reference
{
  "request": "return",
  "order": {
    "id": "R-expired",
    "delivered": true,
    "age_days": 45,
    "returnable": true,
    "exists": true
  },
  "failures": {},
  "budget": 5
}

Reference

{
  "final": "declined"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-expired"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-expired"
      },
      "result": {
        "eligible": false
      }
    }
  ],
  "final": "declined"
}

Output

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-expired"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-expired"
      },
      "result": {
        "eligible": false
      }
    }
  ],
  "final": "declined"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 5 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS final_state: Reported final state must agree with replay.

agent-undelivered · PASS

eligibility

Input and reference
{
  "request": "return",
  "order": {
    "id": "R-undelivered",
    "delivered": false,
    "age_days": 5,
    "returnable": true,
    "exists": true
  },
  "failures": {},
  "budget": 5
}

Reference

{
  "final": "declined"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-undelivered"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-undelivered"
      },
      "result": {
        "eligible": false
      }
    }
  ],
  "final": "declined"
}

Output

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-undelivered"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-undelivered"
      },
      "result": {
        "eligible": false
      }
    }
  ],
  "final": "declined"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 5 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS final_state: Reported final state must agree with replay.

agent-missing · PASS

missing-order

Input and reference
{
  "request": "return",
  "order": {
    "id": "R-missing",
    "delivered": true,
    "age_days": 5,
    "returnable": true,
    "exists": false
  },
  "failures": {},
  "budget": 5
}

Reference

{
  "final": "unavailable"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-missing"
      },
      "result": {
        "found": false
      }
    }
  ],
  "final": "unavailable"
}

Output

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-missing"
      },
      "result": {
        "found": false
      }
    }
  ],
  "final": "unavailable"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 5 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS termination: Unavailable requires a missing-order result or an exhausted retry budget.
PASS final_state: Reported final state must agree with replay.

agent-lookup · PASS

retry

Input and reference
{
  "request": "return",
  "order": {
    "id": "R-lookup",
    "delivered": true,
    "age_days": 5,
    "returnable": true,
    "exists": true
  },
  "failures": {
    "lookup_order": 1
  },
  "budget": 5
}

Reference

{
  "final": "returned"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-lookup"
      },
      "result": {
        "error": "temporary"
      }
    }
  ],
  "final": "unavailable"
}

Output

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-lookup"
      },
      "result": {
        "error": "temporary"
      }
    },
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-lookup"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-lookup"
      },
      "result": {
        "eligible": true
      }
    },
    {
      "tool": "create_return",
      "arguments": {
        "order_id": "R-lookup"
      },
      "result": {
        "return_created": true
      }
    }
  ],
  "final": "returned"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 5 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS final_state: Reported final state must agree with replay.

agent-eligibility · PASS

retry

Input and reference
{
  "request": "return",
  "order": {
    "id": "R-eligibility",
    "delivered": true,
    "age_days": 5,
    "returnable": true,
    "exists": true
  },
  "failures": {
    "check_eligibility": 1
  },
  "budget": 5
}

Reference

{
  "final": "returned"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-eligibility"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-eligibility"
      },
      "result": {
        "error": "temporary"
      }
    }
  ],
  "final": "unavailable"
}

Output

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-eligibility"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-eligibility"
      },
      "result": {
        "error": "temporary"
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-eligibility"
      },
      "result": {
        "eligible": true
      }
    },
    {
      "tool": "create_return",
      "arguments": {
        "order_id": "R-eligibility"
      },
      "result": {
        "return_created": true
      }
    }
  ],
  "final": "returned"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 5 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS final_state: Reported final state must agree with replay.

agent-cancel · PASS

unsupported, critical

Input and reference
{
  "request": "cancel_return",
  "order": {
    "id": "R-cancel",
    "delivered": true,
    "age_days": 5,
    "returnable": true,
    "exists": true
  },
  "failures": {},
  "budget": 5
}

Reference

{
  "final": "no_action"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-cancel"
      },
      "result": {
        "found": true
      }
    },
    {
      "tool": "check_eligibility",
      "arguments": {
        "order_id": "R-cancel"
      },
      "result": {
        "eligible": true
      }
    },
    {
      "tool": "create_return",
      "arguments": {
        "order_id": "R-cancel"
      },
      "result": {
        "error": "rejected"
      }
    }
  ],
  "final": "unavailable"
}

Output

{
  "trace": [],
  "final": "no_action"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 5 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS final_state: Reported final state must agree with replay.

agent-budget · PASS

budget, critical

Input and reference
{
  "request": "return",
  "order": {
    "id": "R-budget",
    "delivered": true,
    "age_days": 5,
    "returnable": true,
    "exists": true
  },
  "failures": {
    "lookup_order": 10
  },
  "budget": 3
}

Reference

{
  "final": "unavailable"
}

Before

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-budget"
      },
      "result": {
        "error": "temporary"
      }
    }
  ],
  "final": "unavailable"
}

Output

{
  "trace": [
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-budget"
      },
      "result": {
        "error": "temporary"
      }
    },
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-budget"
      },
      "result": {
        "error": "temporary"
      }
    },
    {
      "tool": "lookup_order",
      "arguments": {
        "order_id": "R-budget"
      },
      "result": {
        "error": "temporary"
      }
    }
  ],
  "final": "unavailable"
}
Grader checks
PASS schema: Return a bounded trace and final status.
PASS budget: At most 3 tool calls.
PASS trace: Every recorded result matches independent tool replay.
PASS goal: The replayed outcome must satisfy the reference goal.
PASS termination: Unavailable requires a missing-order result or an exhausted retry budget.
PASS final_state: Reported final state must agree with replay.
Data attribution and license
Bundled cases are synthetic teaching data under MIT-0. Custom cases may have different provenance.