Practical Eval Lab · saved experiment

tool_calling · baseline → improved

8/10 passed (80.0%) · change +30.0% · 1 regressions

holdout · 2026-10-01T20:51:14.696093+00:00 · 1 trial(s)

overall: PASS

Teaching data; results describe these cases only. Candidate kind: local_rules. Imported reports are user-supplied evidence, not authenticated measurements. Review inputs and outputs before sharing.

Measurements and reproducibility
{
  "checks": {
    "call_schema": {
      "passed": 10,
      "total": 10,
      "rate": 1.0
    },
    "tool": {
      "passed": 9,
      "total": 10,
      "rate": 0.9
    },
    "arguments": {
      "passed": 8,
      "total": 10,
      "rate": 0.8
    },
    "outcome": {
      "passed": 8,
      "total": 10,
      "rate": 0.8
    }
  },
  "measurements": {},
  "candidate_latency_ms": {
    "median": 0.004,
    "p95": 0.009
  },
  "usage": {},
  "usage_coverage": 0,
  "trial_scores": [
    0.8
  ],
  "unstable_cases": 0
}

Dataset c83ed18048c9691ed7cf7bf1f88d5705794c324325b5fe72bc2a2a4844af7929
Grader 1b0a9e36d57f2e6711a2e65c81eb426f18e0e17a341dd186b335f1483e157c87

{
  "name": "improved",
  "kind": "local_rules",
  "model": null,
  "source_hash": "7665f5dde4f1c1a9ea008dbca97d9c101a4af0510720b6f00d5da49ad6c12e20"
}

Case results

tool-holdout-01 · PASS

lookup, typical

Input and reference
Track order B-205 please.

Reference

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "B-205"
  }
}

Before

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "B-205"
  }
}

Output

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "B-205"
  }
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected lookup_order
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.

tool-holdout-02 · PASS

lookup, not-found

Input and reference
What is the status of Z-999?

Reference

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "Z-999"
  }
}

Before

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "Z-999"
  }
}

Output

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "Z-999"
  }
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected lookup_order
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.

tool-holdout-03 · PASS

boundary, negation

Input and reference
Do not create a ticket: Excel crashes.

Reference

{
  "tool": "no_action",
  "arguments": {}
}

Before

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Software",
    "summary": "Excel crashes."
  }
}

Output

{
  "tool": "no_action",
  "arguments": {}
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected no_action
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.

tool-holdout-04 · PASS

ticket, typical

Input and reference
Open a support ticket: The screen is cracked.

Reference

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Hardware",
    "summary": "The screen is cracked."
  }
}

Before

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Hardware",
    "summary": "The screen is cracked."
  }
}

Output

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Hardware",
    "summary": "The screen is cracked."
  }
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected create_ticket
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.

tool-holdout-05 · FAIL

ticket, vocabulary

Input and reference
Create a ticket: My headphones have no sound.

Reference

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Hardware",
    "summary": "My headphones have no sound."
  }
}

Before

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Other",
    "summary": "My headphones have no sound."
  }
}

Output

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Other",
    "summary": "My headphones have no sound."
  }
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected create_ticket
FAIL arguments: All argument values must match the reference.
FAIL outcome: Compare simulated side effects or lookup results.

tool-holdout-06 · FAIL

lookup, vocabulary

Input and reference
Could you locate A-104?

Reference

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "A-104"
  }
}

Before

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "A-104"
  }
}

Output

{
  "tool": "no_action",
  "arguments": {}
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
FAIL tool: Expected lookup_order
FAIL arguments: All argument values must match the reference.
FAIL outcome: Compare simulated side effects or lookup results.

tool-holdout-07 · PASS

boundary, cancellation

Input and reference
Cancel C-306 please.

Reference

{
  "tool": "no_action",
  "arguments": {}
}

Before

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "C-306"
  }
}

Output

{
  "tool": "no_action",
  "arguments": {}
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected no_action
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.

tool-holdout-08 · PASS

missing, lookup

Input and reference
Track my order please.

Reference

{
  "tool": "no_action",
  "arguments": {}
}

Before

{
  "tool": "no_action",
  "arguments": {}
}

Output

{
  "tool": "no_action",
  "arguments": {}
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected no_action
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.

tool-holdout-09 · PASS

boundary, reference

Input and reference
A-104 appears in a sample screenshot.

Reference

{
  "tool": "no_action",
  "arguments": {}
}

Before

{
  "tool": "lookup_order",
  "arguments": {
    "order_id": "A-104"
  }
}

Output

{
  "tool": "no_action",
  "arguments": {}
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected no_action
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.

tool-holdout-10 · PASS

ticket, multilingual

Input and reference
File a ticket: Mi teclado no responde.

Reference

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Hardware",
    "summary": "Mi teclado no responde."
  }
}

Before

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Other",
    "summary": "Mi teclado no responde."
  }
}

Output

{
  "tool": "create_ticket",
  "arguments": {
    "category": "Hardware",
    "summary": "Mi teclado no responde."
  }
}
Grader checks
PASS call_schema: The simulated tool accepted the call.
PASS tool: Expected create_ticket
PASS arguments: All argument values must match the reference.
PASS outcome: Compare simulated side effects or lookup results.
Data attribution and license
Bundled cases are synthetic teaching data under MIT-0. Custom cases may have different provenance.