Practical Eval Lab · saved experiment

extraction · baseline → improved

20/20 passed (100.0%) · change +50.0% · 0 regressions

dev · 2026-10-01T20:51:14.676183+00:00 · 1 trial(s)

overall: PASS

Teaching data; results describe these cases only. Candidate kind: local_rules. Imported reports are user-supplied evidence, not authenticated measurements. Review inputs and outputs before sharing.

Measurements and reproducibility
{
  "checks": {
    "json_object": {
      "passed": 20,
      "total": 20,
      "rate": 1.0
    },
    "schema": {
      "passed": 20,
      "total": 20,
      "rate": 1.0
    },
    "field:order_id": {
      "passed": 20,
      "total": 20,
      "rate": 1.0
    },
    "field:quantity": {
      "passed": 20,
      "total": 20,
      "rate": 1.0
    },
    "field:item": {
      "passed": 20,
      "total": 20,
      "rate": 1.0
    },
    "field:priority": {
      "passed": 20,
      "total": 20,
      "rate": 1.0
    }
  },
  "measurements": {},
  "candidate_latency_ms": {
    "median": 0.004,
    "p95": 0.008
  },
  "usage": {},
  "usage_coverage": 0,
  "trial_scores": [
    1
  ],
  "unstable_cases": 0
}

Dataset da1d0001abfa4a1785184b15be60e4fc42227742c719d413fb59bbba3f8f7fbb
Grader 7f4a79e5cafae9bdeef6d464191795315b45e142b4ddfb450f758a0b919e8d4c

{
  "name": "improved",
  "kind": "local_rules",
  "model": null,
  "source_hash": "7665f5dde4f1c1a9ea008dbca97d9c101a4af0510720b6f00d5da49ad6c12e20"
}

Case results

extract-dev-01 · PASS

typical

Input and reference
Order A-104: 3 keyboards.

Reference

{
  "order_id": "A-104",
  "quantity": 3,
  "item": "keyboard",
  "priority": "normal"
}

Before

{
  "order_id": "A-104",
  "quantity": 3,
  "item": "keyboard",
  "priority": "normal"
}

Output

{
  "order_id": "A-104",
  "quantity": 3,
  "item": "keyboard",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "A-104"
PASS field:quantity: Expected quantity = 3
PASS field:item: Expected item = "keyboard"
PASS field:priority: Expected priority = "normal"

extract-dev-02 · PASS

typical, priority

Input and reference
Order B-205: 2 monitors, high priority.

Reference

{
  "order_id": "B-205",
  "quantity": 2,
  "item": "monitor",
  "priority": "high"
}

Before

{
  "order_id": "B-205",
  "quantity": 2,
  "item": "monitor",
  "priority": "high"
}

Output

{
  "order_id": "B-205",
  "quantity": 2,
  "item": "monitor",
  "priority": "high"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "B-205"
PASS field:quantity: Expected quantity = 2
PASS field:item: Expected item = "monitor"
PASS field:priority: Expected priority = "high"

extract-dev-03 · PASS

typical

Input and reference
Order C-306: 1 mouse.

Reference

{
  "order_id": "C-306",
  "quantity": 1,
  "item": "mouse",
  "priority": "normal"
}

Before

{
  "order_id": "C-306",
  "quantity": 1,
  "item": "mouse",
  "priority": "normal"
}

Output

{
  "order_id": "C-306",
  "quantity": 1,
  "item": "mouse",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "C-306"
PASS field:quantity: Expected quantity = 1
PASS field:item: Expected item = "mouse"
PASS field:priority: Expected priority = "normal"

extract-dev-04 · PASS

typical

Input and reference
Order D-407: 5 cables.

Reference

{
  "order_id": "D-407",
  "quantity": 5,
  "item": "cable",
  "priority": "normal"
}

Before

{
  "order_id": "D-407",
  "quantity": 5,
  "item": "cable",
  "priority": "normal"
}

Output

{
  "order_id": "D-407",
  "quantity": 5,
  "item": "cable",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "D-407"
PASS field:quantity: Expected quantity = 5
PASS field:item: Expected item = "cable"
PASS field:priority: Expected priority = "normal"

extract-dev-05 · PASS

typical, priority

Input and reference
Order E-508: 4 headsets, urgent.

Reference

{
  "order_id": "E-508",
  "quantity": 4,
  "item": "headset",
  "priority": "high"
}

Before

{
  "order_id": "E-508",
  "quantity": 4,
  "item": "headset",
  "priority": "high"
}

Output

{
  "order_id": "E-508",
  "quantity": 4,
  "item": "headset",
  "priority": "high"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "E-508"
PASS field:quantity: Expected quantity = 4
PASS field:item: Expected item = "headset"
PASS field:priority: Expected priority = "high"

extract-dev-06 · PASS

typical

Input and reference
Order F-609: 2 webcams.

Reference

{
  "order_id": "F-609",
  "quantity": 2,
  "item": "webcam",
  "priority": "normal"
}

Before

{
  "order_id": "F-609",
  "quantity": 2,
  "item": "webcam",
  "priority": "normal"
}

Output

{
  "order_id": "F-609",
  "quantity": 2,
  "item": "webcam",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "F-609"
PASS field:quantity: Expected quantity = 2
PASS field:item: Expected item = "webcam"
PASS field:priority: Expected priority = "normal"

extract-dev-07 · PASS

word-number

Input and reference
Order G-710: three keyboards.

Reference

{
  "order_id": "G-710",
  "quantity": 3,
  "item": "keyboard",
  "priority": "normal"
}

Before

{
  "order_id": "G-710",
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Output

{
  "order_id": "G-710",
  "quantity": 3,
  "item": "keyboard",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "G-710"
PASS field:quantity: Expected quantity = 3
PASS field:item: Expected item = "keyboard"
PASS field:priority: Expected priority = "normal"

extract-dev-08 · PASS

word-number, plural

Input and reference
Order H-811: two mice.

Reference

{
  "order_id": "H-811",
  "quantity": 2,
  "item": "mouse",
  "priority": "normal"
}

Before

{
  "order_id": "H-811",
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Output

{
  "order_id": "H-811",
  "quantity": 2,
  "item": "mouse",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "H-811"
PASS field:quantity: Expected quantity = 2
PASS field:item: Expected item = "mouse"
PASS field:priority: Expected priority = "normal"

extract-dev-09 · PASS

case

Input and reference
Order i-912: 1 monitor.

Reference

{
  "order_id": "I-912",
  "quantity": 1,
  "item": "monitor",
  "priority": "normal"
}

Before

{
  "order_id": null,
  "quantity": 1,
  "item": "monitor",
  "priority": "normal"
}

Output

{
  "order_id": "I-912",
  "quantity": 1,
  "item": "monitor",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "I-912"
PASS field:quantity: Expected quantity = 1
PASS field:item: Expected item = "monitor"
PASS field:priority: Expected priority = "normal"

extract-dev-10 · PASS

case

Input and reference
Order J-013: 2 KEYBOARDS.

Reference

{
  "order_id": "J-013",
  "quantity": 2,
  "item": "keyboard",
  "priority": "normal"
}

Before

{
  "order_id": "J-013",
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Output

{
  "order_id": "J-013",
  "quantity": 2,
  "item": "keyboard",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "J-013"
PASS field:quantity: Expected quantity = 2
PASS field:item: Expected item = "keyboard"
PASS field:priority: Expected priority = "normal"

extract-dev-11 · PASS

priority, negation

Input and reference
Order K-114: 4 cables, no rush.

Reference

{
  "order_id": "K-114",
  "quantity": 4,
  "item": "cable",
  "priority": "low"
}

Before

{
  "order_id": "K-114",
  "quantity": 4,
  "item": "cable",
  "priority": "normal"
}

Output

{
  "order_id": "K-114",
  "quantity": 4,
  "item": "cable",
  "priority": "low"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "K-114"
PASS field:quantity: Expected quantity = 4
PASS field:item: Expected item = "cable"
PASS field:priority: Expected priority = "low"

extract-dev-12 · PASS

priority, negation

Input and reference
Order L-215: 2 headsets, not urgent.

Reference

{
  "order_id": "L-215",
  "quantity": 2,
  "item": "headset",
  "priority": "low"
}

Before

{
  "order_id": "L-215",
  "quantity": 2,
  "item": "headset",
  "priority": "high"
}

Output

{
  "order_id": "L-215",
  "quantity": 2,
  "item": "headset",
  "priority": "low"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "L-215"
PASS field:quantity: Expected quantity = 2
PASS field:item: Expected item = "headset"
PASS field:priority: Expected priority = "low"

extract-dev-13 · PASS

priority

Input and reference
Order M-316: 1 webcam, ASAP.

Reference

{
  "order_id": "M-316",
  "quantity": 1,
  "item": "webcam",
  "priority": "high"
}

Before

{
  "order_id": "M-316",
  "quantity": 1,
  "item": "webcam",
  "priority": "normal"
}

Output

{
  "order_id": "M-316",
  "quantity": 1,
  "item": "webcam",
  "priority": "high"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "M-316"
PASS field:quantity: Expected quantity = 1
PASS field:item: Expected item = "webcam"
PASS field:priority: Expected priority = "high"

extract-dev-14 · PASS

priority, plural

Input and reference
Order N-417: 3 mice, priority low.

Reference

{
  "order_id": "N-417",
  "quantity": 3,
  "item": "mouse",
  "priority": "low"
}

Before

{
  "order_id": "N-417",
  "quantity": 3,
  "item": "mouse",
  "priority": "normal"
}

Output

{
  "order_id": "N-417",
  "quantity": 3,
  "item": "mouse",
  "priority": "low"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "N-417"
PASS field:quantity: Expected quantity = 3
PASS field:item: Expected item = "mouse"
PASS field:priority: Expected priority = "low"

extract-dev-15 · PASS

word-number

Input and reference
Order O-518: one keyboard.

Reference

{
  "order_id": "O-518",
  "quantity": 1,
  "item": "keyboard",
  "priority": "normal"
}

Before

{
  "order_id": "O-518",
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Output

{
  "order_id": "O-518",
  "quantity": 1,
  "item": "keyboard",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "O-518"
PASS field:quantity: Expected quantity = 1
PASS field:item: Expected item = "keyboard"
PASS field:priority: Expected priority = "normal"

extract-dev-16 · PASS

missing

Input and reference
Please send 2 monitors.

Reference

{
  "order_id": null,
  "quantity": 2,
  "item": "monitor",
  "priority": "normal"
}

Before

{
  "order_id": null,
  "quantity": 2,
  "item": "monitor",
  "priority": "normal"
}

Output

{
  "order_id": null,
  "quantity": 2,
  "item": "monitor",
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = null
PASS field:quantity: Expected quantity = 2
PASS field:item: Expected item = "monitor"
PASS field:priority: Expected priority = "normal"

extract-dev-17 · PASS

missing

Input and reference
Order P-619 needs review.

Reference

{
  "order_id": "P-619",
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Before

{
  "order_id": "P-619",
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Output

{
  "order_id": "P-619",
  "quantity": null,
  "item": null,
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "P-619"
PASS field:quantity: Expected quantity = null
PASS field:item: Expected item = null
PASS field:priority: Expected priority = "normal"

extract-dev-18 · PASS

missing

Input and reference
No order has been placed.

Reference

{
  "order_id": null,
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Before

{
  "order_id": null,
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Output

{
  "order_id": null,
  "quantity": null,
  "item": null,
  "priority": "normal"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = null
PASS field:quantity: Expected quantity = null
PASS field:item: Expected item = null
PASS field:priority: Expected priority = "normal"

extract-dev-19 · PASS

word-number, priority

Input and reference
Order Q-720: five cables, low priority.

Reference

{
  "order_id": "Q-720",
  "quantity": 5,
  "item": "cable",
  "priority": "low"
}

Before

{
  "order_id": "Q-720",
  "quantity": null,
  "item": null,
  "priority": "normal"
}

Output

{
  "order_id": "Q-720",
  "quantity": 5,
  "item": "cable",
  "priority": "low"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "Q-720"
PASS field:quantity: Expected quantity = 5
PASS field:item: Expected item = "cable"
PASS field:priority: Expected priority = "low"

extract-dev-20 · PASS

typical, priority

Input and reference
Order R-821: 12 headsets, priority high.

Reference

{
  "order_id": "R-821",
  "quantity": 12,
  "item": "headset",
  "priority": "high"
}

Before

{
  "order_id": "R-821",
  "quantity": 12,
  "item": "headset",
  "priority": "high"
}

Output

{
  "order_id": "R-821",
  "quantity": 12,
  "item": "headset",
  "priority": "high"
}
Grader checks
PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "R-821"
PASS field:quantity: Expected quantity = 12
PASS field:item: Expected item = "headset"
PASS field:priority: Expected priority = "high"
Data attribution and license
Bundled cases are synthetic teaching data under MIT-0. Custom cases may have different provenance.