Teaching data; results describe these cases only. Candidate kind: local_rules.
Imported reports are user-supplied evidence, not authenticated measurements. Review inputs and outputs before sharing.
Measurements and reproducibility {
"checks": {
"json_object": {
"passed": 10,
"total": 10,
"rate": 1.0
},
"schema": {
"passed": 10,
"total": 10,
"rate": 1.0
},
"field:order_id": {
"passed": 10,
"total": 10,
"rate": 1.0
},
"field:quantity": {
"passed": 8,
"total": 10,
"rate": 0.8
},
"field:item": {
"passed": 8,
"total": 10,
"rate": 0.8
},
"field:priority": {
"passed": 10,
"total": 10,
"rate": 1.0
}
},
"measurements": {},
"candidate_latency_ms": {
"median": 0.005,
"p95": 0.012
},
"usage": {},
"usage_coverage": 0,
"trial_scores": [
0.8
],
"unstable_cases": 0
}
Dataset 6d0b67864847a451dbbac3bf6491035a711177f22883b6f77d54c2722cb61c037f4a79e5cafae9bdeef6d464191795315b45e142b4ddfb450f758a0b919e8d4c
{
"name": "improved",
"kind": "local_rules",
"model": null,
"source_hash": "7665f5dde4f1c1a9ea008dbca97d9c101a4af0510720b6f00d5da49ad6c12e20"
} Case results extract-holdout-01 · PASS
typical
Input and reference Order S-922: 6 monitors. Reference {
"order_id": "S-922",
"quantity": 6,
"item": "monitor",
"priority": "normal"
} Before {
"order_id": "S-922",
"quantity": 6,
"item": "monitor",
"priority": "normal"
} Output {
"order_id": "S-922",
"quantity": 6,
"item": "monitor",
"priority": "normal"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "S-922"
PASS field:quantity: Expected quantity = 6
PASS field:item: Expected item = "monitor"
PASS field:priority: Expected priority = "normal" extract-holdout-02 · PASS
word-number, priority
Input and reference Order T-023: four webcams, no rush. Reference {
"order_id": "T-023",
"quantity": 4,
"item": "webcam",
"priority": "low"
} Before {
"order_id": "T-023",
"quantity": null,
"item": null,
"priority": "normal"
} Output {
"order_id": "T-023",
"quantity": 4,
"item": "webcam",
"priority": "low"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "T-023"
PASS field:quantity: Expected quantity = 4
PASS field:item: Expected item = "webcam"
PASS field:priority: Expected priority = "low" extract-holdout-03 · FAIL
word-number, plural
Input and reference Order U-124: six mice. Reference {
"order_id": "U-124",
"quantity": 6,
"item": "mouse",
"priority": "normal"
} Before {
"order_id": "U-124",
"quantity": null,
"item": null,
"priority": "normal"
} Output {
"order_id": "U-124",
"quantity": null,
"item": null,
"priority": "normal"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "U-124"
FAIL field:quantity: Expected quantity = 6
FAIL field:item: Expected item = "mouse"
PASS field:priority: Expected priority = "normal" extract-holdout-04 · PASS
case, priority
Input and reference Order v-225: 8 CABLES, urgent. Reference {
"order_id": "V-225",
"quantity": 8,
"item": "cable",
"priority": "high"
} Before {
"order_id": null,
"quantity": null,
"item": null,
"priority": "high"
} Output {
"order_id": "V-225",
"quantity": 8,
"item": "cable",
"priority": "high"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "V-225"
PASS field:quantity: Expected quantity = 8
PASS field:item: Expected item = "cable"
PASS field:priority: Expected priority = "high" extract-holdout-05 · FAIL
vocabulary
Input and reference Order W-326: 2 printers. Reference {
"order_id": "W-326",
"quantity": 2,
"item": "printer",
"priority": "normal"
} Before {
"order_id": "W-326",
"quantity": null,
"item": null,
"priority": "normal"
} Output {
"order_id": "W-326",
"quantity": null,
"item": null,
"priority": "normal"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "W-326"
FAIL field:quantity: Expected quantity = 2
FAIL field:item: Expected item = "printer"
PASS field:priority: Expected priority = "normal" extract-holdout-06 · PASS
priority
Input and reference Order X-427: 3 keyboards, low priority. Reference {
"order_id": "X-427",
"quantity": 3,
"item": "keyboard",
"priority": "low"
} Before {
"order_id": "X-427",
"quantity": 3,
"item": "keyboard",
"priority": "normal"
} Output {
"order_id": "X-427",
"quantity": 3,
"item": "keyboard",
"priority": "low"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "X-427"
PASS field:quantity: Expected quantity = 3
PASS field:item: Expected item = "keyboard"
PASS field:priority: Expected priority = "low" extract-holdout-07 · PASS
negation, priority
Input and reference Order Y-528: 1 headset, not urgent. Reference {
"order_id": "Y-528",
"quantity": 1,
"item": "headset",
"priority": "low"
} Before {
"order_id": "Y-528",
"quantity": 1,
"item": "headset",
"priority": "high"
} Output {
"order_id": "Y-528",
"quantity": 1,
"item": "headset",
"priority": "low"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "Y-528"
PASS field:quantity: Expected quantity = 1
PASS field:item: Expected item = "headset"
PASS field:priority: Expected priority = "low" extract-holdout-08 · PASS
missing, word-number
Input and reference Please send five monitors ASAP. Reference {
"order_id": null,
"quantity": 5,
"item": "monitor",
"priority": "high"
} Before {
"order_id": null,
"quantity": null,
"item": null,
"priority": "normal"
} Output {
"order_id": null,
"quantity": 5,
"item": "monitor",
"priority": "high"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = null
PASS field:quantity: Expected quantity = 5
PASS field:item: Expected item = "monitor"
PASS field:priority: Expected priority = "high" extract-holdout-09 · PASS
missing
Input and reference Order Z-629 was recorded. Reference {
"order_id": "Z-629",
"quantity": null,
"item": null,
"priority": "normal"
} Before {
"order_id": "Z-629",
"quantity": null,
"item": null,
"priority": "normal"
} Output {
"order_id": "Z-629",
"quantity": null,
"item": null,
"priority": "normal"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "Z-629"
PASS field:quantity: Expected quantity = null
PASS field:item: Expected item = null
PASS field:priority: Expected priority = "normal" extract-holdout-10 · PASS
typical
Input and reference Order A-730: 7 webcams, normal priority. Reference {
"order_id": "A-730",
"quantity": 7,
"item": "webcam",
"priority": "normal"
} Before {
"order_id": "A-730",
"quantity": 7,
"item": "webcam",
"priority": "normal"
} Output {
"order_id": "A-730",
"quantity": 7,
"item": "webcam",
"priority": "normal"
} Grader checks PASS json_object: Output must be a JSON object.
PASS schema: Check required fields and types; a boolean is not an integer.
PASS field:order_id: Expected order_id = "A-730"
PASS field:quantity: Expected quantity = 7
PASS field:item: Expected item = "webcam"
PASS field:priority: Expected priority = "normal" Data attribution and license Bundled cases are synthetic teaching data under MIT-0. Custom cases may have different provenance. Standalone report · no scripts, external assets, or model calls