Practical Eval Lab · saved experiment

classification · baseline → improved

7/10 passed (70.0%) · change +30.0% · 0 regressions

holdout · 2026-10-01T20:51:14.670877+00:00 · 1 trial(s)

overall: FAIL

Teaching data; results describe these cases only. Candidate kind: local_rules. Imported reports are user-supplied evidence, not authenticated measurements. Review inputs and outputs before sharing.

Measurements and reproducibility
{
  "checks": {
    "label": {
      "passed": 7,
      "total": 10,
      "rate": 0.7
    }
  },
  "measurements": {},
  "candidate_latency_ms": {
    "median": 0.003,
    "p95": 0.005
  },
  "usage": {},
  "usage_coverage": 0,
  "trial_scores": [
    0.7
  ],
  "unstable_cases": 0,
  "confusion_matrix": {
    "Hardware": {
      "Hardware": 2,
      "Software": 0,
      "Other": 2,
      "invalid": 0
    },
    "Software": {
      "Hardware": 0,
      "Software": 2,
      "Other": 1,
      "invalid": 0
    },
    "Other": {
      "Hardware": 0,
      "Software": 0,
      "Other": 3,
      "invalid": 0
    }
  },
  "macro_f1": 0.7111111111111111
}

Dataset 6a702bd60d06f05daf8d1e6495d7b106eeb0e5606b045c35fdac7f2577577fa8
Grader 4980bf42bfe64d2b690f45b99d749c0d5c7b83de785069758b2846a3f8450ff2

{
  "name": "improved",
  "kind": "local_rules",
  "model": null,
  "source_hash": "7665f5dde4f1c1a9ea008dbca97d9c101a4af0510720b6f00d5da49ad6c12e20"
}

Case results

class-holdout-01 · PASS

hardware, typical

Input and reference
My headset cable is visibly broken.

Reference

Hardware

Before

Hardware

Output

Hardware
Grader checks
PASS label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-02 · PASS

software, typical

Input and reference
The browser freezes on every page.

Reference

Software

Before

Software

Output

Software
Grader checks
PASS label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-03 · PASS

other, business-rule

Input and reference
Please procure a monitor for the new hire.

Reference

Other

Before

Hardware

Output

Other
Grader checks
PASS label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-04 · PASS

hardware, multilingual

Input and reference
La pantalla está rota.

Reference

Hardware

Before

Other

Output

Hardware
Grader checks
PASS label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-05 · FAIL

hardware, vocabulary

Input and reference
My headphones only play through one ear.

Reference

Hardware

Before

Other

Output

Other
Grader checks
FAIL label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-06 · FAIL

software, vocabulary

Input and reference
The operating system cannot boot.

Reference

Software

Before

Other

Output

Other
Grader checks
FAIL label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-07 · PASS

other, typical

Input and reference
What is the visitor parking policy?

Reference

Other

Before

Other

Output

Other
Grader checks
PASS label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-08 · PASS

software, account

Input and reference
The VPN cannot authenticate my account.

Reference

Software

Before

Software

Output

Software
Grader checks
PASS label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-09 · PASS

other, business-rule

Input and reference
I want to donate my old laptop.

Reference

Other

Before

Hardware

Output

Other
Grader checks
PASS label: Compare the label; normalization removes outer whitespace and ignores case.

class-holdout-10 · FAIL

hardware, multilingual

Input and reference
Le clavier ne fonctionne plus.

Reference

Hardware

Before

Other

Output

Other
Grader checks
FAIL label: Compare the label; normalization removes outer whitespace and ignores case.
Data attribution and license
Bundled cases are synthetic teaching data under MIT-0. Custom cases may have different provenance.