Spec: Benchmark Scorecard v1

A Benchmark Scorecard is a benchmark result treated as an artifact of the same kind as a receipt: typed, validated, and honest about its own gaps. Its design rule is the one that matters: a metric that could not be measured is recorded as a gap with a reason — it is never silently omitted, and it never averages into the totals as a zero or a pass.

The benchmark is not live yet

This schema is published ahead of the public benchmark itself, so that scorecards are checkable artifacts from their first release. Until scorecards are actually published, nothing on this page is a claim about Mildport’s measured quality.

Schema: benchmark-scorecard.v1.schema.json · current version: scorecardVersion: 1

The document

Field	Semantics
scorecardVersion	Always 1 for this revision.
benchmark	Always openimportbench.
corpusVersion	Which corpus revision was run — scores are only comparable within a corpus version.
importer	The scored implementation: name and version marker. Nothing restricts this to Mildport.
generatedAt	ISO 8601 timestamp of the run.
caseResults	One entry per corpus case — see below.
tracks	Per-track rollups: case count, how many actually scored, aggregated metrics.
overall	The cross-track aggregate of the same metrics.

Cases, tracks, and gaps

Each case belongs to one track — crm-csv, invoice, scan-ocr, relational-workbook, or scale — and reports either:

• status: "scored", with metrics and optionally the raw counts (expected / assigned / correct) backing the rates, or
• status: "gap", with a gapReason saying why it could not run.

Metrics

All metrics are rates in 0..1 — or null, which means “not measurable in this run” and is a recorded gap, never a zero:

Metric	What it measures
autoMapRate	Share of expected columns mapped without a human answer.
mappingPrecision	Of the columns assigned, how many were right.
mappingRecall	Of the columns that should have been assigned, how many were.
extractionFieldAccuracy	Field-level accuracy of document extraction.
abstentionPrecision	When the importer said “I don’t know”, how often that was the right call.

abstentionPrecision is the culturally load-bearing one: it makes refusing to guess a scored, first-class behavior instead of a failure.

Signing

A published scorecard is expected to travel signed, using the same envelope discipline as receipts (detached JWS over canonical JSON). The v1 schema specifies the scorecard document itself; the signing wrapper is identical to the receipt’s and adds no scorecard-specific fields.

---

This specification page and the linked schema are licensed CC BY 4.0 — see the spec overview for scope.