October Model Performance: Separate the Scorecards

What qualifies for this scorecard

The frozen October individual-release cohort contains 313 products with timestamp-eligible final pre-close research outputs. The analysis retains 7,607 outputs across 26 recorded labels and 19 recorded family groups.

Four cards did not yet have an outcome and two were held for evidence review, leaving 307 cards in the main outcome denominator. Unknown and held outcomes remain excluded rather than being treated as zero.

For each product and recorded label, the calculation keeps the latest eligible final pre-close output. It does not search all historical snapshots for whichever estimate landed nearest the result.

The missing operational denominator

None of the 7,607 retained outputs carries a prospective operational selection. All are marked research only. That means this cohort cannot answer how a single model chosen before close performed in live operation.

Missing selection evidence is not zero-percent accuracy, and it cannot be repaired by naming the winning model after seeing the answer. A prospective score would require a fixed policy, its effective date, eligible releases, and the selected output preserved before outcomes.

Each model needs its own denominator

Model 07 posted the lowest observed median absolute percentage error at 29.3% across 307 analyzable outcomes. Model 12 posted the largest observed within-20 rate at 114 of 307, or 37.1%. Those measures reward different parts of the error distribution, so neither should be read as a forecast guarantee.

Most labels cover the full main denominator, but Model 10 and Model 11 each cover 80 outcomes while Model 32 covers 242. A percentage from a narrower subset is not directly comparable with a full-cohort score unless the sample size stays attached.

Closest-of-many coverage is a different score

At least one recorded label fell within 20% of the final result on 275 of 307 analyzable cards, or 89.6%. At least one landed within 10% on 240 cards, or 78.2%.

That is retrospective closest-of-many coverage identified after outcomes. A release with more eligible labels has more opportunities to produce a close answer, and family labels do not prove statistically independent attempts. This coverage must not be advertised as the prospective accuracy of one chosen forecast.

Category leaders changed with the sport

The observed category leaders differed across the largest sport cohorts. These are descriptive results on October's retained data, not rules for selecting a future model.

Small programs should be read from their raw rows instead of being promoted into a leaderboard. Sample size matters as much as the percentage beside it.

The misses stay in the record

Even the closest recorded model missed Nicolò Tresoldi UCL Card 20 by 64.6% against the recorded print run of 1,078. That result survives the evidence quarantine and shows why a selection of attractive success stories would misrepresent the cohort.

The useful habit is to inspect a model's own sample and tail errors, then compare the eventual official outcome. A closest-model badge is a retrospective summary, not a profitable policy, an order-size recommendation, or a promise that a future release will be close.

Methods and limits

Absolute percentage error measures distance from the recorded outcome relative to that outcome. The median describes the middle error, the mean is more sensitive to large misses, and the 90th percentile exposes the tail.

The comparator is a fixed sport median drawn from finalized outcomes visible in a September 30 export. It is deliberately simple and separate from PrintRun's production models.

Stored timestamps establish chronological eligibility inside the retained research data. They do not independently attest that every row was immutable at the time or prove the training data available then. Pending outcomes, held identities, differing model cohorts, and correlated model families remain limitations.