MetroLLM-Bench

Motivation

Running AI on infrastructure an organization owns is only viable if the model is small enough to own and capable enough to deploy. Whether that point has been reached is an empirical question rather than a matter of opinion, and it is the question this benchmark was built to answer.

A public-transit kiosk is a deliberate choice of testbed. Its output must be correct, since a wrong fare is a billing error. It runs under real constraints, often without a network, and it cannot send passenger data to someone else's servers. A model that handles that task well needs no network and no outside service to do it.

Benchmark design

The benchmark treats a language model as the decision-making runtime for a public-transit ticket kiosk. The model reads a plain description of how the kiosk should behave, then handles each request by calling tools for routing, fares, and live disruptions. Every answer is checked against deterministic ground truth, so a score reflects what the model got right rather than how plausible it sounded.

The cases span six real metro systems, from MARTA in Atlanta with 38 stations to the Beijing Subway with 414, and three distinct fare models. They reach well beyond simple lookups, into multi-turn dialogue, accessibility and policy questions, live disruptions, and adversarial inputs constructed to induce failure.

A scripted kiosk encodes its rules in code, so a station closure, a holiday schedule, or a new weather advisory usually means a code change and a release before the kiosk reflects it. A language model reads the same instruction as text, so an operator can describe the exception in plain language and the model applies it. The benchmark includes disruption cases of this kind across the six systems, among them a Taipei typhoon warning that suspends a line for high winds, a Doha sandstorm advisory, a San Francisco earthquake that suspends the Transbay Tube, and a Chicago polar vortex that escalates from cold-weather delays to a full closure. In each, the advisory is plain operator prose, and the model must interpret it and respond correctly, where a scripted kiosk would need the contingency written into its code in advance.

Cases

955

Metro systems

6

Capability categories

11

Scoring components

22

Tools

6

Train / held-out split

717 / 238

Results

A four billion parameter model we fine-tuned, shipped as a single 2.6 GB file, runs offline on a laptop and matches a frontier hosted model on this task. On the held-out partition it scored 91.32 on the core Tier-1 metric, against 91.37 for Azure GPT-5.4 at its highest reasoning effort. At standard effort it scored 2.15 points higher.

Nineteen models from four families were evaluated. Six of them fall within 1.5 points on the composite score, and the 2.6 GB student sits inside that group, alongside models several times its size and a frontier proprietary API.

Held-out leaderboard, 238 cases, ranked by composite score. The fine-tuned 4B student is highlighted. Two Gemma 4 edge variants were evaluated but excluded for exhausting the tool-call budget. Full table, licences, and vendors are in the paper.

The claim is stated precisely. On the core Tier-1 metric the student matches GPT-5.4 at maximum reasoning effort and trails it by roughly one point on the composite. We read this as parity at the deployment frontier of small, self-hosted models, not as surpassing a frontier API.

The order at the very top of the table is within measurement noise, where sub-point differences carry no statistical weight, so a higher-placed open model is not the generally stronger one. The task is demanding but bounded. It rewards correct tool use and a strict output contract rather than the open-ended reasoning of a mathematics olympiad, and a task of that shape has a ceiling that several capable models reach. GPT-5.4 keeps its edge where reasoning is the work, leading the temporal category by more than thirteen points. The relevant finding is that a 2.6 GB model reaches the same ceiling on this task.

Capability also ceases to track size earlier than expected. The fine-tuned 4B, 9B, and 27B students fall within 0.4 Tier-1 points of one another, so the 2.6 GB model concedes almost nothing to the 16 GB one on this task.

A model this size runs on consumer hardware, not datacenter accelerators. Memory at decode is the binding constraint, and the fine-tuned students fit on a modern laptop or a single consumer GPU. Decode speed is bandwidth-bound, so it scales with the device. On a fanless M2 Air the 2B student sustains about 39 tokens per second, enough to reproduce the results locally, while the 9B student runs at around 190 tokens per second single-stream on an RTX 5090, comfortably fast for a live deployment.

A second result is less expected. Beyond a certain scale, fine-tuning degraded the model. The gain over each base model narrowed as the base grew, from more than five Tier-1 points at 2B to a measured loss at 27B, with both training seeds agreeing on direction at every size.

Fine-tuning gain over each base model, held-out Tier-1, mean of two training seeds. The benefit shrinks as the base grows and turns negative at 27B.

The full 955-case run, which carries the statistical power the 238-case held-out partition lacks, certifies both ends, a +1.72 Tier-1 gain for the 4B student and a −1.09 loss for the 27B. A rule-based baseline reaches 84.6 Tier-1 and locates the model's real advantage in temporal reasoning, policy adaptation, and disruption handling, the categories a fixed script cannot cover. The held-out confidence intervals, and which differences they do and do not certify, are reported in full in the paper.

Availability

The benchmark, the fine-tuned weights, and the evaluation harness are released under a permissive licence, so every figure above can be reproduced. The accompanying paper is under review, and its arXiv link will be added here once it is live.

Outlook

Models that handle work like this keep getting smaller, while the inference hardware in everyday devices keeps getting faster, with dedicated accelerators and NPUs now standard in consumer SoCs. On capable consumer hardware a model of this size already runs fast enough for live use, and each hardware generation raises that floor. Most operational software is built from bounded, well-specified tasks of this kind, and for those the model can run on hardware the operator owns, without a remote frontier API. The same applies to point-of-sale terminals, industrial controllers, and back-office workflows. Frontier models stay necessary where the problem is genuinely open-ended.