OpenAI's email landed with a deadline: GPT-4o, gpt-4o-mini, GPT-4.1 and GPT-4.1 mini will retire on January 14, 2027. We had to start making a choice immediately. We also realised we had not revisited our evaluations and benchmarks in about a month, so this was a good moment to run them again against what is available now. Earlier rounds had not found a replacement that matched gpt-4o-mini on both the cost and accuracy we needed.
We focused on calendar-event and party-package extraction. The question was practical: which smaller model can preserve the fields local listings need without making every page slow or expensive? This is a benchmark of a few dozen real pages, with repeated runs for the main event comparison. It is not a general test of web extraction.
Method
We used two stages. First, we screened many candidates and effort levels on a diverse sample of pages that we chose and hand-labelled, with about 15 pages per task and one run. That gave us a quick read on cost, speed, and quality. Second, the finalists that passed those gates went through our proprietary in-house benchmark: real local pages that we hand-labelled and maintain so we can re-run them when models change. We built it while running TinyJawns, our family events site. Parents kept reporting inaccuracies and flagging listings, so we turned those reports into test cases. It contains the awkward cases we have collected over years of extracting messy local data, including dates without a year, recurring schedules stated in a section heading, dense pages with many similar items, date ranges, and pricing that is “inquire only” or varies by package.
The second stage used 32 pages and 87 events, with repeat runs. We tracked recall, precision, field accuracy, typical latency, and relative cost. Recall is how many expected items were found. Precision is a flooding indicator for this evaluation: it tells us how many returned items were acceptable here, not a universal measure of false listings. Field accuracy is, for the events a model found, the share of labelled details it got right: dates and times, whether and how the event repeats, price, ages, and location. Text fields count as right when they substantially match; dates and numbers must match exactly. Because it only covers events the model found, we read it alongside recall.
Results: gpt-4o-mini and the finalists
Events: full evaluation
| Model and setup | Recall | Precision | Field accuracy | Latency | Relative cost |
|---|---|---|---|---|---|
gpt-4o-mini | 92% | 52% | 92.2% | 11.6 s p50 | 1.00x reference |
gpt-6-luna, low reasoning | 95% | 52–53% | 92.0–92.2% | 12.6–14.0 s p50 | about 0.72x vs cold gpt-4o-mini |
gpt-6-luna, low reasoning, targeted date/recurrence check | 95% | 52–53% | 93.3–93.7% | about 14.5–16 s p50 (estimated) | about 0.77–0.85x |
The full event evaluation gives us the clearest comparison. Low-reasoning GPT-6 Luna (gpt-6-luna) found more expected events than gpt-4o-mini, kept precision in the same range, and landed at similar field accuracy. Its relative cost was about 0.72x the cold gpt-4o-mini reference. A targeted second look at dates and recurrence lifted field accuracy to 93.3–93.7%, with estimated latency of about 14.5–16 s p50 and relative cost of about 0.77–0.85x.
Events: directional effort sweep
| Model and setup | Recall | Precision | Field accuracy | Latency | Relative cost |
|---|---|---|---|---|---|
gpt-5-nano, minimal | 58% | 36% | 71% | 1.2 s p50 | 0.30x |
gpt-5-nano, low | 100% | 51% | 85% | 10 s p50 | 0.62x |
gpt-5-nano, medium/default-like | 100% | 46% | 91% | 79 s p50 | 2.04x |
This sweep made the main trade-off hard to miss. Minimal gpt-5-nano was fast and inexpensive, but recall fell to 58% and field accuracy to 71%. Low effort recovered recall to 100%, with 85% field accuracy and 10 s p50 latency. Medium effort kept recall at 100% and raised field accuracy to 91%, but latency jumped to 79 s p50 and relative cost to 2.04x. Dates and recurrence were the weak point in the cheaper settings, which is why a gated second look is more useful to us than applying high reasoning effort to every extraction.
Party packages: directional sample
| Model and setup | Recall | Precision | Field accuracy | Latency | Relative cost |
|---|---|---|---|---|---|
gpt-5-nano, low | 94% | 100% | 83% | 8.1 s p50 | 0.63x |
gpt-5-nano, medium | 100% | 100% | 94% | 40.2 s p50 | 2.12x |
gpt-6-luna, low | 100% | 100% | 98% | 6.5 s p50 | about 0.97x per call |
Party packages showed a cleaner directional result. Low-reasoning gpt-6-luna reached 100% recall and precision, 98% field accuracy, and 6.5 s p50 latency. Medium gpt-5-nano also reached 100% recall and precision, but at 40.2 s p50 and 2.12x relative cost. These samples are useful for deciding what to test next, but they are too small to establish a universal ranking.
Open-weight candidates offered a different trade-off. DeepSeek V4.1 Flash reached 100% event recall, 59% precision, and 89% field accuracy at about 0.63x relative cost, but had a 15 s page p50 and a long shared-host tail. Gemma 4 was interesting as a way to sample and offload some cost, but it added latency. In one Gemini API probe, Gemma 4 took 24–51 s per page versus about 1 s for Gemini 3.1 Flash-Lite; Gemma 4 was free-tier-only there. We pay for compute time while a page is in flight, so the extra seconds per page cost more than the token savings as we scale. We did not adopt it.
Findings
The results point to reasoning effort as the largest cost and quality lever in this comparison. They also reinforce why the in-house benchmark matters: it deliberately puts messy local-page patterns in front of the models, so a model that looks fine on a generic extraction test can lose accuracy here.
Repeat runs showed that recurrence fields can vary more than core extraction fields. The targeted date and recurrence check helped in the full event evaluation, but improvements tuned on an initial subset did not clearly reproduce on held-out pages. That is a useful warning against treating one promising adjustment as settled evidence.
The main event results are stronger than the directional samples. We should not treat the sample differences as definitive, especially when the same dense listing pages produced extras across models. That page-driven flooding is an evaluation caveat, not a claim about live output quality. Latency tails also depend on page shape and shared-host conditions, so p50 is more comparable than p95. There is still work to do, but at least the retirement email turned a stale benchmark into a current decision.