🔍 Read the full analysis: Testing AI On My News Pipeline Revealed A Gap Between Best And Practical on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Thorsten Meyer reports testing 71 model configurations on news-writing tasks from September 8 to October 13, 2026. GPT-6 Astra scored highest, but GPT-6 Luna produced usable articles at roughly one-hundredth of the price; two fallback models that fabricated quotes or failed output checks were removed from the pipeline.
Thorsten Meyer says a test of 71 AI model configurations for news writing found that GPT-6 Astra scored highest, while GPT-6 Luna offered a much cheaper option for routine production. Meyer also removed two backup models from his pipeline after his evaluation found they invented quotations or failed to produce usable output.
In a report published October 13 on ThorstenMeyerAI.com, Meyer described testing models on the same prompt and news sources used by his production pipeline. The test covered English and German articles, using one longer source and one short source. Two AI judges scored articles for source fidelity, fabrication, clarity about uncertainty, news value, structure and polish; German-language output was also judged for language quality.
Meyer reported that GPT-6 Astra scored 4.00 out of 5 in both languages in the standard test, with a source-fidelity score of 5.00. In a 12-article robustness run, it scored 3.93. GPT-6 Luna scored 3.67 in English and 3.68 in German in the standard test, and 3.67 in the robustness run, with source fidelity of 4.71. Meyer said all of Luna’s articles were valid under the pipeline’s output checks.
The cost difference shaped the production choice: Meyer estimated Astra at about $0.12 per article and Luna at roughly $0.0012, with reported generation times of about 50 seconds and 17 seconds, respectively. His tests of six reasoning-effort settings for Luna found no reliable quality gain from higher settings; he said the maximum setting cost and took about 12 times as much as the no-effort setting.
Infographic 01 · Who writes the best news article?
The best writer is not the one in production
71 model configurations, one real job: turn a news source into an article for an English and a German edition. GPT-6 Astra writes the best articles. GPT-6 Luna writes them for about a hundredth of the price.
Robustness run: 12 real news sources, English edition
Dot = mean score (1–5), bar = 95% bootstrap interval. German edition mean below each score. Overlapping bars are not a real difference.
Two judges, two caveats: GPT-6.1 Sol and Claude Opus 5.5 grade every article and were also tested as writers, so they partly grade themselves. Claude models ran with effort medium unless noted.
Source: Thorsten Meyer AI model-scout benchmark, 8 Sep – 13 Oct 2026. Task: write a news article from a real source with the production prompt, English and German editions. Judges: GPT-6.1 Sol (reasoning high) and Claude Opus 5.5, both with the source in context; score 1–5, mean of both judges. Robustness run: 2 test articles plus 10 more real sources, mostly short like production; mean with 95% bootstrap interval.
Powered by Thorsten Meyer AI
Quality Versus Cost in Production
The results highlight a practical distinction for publishers using generative AI: the highest-scoring model may not be the best fit for a high-volume workflow. Meyer’s reported price estimate puts Astra at about 100 times Luna’s cost, while the robustness scores suggest Luna remained reasonably consistent across the tested articles. That makes the question less about naming a universal “best” model and more about whether a lower-cost model meets a publisher’s own quality threshold.
The fallback findings also show why evaluating only the primary model can miss a risk. Meyer said one backup generated invented quotations and another produced no valid articles in four attempts. In a news pipeline, fabricated attributions can mislead readers and damage trust. His account suggests that testing the complete route—including what happens when the main model is unavailable—matters alongside model rankings.
These conclusions are specific to Meyer’s sources, prompt, judges and production requirements. They do not establish that Luna or Astra will perform the same way for other publishers or story types. The report does, however, provide a concrete example of how task-specific testing can change a deployment decision.
As an affiliate, we earn on qualifying purchases.
How Meyer Tested the Models
Meyer said the evaluation ran from September 8 to October 13, 2026, and included API, subscription, free-tier and locally run open-weight models. The sample included two sources: a 6,340-character product filing and a 619-character political news item. He described the shorter source as more representative of the material his pipeline often handles.
Each source was turned into English and German articles using the production prompt. GPT-6.1 Sol and Claude Opus 5.5 served as judges, and both received the original source. Meyer also ran a robustness test in which stronger configurations generated 10 additional articles, mostly from short sources. He reported 95% bootstrap intervals and said that, with only two articles in the standard test, score gaps below about 0.3 points should be treated as noise. He regarded the larger robustness run as the more dependable ranking.
For an article to count, it had to produce valid output after the same repair step used in production. Meyer said he kept GPT-6 Luna in the production route and decided on October 7 that Astra was too expensive for the task. The test was a report on one publisher’s workflow, not an independently replicated comparison across newsrooms.
“Public leaderboards tell you which model is smartest. They don’t tell you which model should write your articles.”
— Thorsten Meyer, in his October 13 report
Limits of the Test Results
The findings are based on one publisher’s prompt, two initial sources and two AI judges. Meyer’s robustness run added 10 articles, but the report does not describe a large, independently run evaluation across publishers, topics, languages or changing news conditions. It is not clear whether the same rankings would hold with different prompts, more source types or human editors as evaluators.
The reported scores and prices are Meyer’s test results and estimates, not independently verified measurements. The report also does not establish whether the judges’ assessments match reader judgments or how often human review would catch errors in ordinary production. The specific examples of invented quotations are attributed to Meyer’s evaluation; the available account does not establish how frequently such failures would occur outside that test.
Luna Remains in the News Route
Meyer said GPT-6 Luna is the model now used for routine news production, while the two tested fallbacks have been removed. His report does not identify a replacement backup model or describe how the pipeline will handle Luna outages, so the current failover plan is not clear.
The next useful step would be continued monitoring on a wider range of real sources, with repeated checks for fabricated details, invalid output and changes in cost or latency. Meyer’s account provides no announced date for another benchmark or planned expansion of the test.
Key Questions
Which model scored highest in Meyer’s test?
Meyer reported that GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It scored 3.93 across 12 articles in the robustness run.
Why did Meyer choose GPT-6 Luna for production?
Meyer said Luna scored 3.67 in the robustness run and produced valid articles in the test. He estimated it cost about $0.0012 per article, compared with roughly $0.12 for Astra.
What happened when he increased reasoning effort?
Across six settings, Meyer found no reliable quality improvement. He reported that the maximum setting cost and took about 12 times as much as the no-effort setting.
Why were two backup models removed?
Meyer said GPT-4.1 nano received low source-fidelity scores and generated invented quotations, while locally run gpt-oss-120b produced zero valid articles out of four attempts. He said both were removed from the news route on October 7.
Do these results show which model is best for every newsroom?
No. The findings apply to Meyer’s prompt, sources, judges and pipeline checks. His report does not show that the same rankings would hold for other publishers or tasks.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
