🔍 Read the full analysis: The AI Metrics Debate: What Amazon’s Bets Add To The Conversation on ThorstenMeyerAI.com
Get audio and creator gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Amazon published engineering guidance on Oct. 27 saying developers should be judged by customer, business and team impact, not by token use or other AI activity counts. The memo says those measures can still help identify patterns, but turning them into targets risks higher costs and complexity. Amazon’s Nova and Anthropic bets add a company-level question: how should AI investment be judged by results?
Amazon published new engineering guidance on Oct. 27 telling software teams to evaluate AI work by its impact on customers, the business and colleagues—not by token counts or how often employees use AI tools. The memo arrives after reports of workers using internal tools to boost leaderboard scores, and as Amazon’s own AI investments face different tests of whether they are producing value.
The memo, written by Terk Terkowitz, a vice president and distinguished engineer in Amazon’s stores business, introduces a revised role guideline for software engineers. It says measures such as token use, AI-tool adoption, code changes and reviews can be disconnected from the outcomes that matter. When treated as targets rather than indicators, they can increase costs and make systems more complex, the memo says.
The guidance does not say usage data is useless or that employees should avoid AI. It says those inputs currently correlate well with important outputs, and can help teams see where adoption is high or low. Amazon advises leaders to examine both ends: why some teams use AI little, and whether heavy users are delivering better work. Engineers whose tasks do not benefit from current tools should document that assessment, the memo says.
The memo also asks leaders to use AI on their own production work and identify tasks where it helped and where it did not. It says the company’s most senior engineers deliberately avoided writing a fixed AI definition into the role guideline, because technology is changing quickly. Instead, the document centers on lasting expectations around impact and judgment.
Six months at Amazon
From token leaderboards to an outcomes memo
Amazon spent 2026 learning the same lesson twice: once with how its engineers use AI, once with its own AI models.
- May 2026Token leaderboards backfireFinancial Times (via The Decoder): staff give an internal agent tool pointless tasks to inflate token counts. Reported, anonymous sources.
- Late May 2026KiroRank “deprecated”Amazon spokesperson to Business Insider. SVP Dave Treadwell: “Please don’t use AI just for the sake of using AI.”
- 24 Jul 2026San Francisco AGI Lab site closesConfirmed by Amazon to GeekWire. Frontier model research continues under Pieter Abbeel.
- 28 Jul 2026Most flagship Nova models wound downBusiness Insider, people familiar: Premier, Omni, Reel, Canvas to “keep the lights on”. Amazon: models “remain one of the most important things we’re working on”.
- 14 / 30 Sep 2026Nova Sonic v1, Nova Reel, Nova Canvas reach end of lifeAWS Bedrock model lifecycle page. Requests to them now fail.
- 26 Oct 2026Anthropic files its S-1Amazon notes: $6.2 billion principal, 12% ownership cap that Amazon can waive. Anthropic commits at least $110 billion to Amazon through April 2036.
- 27 Oct 2026The memo: outcomes, not tokenmaxxingUsage and throughput metrics are “deeply flawed” as targets. “Tokenmaxxing is at odds with frugality.”
- 29 Oct 2026Amazon’s third-quarter resultsAnnounced by Amazon. Not yet published when this article went live.
Sources: Amazon engineering memo (27 Oct 2026); Business Insider (29 May and 28 Jul 2026); GeekWire (24 Jul 2026); The Decoder (12 May 2026, summarising the Financial Times); AWS Bedrock model lifecycle (read 28 Oct 2026); Anthropic S-1 (26 Oct 2026).
Powered by Thorsten Meyer AI
Why Amazon Is Reframing AI Performance
The guidance addresses a practical problem for employers adopting generative AI: activity is easier to count than value. Token totals, tool logins and code volume can show that a system is being used, but do not establish that the work is more useful, faster, safer or less costly. If workers are rewarded for those counts, they may have reason to produce more activity without improving results.
Amazon’s position matters beyond its own engineering teams because companies are setting expectations for AI use while the tools and their benefits remain uneven. The memo offers a distinction between using metrics to investigate behavior and using them to score employees. That approach may help companies avoid turning experimentation into a quota, while still identifying teams that need training or better tools.
The company-level comparison is also instructive, though not a like-for-like performance test. Amazon’s reported retrenchment in parts of its Nova portfolio and the rising paper value of its Anthropic stake show that building a model and investing in a model maker can produce very different financial and operational outcomes. Neither fact alone settles whether the bets are succeeding for customers.
Products Worth Considering
LG 34GX90SA-W 34-inch Ultragear WQHD OLED Curved Gaming Monitor
Jackery Solar Generator 1000 v2 with 200W Solar Panel
Logitech MX Keys S Wireless Keyboard, Graphite
As an affiliate, we earn on qualifying purchases.
From KiroRank to Company Bets
The memo follows earlier reporting about internal AI-use tracking. In May, the Financial Times reported that some Amazon employees gave an internal agent tool, MeshClaw, unnecessary tasks to raise token counts. The accounts were anonymous and remain reported allegations, not a confirmed company-wide practice. The report also described a weekly AI-adoption target and internal leaderboards.
Amazon did confirm to Business Insider at the end of May that a leaderboard called KiroRank had been deprecated. The company described it as an informal tracker made by a group of employees and said it was never meant to encourage AI use for its own sake. Senior vice president Dave Treadwell also told staff, “Please don’t use AI just for the sake of using AI.” The new guideline formalizes that principle in the engineering role framework.
Amazon’s AI portfolio provides a separate backdrop. Business Insider reported in July that most flagship Nova models had been moved to “keep the lights on.” Amazon’s lifecycle information lists Nova Canvas, Nova Reel and the first Nova Sonic as retired. The source material also reports that Amazon invested about $18 billion in Anthropic and valued its holdings at about $190 billion in a June 30 filing. Those figures are reported values, not cash proceeds or a measure of customer impact.
““Tokenmaxxing is at odds with frugality.””
— Amazon engineering memo
What the Metrics Cannot Yet Show
The memo does not provide a company-wide measure of how much AI has improved engineering output, nor does it quantify any cost increases caused by usage targets. It says input measures correlate with outcomes today, but does not specify the data, time period or size of that relationship. Correlation is not proof that higher usage caused better results.
The available information also does not establish how widely tokenmaxxing occurred or whether it affected formal performance reviews. The May accounts were anonymous, and Amazon characterized KiroRank as an informal employee tracker. Details of the internal tool use and the scale of any resulting waste remain unclear.
On the investment side, the Nova status reports and the stated value of Amazon’s Anthropic holdings do not reveal the full economics of either strategy. A paper valuation can change, and model availability alone does not show how much customers use a product or whether it is profitable. Amazon had not yet reported its third-quarter results in the source material.
Amazon’s Results Set the Next Test
Amazon was scheduled to report third-quarter results on Oct. 29, one day after the memo’s publication. Those results may offer additional information about the company’s spending and AI-related business, but the source material does not establish which AI metrics Amazon will disclose. The memo itself sets no deadline for a new company-wide outcome score.
For now, the next test is how managers apply the guidance: whether they use adoption and token data to ask better questions, while evaluating teams on the value and quality of their work. Amazon’s stated approach calls for leaders to compare heavy and light users, examine the reasons behind the gap and judge whether AI improved the work—not to reward usage alone.
Key Questions
What did Amazon’s new engineering memo change?
It set out revised guidance for software engineers that centers on customer, business and team impact. It warns against using token counts, AI adoption, code changes or reviews as performance targets.
Is Amazon telling employees to stop using AI?
No. The memo says usage measures can help identify patterns and that teams with higher adoption are currently correlated with stronger outputs. Its warning is against scoring people simply for using AI.
What was KiroRank?
KiroRank was an informal internal leaderboard for AI use, according to Amazon’s statement reported by Business Insider. Amazon said it had been deprecated and was not intended to promote use for its own sake.
What remains unknown about Amazon’s AI results?
The memo does not quantify AI’s effect on engineering productivity, costs or product quality. The reported Nova changes and Anthropic investment valuation also do not, by themselves, show customer impact or profitability.
Source: ThorstenMeyerAI.com
