In mid-July 2026, Anthropic’s Claude Code lead Boris Cherny told enterprises to stop fixating on AI token consumption as a primary success metric. The better measure, he argued, is the engineering work that didn’t have to happen.
Cherny’s advice arrived in a series of X posts on July 17, as first reported by Business Insider. It lands at a moment when corporate wallets are tightening and the industry’s early “tokenmaxxing” craze—flaunting the sheer volume of AI requests—has given way to hard-edged return-on-investment questions.
“Usage is worth watching (e.g., a dashboard), but it measures activity, not return,” Cherny wrote. The real question for engineering leaders, he said, is whether a company would have spent human effort on a task without AI. “If yes, how much and what would it have cost in manual eng-hours? That’s your return.”
The dashboard isn’t the scorecard
Cherny’s critique isn’t that telemetry is useless. Anthropic itself has recently expanded Claude Enterprise analytics with views for usage, cost, and value. Admins need those dashboards for capacity planning, budget controls, and detecting unusual spikes. But a chart of tokens consumed tells you nothing about whether the work getting done is the work that matters.
A spike in token usage could signal a team experimenting with novel approaches, wrestling a difficult legacy codebase, or simply running inefficient, long-winding agent loops. None of those outcomes proves that the AI deployment saved money, sped up delivery, or prevented an outage. The high-token team might be generating plenty of activity but little shippable code. The low-token team might be quietly automating their most critical maintenance tasks.
“Cost telemetry needs an outcome beside it,” Cherny conveyed. Without that pairing, dashboards become leaderboards for consumption, not tools for engineering management.
What this means for Windows teams
For Windows-focused engineering organizations, the message hits home in very concrete ways. A sysadmin team might use an AI agent to modernize a collection of PowerShell deployment scripts. A .NET shop might let Claude Code draft test coverage for a sprawling Win32 integration. A platform engineering group might offload the investigation of a recurring pipeline failure to an AI assistant.
Counting the prompts or tokens involved in any of those tasks says very little about the impact. The relevant questions are far more operational: Did the script update ship and reduce manual steps? How much review did the AI-generated test suite require? Did investigating the pipeline failure with AI avoid an incident ticket that otherwise would have sat in the backlog for weeks?
AI-assisted code isn’t free just because the model generated it. It still demands code review, security scanning, integration testing, and accountability for production changes. A task that spits out thousands of lines but creates cascading cleanup work isn’t a win, even if the token bill was modest. For every hour of generated code, managers need to measure the hours spent reviewing, correcting, and hardening it.
The bigger payoff: work that was out of reach
Cherny went one step further in a follow-up post. The largest return, he said, isn’t always a direct substitution of AI for a developer’s hour. It comes “when fixing and maintaining happens in the background,” freeing teams to build features and tackle improvements that previously were too small, too slow, or too expensive to justify.
That shift changes the measurement framework. Instead of asking “How much manual work did we avoid?” you also ask “What new work are we now doing that we never had the bandwidth to attempt?” In Windows environments, that could mean finally addressing technical debt in aging installers, systematically hardening Group Policy templates, or implementing automated compliance checks that always got deprioritized.
Measuring that kind of impact requires capturing outcomes at the workflow level, not the model gateway. Teams should track completed work against a pre-AI baseline: lead time for a class of bug tickets, time from vulnerability report to verified fix, release frequency, backlog age, and the volume of maintenance work closed without delaying planned feature development. Those numbers won’t isolate the AI contribution perfectly, but they reflect business and engineering outcomes far better than a token meter ever could.
How we got to token obsession
The path to this moment began in the first half of 2026, when “tokenmaxxing” became a bizarre badge of honor for some organizations. Executives touted their enormous token consumption as proof they were AI-forward. The higher the number, the thinking went, the more thoroughly the company had embraced the future.
That narrative collapsed under its own weight as CFOs and engineering leaders started asking what all that burning compute was actually delivering. JPMorgan CEO Jamie Dimon told CNBC on July 16 that AI costs are “going up rapidly” and companies will have to be “rational about it like any other resource.” OpenAI CEO Sam Altman, speaking from the Allen & Co. Sun Valley Conference the same week, said the top question he hears is “what we can do to help reduce spend or increase value.”
Coinbase and Vercel publicly shared their own cost-cutting strategies, including routing some workloads to cheaper Chinese models without limiting employee access to AI tools. The industry’s mood had shifted from experimentation to accountability. Into that environment stepped Cherny with a disarmingly simple framework: measure the engineering hours you didn’t have to pay for.
Putting the framework into practice
For Windows IT managers and development leads, Cherny’s advice translates into a concrete, four-step evaluation method. It isn’t a one-time calculation but a discipline to embed in how teams assess every AI-assisted sprint.
1. Pick a defined category of work.
Start with a repeatable, scoped slice of engineering activity—say, updating legacy PowerShell scripts across server fleets, or writing unit tests for a WinForms application. Avoid mixing wildly different task types until the measurement habit is established.
2. For every task, ask the two essential questions.
Looking at completed AI-assisted work, decide: would this task have been assigned to a human engineer in the normal course of work? If yes, estimate the manual engineering hours it would have consumed. Be realistic—don’t assume the fastest possible developer; use your actual team capacity and cadence.
3. Compare avoided hours to total cost.
Add up all direct costs: model API charges, platform fees for tools like Claude Enterprise, compute for running local agents, and—crucially—the human time spent reviewing, testing, and occasionally reworking the AI’s output. If the AI-generated code required an extra round of security auditing or a full rewrite of error handling, those hours count against the return. The net ROI is the avoided manual hours minus these overhead hours and dollar costs.
4. Layer on process-level metrics.
Once you’re comfortable with task-level ROI, expand the view. Track lead time for common issue categories before and after AI adoption. Monitor release frequency. Measure how much maintenance work gets completed without stealing developer time from planned features. These broader indicators will surface the “background fixing” gains that Cherny says deliver the most enduring payoff.
Administrators should still maintain spend controls, audit logs, and guardrails—especially when coding agents have access to repositories, terminals, or internal infrastructure. A tool like the open-source Clawdmeter desktop widget can give developers personal visibility into their own usage, but the real management dashboard must pair model costs with delivery data and review burden, not just raw consumption.
What’s next
Cherny’s call for ROI clarity is part of a larger maturation. The industry is moving past the magic-show phase of AI and into the hard work of measuring its impact on engineering bottom lines. Expect enterprise AI platforms—including Anthropic’s own Claude offerings—to build more outcome-tracking features, linking model sessions to committed changes, test pass rates, and deployment success.
The deeper promise isn’t just cost avoidance. When routine maintenance, security hardening, and legacy modernization run quietly in the background, engineering organizations can finally tackle work that was permanently stuck below the priority line. That’s the metric worth chasing—and it never appears on a token dashboard.