A 56% cost reduction with only a 2% drop in quality. That’s not a rounding error — that’s a strategy. AT&T has reported significant savings on AI coding and advanced tasks by routing employee queries to cheaper models when the task doesn’t actually require a frontier model. The results are hard to argue with, and the approach is replicable by almost any enterprise running internal AI tooling at scale.
The tool doing the heavy lifting here is LiteLLM, a model router that evaluates query complexity and decides whether a request needs to go to something expensive like GPT-4o or Claude Sonnet, or whether a cheaper, lighter model can handle it just fine. For a telecom the size of AT&T, with thousands of employees using AI daily, even small per-query savings add up fast. Mark Austin, the VP overseeing AT&T’s internal AI deployment, confirmed the numbers in an interview with The Information.
The broader cost strategy runs alongside the routing play. AT&T currently sends about 40% of employee queries to open-source or open-weight models, and it wants to push that figure to between 60% and 70% over the next few years. The goal is straightforward: keep spending on Anthropic and OpenAI flat while doing more total work. The models AT&T is already using for this include Nvidia’s Nemotron, Meta’s Llama, and Google’s Gemma — all capable, production-grade options that have matured considerably over the past 18 months.
Notably, AT&T is not using DeepSeek or Moonshot, the Chinese open-weight models that have attracted both excitement and scrutiny across the industry. Austin said the company is still evaluating the risk profile of those models. That’s a reasonable position given ongoing concerns around data governance and geopolitical exposure, and it signals that AT&T’s cost strategy has guardrails around it.
The capability gap between open-source and frontier models is also closing faster than many expected. Austin’s read is that open-source models have historically trailed frontier models by six to ten months, but that gap is narrowing. His assessment: they’re now “just as good or better” than older versions of Anthropic and OpenAI models. That’s meaningful. It means companies don’t have to sacrifice much to run a larger share of workloads on cheaper infrastructure.
This fits into a trend that’s been building throughout 2026. The shift from simple chatbots to agentic workflows has driven AI compute costs sharply higher, and the move from flat subscriptions to token-based billing has made that cost visible in ways it wasn’t before. Companies that spent two years encouraging heavy model usage are now looking hard at what they’re actually getting per token. The era of defaulting to the biggest available model for every task is ending.
- LiteLLM routes queries based on task complexity, reducing spend without major quality trade-offs
- Open-source models (Llama, Gemma, Nemotron) now handle 40% of AT&T’s internal AI queries
- Target is 60-70% open-source within a few years, keeping frontier model spend flat
- DeepSeek and Moonshot are being evaluated but not yet deployed due to risk concerns
- Performance gap between open-source and frontier models is narrowing to near-parity on many tasks
For developers and AI architects at large enterprises, the AT&T case is a practical blueprint. Model routing isn’t new — LiteLLM has been around for a while — but seeing a company this size publish real numbers on cost and quality trade-offs gives the approach credibility it previously lacked. If your organization is still routing every query to the same flagship model regardless of task complexity, you’re almost certainly overpaying. The infrastructure to do this more intelligently already exists. AT&T just proved it works at scale.
