How AT&T Built a Trillion-Token Telecom AI Using Open Models, AMD GPUs, and Microsoft Foundry

On July 23, AT&T pulled back the curtain on a telecom AI project of extraordinary scale: its OTel 2.0 model family development consumed roughly one trillion tokens, relying on Microsoft Foundry Managed Compute and a heterogeneous GPU fleet that included 430 AMD Instinct MI300X accelerators. The company says the approach saved tens of millions of dollars compared to routing the same workloads through frontier AI services. More than a benchmark, the announcement lays out a new production blueprint for domain-specific AI that runs on open models and dedicated infrastructure.

Inside the trillion-token build

OTel 2.0 isn’t a general-purpose chatbot. It’s a family of models tuned for the highly specialized language of telecommunications—network architecture, standards documents, engineering tickets, spectrum data, and operational runbooks. AT&T developed the models using raw documents from the GSMA trade body alongside synthetic data generated by other open models.

The headline token count deserves careful parsing. The team processed approximately one trillion tokens overall, but only about 400 billion were actually used for training. The rest covered data preparation, synthetic generation, filtering, inference, and other pipeline stages—work that is often invisible but consumes enormous compute.

Phi-4, a compact open model, handled a staggering 700 billion tokens per month during data preparation and synthetic-data generation. Two other open models—OSS 120B and Gemma 4—supported higher-reasoning tasks and broader development workflows. This multi-model approach let AT&T assign the right tool to each stage: small, efficient models for high-volume preprocessing; larger, more capable models for complex analysis.

Infrastructure came from Microsoft Foundry Managed Compute, a preview service that provisions dedicated GPU clusters without requiring customers to manage VMs, Kubernetes, or serving runtimes. AT&T used about 530 GPUs total, spanning multiple architectures. The AMD Instinct MI300X—with 192 GB of HBM3 memory per accelerator—was the workhorse, providing the memory headroom needed for large-context telecom data. A heterogeneous mix let the team optimize for cost, throughput, and model compatibility rather than locking into a single GPU vendor.

Crucially, the platform allowed deployment and scaling in days, not weeks. For a team iterating on trillion-token experiments, that speed kept development momentum high.

What this means for Windows IT and enterprise architects

AT&T’s project isn’t just for telcos. It surfaces trends that directly affect Windows-centric IT environments.

For system administrators and infrastructure leads: The Foundry Managed Compute model abstracts away GPU driver management, container orchestration, and distributed serving. That means your team can expose domain-specific AI models through common endpoints and identity systems—such as Microsoft Entra ID—without becoming AI infrastructure operators. Focus shifts to integration: configuring private networking, role-based access, logging, and monitoring for apps that consume these models. If your organization runs applications on Windows Server, Azure, or hybrid setups, those apps can talk to Foundry-hosted models the same way they call any secure API.

For developers and data engineers: The multi-model, multi-hardware approach is a playbook you can borrow. Instead of funneling everything to a single large language model, consider a pipeline: a compact model for data classification or extraction, a reasoning model for complex tasks, an embedding model for retrieval, and a domain-tuned model for industry-specific language. Managed compute makes it practical to host several open models without each team building its own serving stack.

For decision-makers evaluating AI budgets: AT&T’s reported savings—tens of millions of dollars—come from substituting dedicated open-model infrastructure for per-token consumption pricing on frontier models. This math works when workloads are sustained and large. If your AI usage is spiky or experimental, consumption pricing may still be cheaper. The key is to measure cost against task quality, not just token volume. A cheap model that produces bad synthetic data will cost you downstream.

But there’s a catch: Foundry Managed Compute is still in preview. Microsoft’s docs explicitly state it has no SLA and isn’t recommended for production. AT&T’s success shows what’s possible, but it doesn’t mean you should run a business-critical workload on a preview service without a fallback. Check your internal policies, available regions, accelerator quotas, and support arrangements before committing.

The road to trillion-token telecom AI

AT&T’s ambition didn’t start with OTel 2.0. The original Open Telco model, OTel 1.0, has been downloaded over 25 million times, signaling strong demand for telecom-specialized AI. That early work highlighted the limits of generic foundation models: even the most capable general AI can stumble when interpreting a 5G standards clause or a network fault-management procedure.

Simultaneously, the enterprise AI market has been shifting. Open models from Hugging Face, Meta, Mistral, and others have matured enough for production use, while cloud providers raced to offer managed GPU services that reduce operational toil. Microsoft Foundry emerged as a unified platform for discovering, fine-tuning, and deploying models, and the Managed Compute variant added dedicated capacity for organizations that need to run large-scale workloads without sharing tenant infrastructure.

AMD’s Instinct MI300X entered the picture as a credible alternative to NVIDIA H100s for memory-bound AI tasks. With 192 GB of HBM3, the MI300X is particularly appealing for inference on large models and long-context workloads—precisely the kind of work that dominates domain-specific data processing. By spanning both AMD and NVIDIA GPUs, AT&T signaled that the industry is moving toward accelerator-agnostic AI pipelines, where the workload dictates the hardware, not the other way around.

What to do now: a practical checklist

If AT&T’s approach resonates, here’s how to start evaluating a similar path for your organization:

1. Inventory your domain knowledge. Identify the collections of internal documents, standards, runbooks, or engineering logs that could form the seed corpus for a domain model. Map out data ownership, licensing, and confidentiality constraints.

2. Experiment with open models. Use Hugging Face or Foundry’s model catalog to test small, open models (like Phi-4 or Gemma) on your domain data. Compare their accuracy against generic frontier models for tasks like classification, extraction, and summarization.

3. Design a multi-model pipeline. Don’t expect one model to do everything. Prototype a workflow: a small model for bulk preprocessing, a larger one for reasoning, and an embedding model for search. Measure throughput, cost, and accuracy at each stage.

4. Evaluate managed vs. self-managed infrastructure. Calculate the total cost of ownership for running dedicated GPUs yourself versus a service like Foundry Managed Compute. Factor in not just GPU hours but DevOps time, patching, and capacity planning.

5. Review preview service terms carefully. If you decide to use a preview service, understand the SLA gaps, supported regions, and model lifecycle policies. Have a plan to migrate or scale back if the service changes.

6. Treat synthetic data as suspect until proven. If you generate training data with a model, implement automated quality checks, human spot-reviews, and regression tests against known-correct answers. A synthetic-data error at scale can poison an entire model.

7. Engage legal and compliance early. “Open model” doesn’t always mean “open source” with no restrictions. Review licenses, acceptable-use policies, and data-handling requirements for every model you deploy.

8. Build the integration layer. Prepare your Windows-based applications and identity systems to securely consume model endpoints. Work with Entra ID, private endpoints, and API management tools to govern access.

What to watch next

AT&T’s OTel 2.0 is a milestone, but it’s also a signpost. Expect more enterprises to publish similar stories of domain-specific AI built on open models and heterogeneous GPU fleets. The real inflection point will come when Microsoft Foundry Managed Compute exits preview with an SLA—turning the service from a fascinating experiment into a standard production option. Until then, the smart play is to start prototyping with open models today, so you’re ready when the infrastructure matures.