Microsoft has pushed gpt-realtime, a speech-to-speech model designed for low-latency, natural-sounding conversational agents, into general availability on Azure AI Foundry. The release, accessible via the Realtime API, consolidates months of engineering into a single end-to-end pipeline that skips the traditional assembly of separate speech recognition, language understanding, and text-to-speech components. Developers and enterprises can now build voice applications that sound more human, follow instructions more precisely, and cost about 20% less per token than the previous gpt-4o-realtime preview.

The move signals a deliberate pivot from treating voice AI as an experimental add-on. Instead, Microsoft is weaving expressive speech, multimodal image inputs, and real-world telephony connectivity into a production-grade stack. Two new voices—Marin and Cedar—debut with the model, promising clearer, more lifelike output for everything from customer service bots to interactive narrators. Enterprises evaluating real-time voice agents finally have a single-model solution that blends audio fidelity, function calling, and cost controls.

The end of the pipeline: why a single model matters

Most voice assistants still chain together three separate sub-models: automatic speech recognition (ASR) converts spoken words to text, a language model reasons over that text, and a text-to-speech (TTS) engine vocalizes the reply. Each handoff adds latency and discards acoustic subtleties like tone, pause, or emphasis. The result is often a stilted, robotic interaction that frustrates users.

gpt-realtime collapses that pipeline into one unified model. It ingests raw audio, processes meaning and intent directly, and generates audio output—all within a single neural flow. This design eliminates the conversion artifacts that plague multi-stage systems and preserves paralinguistic features across the exchange. For contact centers, that means a customer’s angry tone can be acknowledged before it is calmed; for accessibility tools, clarity and prosody become standard rather than aspirational. Single-model S2S also simplifies developer architecture: you configure one endpoint, one set of session instructions, and one token budget.

Marin and Cedar: new voices, new benchmarks

The two new voices, Marin and Cedar, are central to Microsoft’s pitch. Described as “natural, expressive,” they aim to surpass the often-monotone defaults that have handicapped earlier voice agents. Marin and Cedar can handle complex scripts—legal disclaimers, alphanumeric order numbers, multi-language snippets—without breaking character. Their introduction underscores a shift from merely intelligible speech to genuinely engaging conversation.

These voices are part of a broader set of output styles that developers can select and tune. While Microsoft hasn’t disclosed the exact architecture that underpins Marin and Cedar’s expressiveness, the real test will come in customer pilots: do they sound empathetic during a complaint call? Do they read a serial number clearly enough to prevent a mis-shipment? Early adopters are encouraged to test the voices across their most demanding scripts.

Sharper instructions, reliable function calls

Voice agents frequently stumble when they must deliver rigidly formatted information. A bot that misreads a confirmation code or paraphrases a legal warning undermines trust. Microsoft says gpt-realtime improves instruction-following accuracy, making it more likely to adhere verbatim to critical phrases. The model also supports function calling—the ability to invoke external APIs and services from within a conversation. Enhancements include asynchronous function flows: while the agent fetches data from a backend system, the session remains alive, eliminating awkward dead air.

For developers, this unlocks richer agent behavior. A voice assistant can authenticate a caller, look up their account, and then speak the balance—all while maintaining a coherent conversational thread. The async capability is especially important in enterprise scenarios where a function may take several seconds. Instead of queuing a waiting tone, the model can fill the gap with context-sensitive commentary and then deliver the result.

Multimodal: seeing while speaking

Perhaps the most underappreciated advancement is multimodal input. gpt-realtime accepts images alongside audio, allowing a user to speak about a photograph without a video stream. A field technician, for instance, can snap a picture of a broken valve and describe the issue while the AI references the visual evidence in its voice reply. This feature opens use cases in remote diagnostics, visual help desks, and even telemedicine, where a patient might describe symptoms while holding up a smartphone image.

Microsoft frames this as a practical upgrade rather than a futuristic demo. The model ingests the image via the same Realtime session, so there are no extra APIs to stitch together. The voice agent can refer to specific objects, colors, or anomalies in the picture, creating a more natural troubleshooting loop than “please describe the image in your own words.”

Real-world telephony: SIP, PSTN, and conversation mode

General availability also brings features designed for live phone systems. gpt-realtime supports SIP and PSTN entry points, meaning you can route a traditional phone call directly into an AI-powered session. Conversation mode introduces server-side Voice Activity Detection (VAD) and turn-taking controls, so the agent knows when to speak and when to listen—critical for human-like barge-in behavior.

These features move the model from lab demos to actual call-center floors. A customer dialing a support line can now be greeted by an agent that understands natural speech, interrupts gracefully, and transfers to a human when needed. The resiliency improvements handle multi-turn interactions, dropped packets, and edge cases that historically crashed voice bots.

Deploying gpt-realtime: a developer’s path

Azure AI Foundry supports two connection patterns: WebRTC for low-latency browser or mobile apps and WebSockets for server-to-server streaming. The recommended workflow starts with creating an Azure OpenAI resource in a supported region, deploying gpt-realtime from the model catalog, and then minting ephemeral session keys. Each key has a one-minute lifetime, a security measure that prevents accidental exposure of long-lived API credentials in frontend code.

From the Audio playground, developers can test session configurations—modalities (audio, text, image), voice selection, and function endpoints. The playground also provides WebRTC samples for rapid prototyping. For production, Microsoft advises instrumenting telemetry to track latency, audio glitches, and token consumption. Real-time audio is token-hungry, and without monitoring, a seemingly modest per-token rate can balloon into a six-figure monthly bill.

Pricing: the 20% cost reduction in context

Microsoft says gpt-realtime pricing is approximately 20% lower than the earlier gpt-4o-realtime preview on a per‑million‑token basis. Billing is metered per 1M tokens, with rates varying by region and token type (text vs. audio). While the reduction is welcome, the absolute cost still requires scrutiny. A typical customer service call can consume tens of thousands of tokens, and at a contact center handling millions of calls, the math demands precise profiling.

The company encourages developers to use the Azure pricing calculator and their own billing data rather than third-party summaries. Some historical preview rates reported by aggregators may not reflect contract-specific discounts. For budgeting, treat the 20% figure as a starting point and validate every number against your actual deployment region and concurrency.

Where gpt-realtime fits: use cases and early adopters

The model targets a wide swath of voice-first applications:

  • Customer support and contact centers: voice bots that triage tickets, guide troubleshooting, and hand off to humans, with image+voice for visual diagnostics.
  • Accessibility tools: natural-language narration for screen readers, voice-driven UI controls, and reading aids that maintain prosody.
  • Interactive media and games: dynamic non-player characters and narrative agents that react to spoken player input.
  • Voice-enabled internal tools: meeting summarizers, voice search over knowledge bases, and phone-based scheduling assistants.

The combination of lower latency, multimodal awareness, and improved instruction fidelity makes gpt-realtime suitable for both pilot projects and full-scale deployments—provided teams design in the operational rigor that voice demands.

Critical analysis: strengths, limitations, and risks

Where gpt-realtime shines

The end-to-end S2S architecture is a genuine leap. By processing audio natively, it conveys empathy and nuance that the ASR→LLM→TTS chain routinely flattens. Features like SIP/PSTN entry and conversation mode close the gap between prototype and production, giving enterprises the tools they need to connect a real-world phone system. The multimodal input unlocks support scenarios that previously required separate vision APIs, and the improved function calling makes it feasible to orchestrate backend systems from voice with fewer brittle middleware layers.

Limitations and open questions

Cost at scale remains the elephant in the room. Even at 20% lower prices, sustained real-time audio concurrency can generate eye-watering bills unless token usage is rigorously optimized. Region and latency dependencies are another variable: performance claims made under ideal lab conditions can degrade across transcontinental networks or high-jitter connections. Every team should benchmark their exact deployment topology.

Vendor throughput claims from competing voice models—such as single-GPU inferencing speeds—must be taken as engineering hypotheses until independently reproduced in your environment. Microsoft’s documentation acknowledges this; responsible teams will build their own cost and performance models.

Safety, privacy, and misuse risks

Synthetic voice at this quality level deepens the risk of deepfakes and impersonation. Any public-facing application should include explicit voice consent flows, verification safeguards, and content-safety checks. Azure’s trust and safety frameworks provide a baseline, but they don’t replace legal and policy controls. Data residency is equally thorny: real-time audio often carries personally identifiable information, requiring encryption in transit and at rest, along with carefully scoped logging and retention policies. And while instruction following has improved, all large language models, including audio-native ones, can hallucinate. For outputs that must be deterministic—legal text, account numbers—pair the model with authoritative function calls that execute verified code.

Operational recommendations for Windows and Azure teams

  • Validate quality and latency using representative call flows in your serving regions. Start with WebRTC test harnesses and the Audio playground.
  • Instrument token consumption end-to-end. Measure tokens per minute of audio under typical usage and bake cloud cost visibility into your CI/CD pipeline.
  • Design multi-tier fallbacks. When bandwidth or latency degrades, fall back to lightweight text or short prompts rather than dropping the session.
  • Use function calls for deterministic actions. Voice outputs handle conversational flow; structured function results handle the authoritative parts—account lookups, payment confirmations.
  • Build governance from day one. Voice consent flows, fraud detection, content-safety checks, and legal review for any spoken output that could be recorded or heard by customers.

A pilot checklist should cover Marin and Cedar’s clarity with scripts and alphanumerics, image+voice accuracy, latency under concurrent load, edge cases in instruction following, and a real-world cost projection based on average session lengths.

The strategic play: voice as a first-class Azure capability

With gpt-realtime and the GA of the Realtime API, Microsoft is sending a clear message: voice is not a side feature—it is a first-class interface for the Azure ecosystem. By bundling expressive synthesis, multimodal inputs, and carrier-grade telephony connectivity into a single model, Azure AI Foundry becomes a credible one-stop platform for enterprises that must meet security, compliance, and scaling requirements.

The move also prepares the ground for Copilot-powered voice assistants and third-party offerings built on Azure. Where voice agents were once fragile, expensive prototypes, they can now be engineered as robust, cost-manageable services. The 20% price cut, while not trivial, is as much a signal of Microsoft’s commitment to making real-time voice economically viable as it is a pricing adjustment.

Bottom line: production-ready voice, with homework attached

gpt-realtime on Azure AI Foundry is a tangible upgrade for anyone building conversational agents. It delivers improved instruction following, two expressive voices, higher audio fidelity, and multimodal support in a single model that is available today. The engineering friction that once plagued voice projects—stitching together ASR, NLU, and TTS—is largely gone. What remains is the operational discipline required to run voice at scale: tracking tokens, securing sessions, complying with privacy regulations, and preventing misuse.

Enterprises should run focused pilots now, validate every performance and cost assumption, and integrate deterministic function endpoints for authoritative actions. Done right, gpt-realtime can turn a voice-first application from a daunting bet into a competitive differentiator. Done poorly, it becomes an expensive lesson in token management. The model is ready; the question is whether the organizations that adopt it are equally prepared.