Speech-to-Speech v2: The End of the Pipeline
Updated
Our own end-to-end voice system, built to sound human. Because human is what converts. Live in the API on August 1st.
For years, "AI voice" was a relay race. Your words became text. The text passed through a language model. The model's reply was synthesized back into speech. Three systems, three handoffs. And at the very first step, transcription quietly discarded everything that made you sound human. Your hesitation, your emphasis, the edge in your voice when something went wrong: all flattened into plain characters before the model ever "heard" you.
Speech-to-Speech v2 ends the relay.
We're not building an IVR
This matters more than it sounds, because of what we actually build. NLPearl is not an IVR. We are not here to route calls through a menu or read a script back to people. We build agents that replace and scale human teams: sales, support, qualification, the conversations that decide whether a customer stays or leaves.
And those conversations only work if they feel human. People behave differently the moment they sense a machine. They give shorter answers, they disengage, they hang up. The instant a caller feels they are speaking with a person, they stay, they open up, they trust. That difference is not cosmetic. It is the conversion rate. An agent that sounds like an IVR performs like an IVR. The closer it gets to human, the closer its outcomes get to a human team's. Sounding human is not something we add on top of the product. It is the product.
Audio to audio, no text in between
v2 is an end-to-end system that runs on audio, not text. Audio goes in, audio comes out. There is no transcription step in the middle to strip the signal, and no synthesis step bolted on at the end to fake it back. The models hear the actual waveform (tone, pace, pauses, emotion) and respond in the same medium, in real time. A classic pipeline is three generic, off-the-shelf components passing flat text between them. v2 is a set of models designed and trained to work together on audio, from the ground up, to do one thing: hold a natural conversation.
Latency that keeps the conversation flowing
What matters is not a number on a slide, it is whether the exchange feels natural. v2 runs at around 800 milliseconds on average, well inside the range where conversation stays fluid and the caller never feels they are waiting on a machine. And it holds that pace while doing real work: calling tools, looking up accounts, checking availability, taking actions in the middle of a conversation. We built a deliberately sophisticated architecture so the system can orchestrate all of that live, as it speaks. Fast where it counts, and getting faster with every release.
Emotion is signal, not decoration
This is the part a pipeline cannot fix. Once speech becomes text, emotion is gone. The model is reasoning over a flat transcript, blind to whether the caller is calm, confused, or furious. Whatever feeling the synthesizer adds at the end is generic, applied after the fact, disconnected from what the person actually felt.
v2 never loses the signal. It hears frustration and softens. It hears hesitation and slows down. It hears a smile and warms up. Emotion is not a voice setting layered on top. It is information the system uses to decide what to say and how to say it, the way a person would.
The real breakthrough: we own it
Most voice AI is rented. Teams wire their product into a third-party realtime API and inherit that vendor's rate limits, pricing, deprecation schedule, and roadmap. They are one customer among thousands, sharing capacity and waiting in someone else's queue.
Speech-to-Speech v2 is ours. We built it ourselves, and that single fact changes everything downstream:
Control. We set the behavior, the guardrails, and the voice. No black box, no forced deprecations, no limits imposed from outside. When something needs to change, we change it. We do not file a request and wait.
Optimization. We tune the entire path (inference, voice, infrastructure) for one workload: real-time conversation at scale. No extra network hops to an external service, no compromises made for someone else's use case.
Speed of evolution. v1 to v2 was us. The next leap is us too. We ship on our own cadence instead of waiting in a vendor's release cycle.
Quality. We tune for production, not demos: the accents, the crosstalk, the interruptions, the messy real-world calls that decide whether a contact center actually works.
Personalization. Custom voices, brand voices, domain adaptation. Because the weights are ours, "make it sound like us" is an engineering task, not a feature request that may never ship.
When the technology is yours, your use case is the priority. Not shared. Not throttled. Not someone else's afterthought.
Under the hood: how a speech-to-speech system is built
Incoming speech is encoded into discrete audio tokens by a neural codec, turning a waveform into a sequence of tokens. A large transformer, pre-trained on massive amounts of speech and paired audio and text, predicts the response directly as audio tokens: what to say, how to say it, and when to call a tool, all in one stream. A vocoder decodes those tokens back into a natural voice. Across training, the system first learns to understand speech and prosody, then is fine-tuned on real conversations, then aligned to behave like a great human agent. No transcription, no bolt-on synthesizer. Just audio in, audio out.
Available in the API, August 1st
On August 1st, Speech-to-Speech v2 goes live in the NLPearl API. The same system behind the conversation in the video above, available to build on directly.
This is the voice layer we always wanted: end-to-end, emotionally aware, fast enough to keep a real conversation flowing, and entirely ours. Built to sound human, because for us, human is the whole point.
David Sztern, CEO and Co-Founder