All posts
6 min read

Giving an AI agent a phone number (and where the audio goes)

A phone number is the most universal interface there is — and the one AI agents handle worst, because voice drags in a swamp: audio pipelines, speech recognition, barge-in detection, latency budgets. The first line we built refused the swamp outright, and that is still the default: on the Twilio bridge, no audio reaches us at all. It is no longer the only way to run a line — and the second way is the interesting part — but the refusal is what got a phone call working before anyone picked one up, so start there.

Text in, text out

The default bridge is built on Twilio's ConversationRelay: Twilio owns speech recognition, speech synthesis and voice-activity detection, and what crosses the wire to us is text. The caller's words arrive as a transcript; the agent's reply streams back as text and Twilio speaks it. That sounds like an implementation detail. It is actually the whole ballgame, for three reasons:

  • It's testable without a phone. A plain text client exercises that entire bridge — every code path a real call takes — with no phone number, no audio hardware, no telephony account.
  • The agent is the same agent. A call turn is a normal agent turn — same prompt, tools, memory and permission checks as chat. The transcript lands in the same conversation, so Tuesday's call and Thursday's WhatsApp message are one thread — keyed by the caller's number.
  • Nothing new to secure. On that bridge there is no audio to store and no voice pipeline to defend. The existing capability and audit machinery covers the calls because a call is just turns.

Then the trade came due

Text in, text out buys a testable bridge, and it costs two things people ask for: latency, because every turn waits on somebody else's speech stack, and choice, because you get Twilio's ears, Twilio's voice and Twilio's number or nothing. So a line can now also run on a streaming engine. The carrier sends raw call audio; our server pipes it, without parsing a frame of it, into your project's own isolated container — and from there either the container does the listening and speaking through our gateway, or the audio is relayed on to OpenAI, which runs the conversation itself and calls back into your agent when it needs something. The fastest of them roughly halves the mouth-to-ear delay, and either way you get six carriers to pick from instead of one.

That is a real trade rather than a free upgrade, so here is the half that is not free: on those engines the call's audio does flow through our infrastructure. It is relayed rather than kept — when you switch recording on, the recording is made and stored in your own carrier account — and it lands in the same container, under the same permissions and the same audit trail as everything else your agent does. But "we never see it" is a sentence about one engine, not about the platform, and it is worth knowing which line you are setting up.

Every call placed or answered so far has run through Twilio; the five other carriers are built to their vendors’ published references and none has been on a live call yet, so treat a non-Twilio line as in testing.

Interruption is three problems, not one

When a caller talks over the agent, cutting the audio isn't enough — the model doesn't know it was interrupted and keeps generating, and half-spoken text is still queued. So an interruption stops three things: Twilio drops the unspoken audio, the agent's turn is aborted, and the delta stream is fenced so no orphan fragment gets spoken after the caller started talking. The transcript then records what was actually heard, not what was generated — so later turns never reason from words nobody received.

Outbound is gated like the hazard it is

An agent that can place calls is an agent that can ring a stranger and spend your money, so outbound is gated four ways: the phone capability asks for approval by default, destinations must be on a per-line allowlist, there's a daily cap, and a per-destination cooldown. Transferring a live call to a human is deliberately frictionless — but only to allowlisted targets, because "transfer me to +1…" would otherwise be a one-line prompt injection.

The part callers never notice is the best part: the agent answering the line is the same one on your WhatsApp and your email — one memory, one permission model, and a phone call is just another door into it. How it sounds from the caller's side is on the phone page; giving your own agent a number is a connect-Twilio-and-paste-one-URL setup, and the assistant walks you through it. If you came here weighing this against a receptionist rather than building one, the business-side write-up covers disclosure, guardrails and what it costs against published alternatives.

See it for yourself — free, $2 of credits once you confirm your email, no card.

Start free