Two AI agents on a phone call should stop talking

Two AI voice agents pick up a phone call to each other. Today, both of them do speech-to-text, then think, then text-to-speech, for every single turn. They are using a phone call to send each other words they could have sent as JSON — paying codec latency, model latency and per-minute telephony charges to do it.

Tonecall is the middleware that lets them notice they are both machines and switch to a data channel mid-call. The phone call stays open, because a human might walk in at any moment. Only the transport for the actual negotiation changes.

Without Tonecall With Tonecall ──────────────── ───────────── ╭── voice ──╮ intro: "hi, this is…" ~10s A ── voice ── B A B every single turn ╰─── JSON ──╯ the actual deal ~3s 4–6 s per turn ╭── voice ──╮ goodbye: "thanks, bye" ~5s A B slow + expensive voice only where it earns its keep

1Why this is a protocol problem, not a prompt problem

The instinct is to solve this in the model: tell each agent to be terse, or to detect a robot from transcript style. That fails for a structural reason — by the time a transcript exists, you have already paid the cost you were trying to avoid. STT ran. The LLM ran. TTS ran. The 4–6 seconds are spent.

Detection has to happen below the language layer, in the audio path, before anyone transcribes anything. That makes this a middleware and signalling problem.

The constraint that shapes everything You do not control the channel. Between the two agents sits the public phone network: carriers, codecs, jitter buffers, and a CPaaS provider in the middle. You cannot add a header. Anything you want to say out-of-band has to survive being turned into audio and back.

2Just enough telephony

A phone call is two conversations at once. SIP is signalling — ring, answer, hang up; roughly HTTP for calls. RTP is the audio itself, usually 20 ms of compressed sound per packet. A CPaaS provider operates all of that and exposes it as a REST API, so this project never touches SIP or RTP directly.

When somebody dials a number you own — a DID — the provider answers and asks your server what to do, by POSTing to a preconfigured answer URL. You reply with a small XML document of call-control verbs.

Verb	What it does	Used here
`<Dial>`	Ring another number and bridge this call to it	yes — the bridge
`<Stream>`	Open a WebSocket and pump the call's audio through it	yes — the audio tap
`<Speak>`	Say text to the caller	intro / outro
`<Hangup>`	End the call	teardown

<Stream> is the primitive that makes the whole thing possible: it turns a phone call into a WebSocket of audio frames you can inspect in real time.

3The handshake: asking "are you a machine?" in audio

The channel only carries sound, so the probe has to be sound. The answer is in-band DTMF — the touch-tone digits every phone system on earth already forwards reliably. Each agent emits a short tone sequence acting as a pairing token; a listener that replies with a well-formed token is, by construction, not a human.

This is a deliberately old idea. DTMF survives transcoding, works through carriers nobody has an agreement with, and needs no cooperation from the network. The comparison I keep coming back to: fax machines, but for modern AI agents. Two machines screeching at each other briefly to agree on a better way to talk.

caller ──▶ +1-NUMBER ──▶ CPaaS provider ──▶ answer URL ──▶ tonecall (SIP + RTP) (XML verbs) middleware │ ┌─────────────────────────────────────────────┴────┐ │ │ peer is an AGENT peer is a HUMAN │ │ voice intro ────┤ ┌──▶ STT ──▶ LLM ──┐ DTMF pairing ───┤ └─── TTS ◀─────────┘ JSON /data WS ──┤ 4–6 s per turn voice outro ────┘

4Four phases

A completed agent-to-agent call moves through four phases. The two voice phases are not decoration — they are what keeps the call legible to a human who joins late, and what leaves an audible record of what was agreed.

t=0s ~10s ~13s ~18s │ │ │ │ ├── DIAL ───┼──── INTRO ──────┼─── DATA (JSON) ───────┼── OUTRO ──▶ hangup │ bridge │ voice, │ side-channel, the │ voice, │ + DTMF │ human-audible │ real negotiation │ human-audible │ pairing │ │ │ └ both legs └ proves the └ one round trip — └ leaves an on one call is real no STT, no TTS, audible record bridge to a listener no LLM voice latency

~10svoice intro

~3sJSON side-channel

~5svoice outro

4–6sper turn, human path

The number that matters is the middle one. Everything the two agents actually need to settle happens in a single round trip over a WebSocket, instead of n turns of 4–6 seconds each. The saving scales with how much the agents have to say to each other.

5The human path is the real test

An optimisation that only works in the happy case is not shippable here. If the peer never answers the DTMF probe, it is a person, and the system has to fall through to a genuine voice agent — STT, an LLM, TTS, in a loop — with no hint that anything was ever attempted.

Where this gets sharp The fallback cannot be a timeout the caller can hear. A human who says "hello?" into two seconds of tones has already had a bad experience. The probe has to be short enough to hide inside a normal greeting, which puts a hard ceiling on how elaborate the handshake is allowed to be.

6Federated, not centralised

The first build put both agents behind one orchestrator. That is easier and it is also cheating: if one process owns both sides, it never has to discover anything, and the protocol is doing no work.

The current design runs two independent middleware instances, each owning exactly one agent. They pair through the in-band DTMF exchange plus a broker that resolves which instance owns a token.

┌──────────────── instance A :3000 ────────────────┐ │ agent: vegetable_vendor broker │ └───────────┬──────────────────────────────┬───────────┘ │ in-band DTMF over the │ pairing lookup │ bridged audio path │ who owns this token? ┌───────────┴──────────────────────────────┴───────────┐ │ agent: pizza_shop │ └──────────────── instance B :3001 ────────────────┘ Neither instance holds both agents. Pairing is discovered in-band — which is the point. No shared orchestrator, no single owner.

You can run the whole topology locally with no telephony account and no API keys at all, because the LLM, STT and TTS are stubbable:

npm run dev:a    # vegetable_vendor + broker on :3000
npm run dev:b    # pizza_shop on :3001

curl -X POST http://localhost:3000/api/simulate \
  -H 'Content-Type: application/json' \
  -d '{"peerBase":"http://localhost:3001"}'

Instance B forwards its events to instance A, so a single dashboard shows both legs of the call. Being able to exercise a four-phase telephony protocol with USE_STUBS=true and two terminals is the single thing that made this tractable to build.

7What I would change

DTMF is a low-bandwidth probe. It is perfect for "are you a machine, and what is your token" and hopeless for anything richer. A capability exchange — which schema version, which codecs — wants to happen over the data channel immediately after pairing, not in tones.
The broker is a soft centralisation. Pairing is discovered in-band, but token resolution still goes through a known endpoint. Fully peer-to-peer discovery is the honest end state.
Trust is unresolved. Right now a well-formed token is sufficient to be believed. Anything real needs the side-channel authenticated, or you have built a machine that will happily negotiate with whoever tones at it first.

The design docs and a page-by-page walkthrough of the current code live in the repo under docs/. The protocol writeup covering the <Dial> bridge and DTMF pairing is docs/avip-1-dial.md.