- Problem
- Digirestro, a startup building POS and payment systems for restaurants, wanted voice AI agents to take orders and reservations over the phone — restaurants lose revenue every time nobody can pick up at peak hours. Getting an LLM to hold the conversation is straightforward; making it feel human on a real phone line is not.
- What I built
- Built the system from scratch and integrated it with Digirestro's kitchen POS, so a phone order reaches the kitchen the same way a counter order does.
- Evaluated three platforms end to end — Azure Voice Live (full pipeline control, but you build the telephony bridge yourself), Infobip (solid telephony, not designed for AI agents), and Vapi (managed STT + LLM + TTS).
- Shipped the first version on Infobip, then hit a wall: it buffers up to 1024 audio frames (~20 seconds) with no way to flush mid-stream, so the agent physically could not be interrupted.
- Solved barge-in with real-time audio frame pacing that keeps the buffer nearly empty. A 3ms difference in frame timing — 17ms versus 20ms — was the line between responsive and broken.
- Migrated to Vapi for agent orchestration: dynamic per-restaurant agent creation, plus tool integrations for date/time handling and a menu refresh service.
- Built a restaurant onboarding flow so a new venue is provisioned with its own agent and menu.
- Deployed on Azure using Azure AI Foundry resources, with a database for orders and history, and analytics tracking per-model cost.
- My role
- Solo — built the product end to end for the client: platform evaluation, the real-time audio pipeline and barge-in fix, POS integration, Azure deployment, agent orchestration, and the cost-analytics layer.
- Outcome
- Interruption latency went from 1–3 seconds to under 100ms, and orders flow from a phone call straight into the kitchen POS. The honest finding: speech models still mishear regional accents, returning common dish names as phonetically similar English words — a model-level limitation no amount of prompting fixes. For a product serving Indian restaurants that is the primary failure mode, not an edge case. Vapi ships fastest, but per-minute pricing scales poorly at hundreds of calls a day.
