- Problem
- Restaurants lose orders when nobody picks up at peak hours. Digirestro, a POS startup, wanted a voice agent that takes orders by phone — and a way to know it gets them right.
- What I built
- Built the agent from scratch, solo — speech-to-text, LLM, text-to-speech, and tool calls into the kitchen POS, payments, and delivery.
- Solved barge-in with real-time audio frame pacing: interruption latency 1–3 seconds → under 100ms.
- Built the evaluation engine: 72 test cases mined from 169 real customer calls, each scored by an LLM judge, 5 runs per case against frozen agent snapshots.
- Used it to raise task accuracy from 60% to 92% on the original 49 cases — with no model change.
- Rewrote a 45k-character prompt into layers, halving tokens per turn; moved POS relay and delivery booking to a background task, cutting order confirmation from ~15s to ~2s on test calls.
- My role
- Solo, end to end — design, architecture, code, deployment, and the evaluation suite. No other engineer on the project.
- Outcome
- Task accuracy 60% → 92%, order confirmation ~15s → ~2s, interruptions answered in under 100ms. The eval-driven improvements are validated on staging and ready for production rollout.
- What I learned
- Isolating failures by pipeline stage showed a model swap would not help — the prompt and tool responses were the levers. Spoken numbers were the proof: prompt-only fixes scored 0/5 on the eval; having tools return the exact sentence to speak scored 5/5.
Client: FedPoint
