Early lessons from building proAgent

Voice-based AI agents feel easy until you try to make them fast. Then you realize: Speed isn’t just a good-to-have feature—it’s the UX.

Below is a breakdown of what we’ve learned so far while building an enterprise-grade, production-ready, real-time voice agent for the consumer finance industry.

System components and dependencies

Most voice agents use this 4-part pipeline:

• LLM for reasoning.
  
• Transcription engine (e.g. Deepgram).
  
• Text-to-speech engine (e.g. ElevenLabs).
  
• Telephony stack (e.g. Twilio).
  

Voice-native LLMs like GPT-4o abstract some of these layers but they sacrifice control and observability. That tradeoff doesn't work for highly regulated industries such as banking and lending, where the reliability of agent outputs is extremely non-negotiable.

Latency anatomy

Every conversational turn involves:

• End-of-speech detection
  
• Transcription
  
• LLM inference
  
• Tool calls
  
• TTS synthesis
  

Latency is measured from the moment the user stops speaking to when the agent starts speaking.

LLM responsiveness is typically tracked via:

• Time to First Token (TTFT) – Time taken for the LLM to generate the first token after receiving a prompt.
  
• Tokens Per Second (TPS) – Number of tokens that a LLM can generate or process in one second.

Key takeaway:
TTFT is the most critical metric for perceived responsiveness.
TPS matters less than how fast the agent starts speaking.

Streaming is non-negotiable

We stream LLM outputs to the TTS engine as they're generated.

Pros:

• Extremely low time-to-first-token (~300–500ms in ideal conditions).
  
• Feels natural, even if the total response takes longer.
  

Cons:

• Impossible to apply full rule-based post-processing.
  
• Guardrails have to be applied before or during generation, not after.
  

Design-time optimizations

We shaved 1–2s off total latency with these changes:

• Reduced the number of tool-invoking steps in prompts
  
• Avoid unnecessary token bloat with intelligent tool arguments (for example, not passing data in prompts if it can be fetched via backend)
  
• Injected fillers like “Okay,” or “Sure,” to create perceived immediacy.
  

Smaller models where possible

If a task doesn’t require GPT-4-quality reasoning, we fall back to lighter-weight models.

Example:

• Intent classification? Use distilled BERT.
  
• Backend data validation? Use structured tool calls, not open-ended prompts.
  

This hybrid approach keeps costs low and response times tight.

Real-time data integration

LLMs don’t know your backend. So we use:

• Tool calling → The agent asks for what it needs.
  
• Validated backend responses → Prevent hallucination.
  
• Dynamic prompt injections → Tailor replies with real customer context.
  

Future work: integrate more consumer-specific inferences from PIE (Prodigal Intelligence Engine)  so that every consumer interaction is further personalized.

Scalability lessons

We’re onboarding multiple customers from our industry. Here's what’s working:

• A vanilla agent that’s easily fine-tuned with customer-specific config.
  
• Shared base models, isolated logic layers per tenant.
  
• Continuous latency tracking for each deployment → auto-alert if thresholds are breached.
  

Summary

Low-latency AI voice agents are possible. But they don’t come from chasing better models. They come from:

• Better and thoughtful architecture
  
• Smarter fallback logic
  
• Ruthless latency audits
  

Build for real-time first. Everything else can come later.

We're hiring rockstar engineers. Join Prodigal