Show HN: I built a sub-500ms latency voice agent from scratch(www.ntik.me)
72 pointsby nicktikhonov2 hours ago12 comments
  • armcat43 minutes ago
    This is an outstanding write up, thank you! Regarding LLM latency, OpenAI introduced web sockets in their Responses client recently so it should be a bit faster. An alternative is to have a super small LLM running locally on your device. I built my own pipeline fully local and it was sub second RTT, with no streaming nor optimisations https://github.com/acatovic/ova
    • nicktikhonov42 minutes ago
      Very cool! starred and on my reading list. Would love to chat and share notes, if you'd like
  • age123456gpgan hour ago
    Hi all! Check out this Handy app https://github.com/cjpais/Handy - a free, open source, and extensible speech-to-text application that works completely offline.

    I am using it daily to drive Claude and it works really-well for me (much better than macOS dictation mode).

  • modelessan hour ago
    IMO STT -> LLM -> TTS is a dead end. The future is end-to-end. I played with this two years ago and even made a demo you can install locally on a gaming GPU: https://github.com/jdarpinian/chirpy, but concluded that making something worth using for real tasks would require training of end-to-end models. A really interesting problem I would love to tackle, but out of my budget for a side project.
  • lukaxan hour ago
    Or you could use Soniox Real-time (supports 60 languages) which natively supports endpoint detection - the model is trained to figure out when a user's turn ended. This always works better than VAD.

    https://soniox.com/docs/stt/rt/endpoint-detection

    Soniox also wins the independent benchmarks done by Daily, the company behind Pipecat.

    https://www.daily.co/blog/benchmarking-stt-for-voice-agents/

    You can try a demo on the home page:

    https://soniox.com/

    Disclaimer: I used to work for Soniox

    Edit: I commented too soon. I only saw VAD and immediately thought of Soniox which was the first service to implement real time endpoint detection last year.

    • nicktikhonovan hour ago
      If you read the post, you'll see that I used Deepgram's Flux. It also does endpointing and is a higher-level abstraction than VAD.
      • lukaxan hour ago
        Sorry, I commented too soon. Did you also try Soniox? Why did you decide to use Deepgram's Flux (English only)?
        • nicktikhonovan hour ago
          I didn't try Soniox, but I made a note to check it out! I chose Flux because I was already using Deepgram for STT and just happened to discover it when I was doing research. It would definitely be a good follow-up to try out all the different endpointing solutions to see what would shave off additional latency and feel most natural.

          Another good follow-up would be to try PersonaPlex, Nvidia's new model that would completely replace this architecture with a single model that does everything:

          https://research.nvidia.com/labs/adlr/personaplex/

  • NickNaraghi2 hours ago
    Pretty exciting breakthrough. This actually mirrors the early days of game engine netcode evolution. Since latency is an orchestration problem (not a model problem) you can beat general-purpose frameworks by co-locating and pipelining aggressively.

    Carmack's 2013 "Latency Mitigation Strategies" paper[0] made the same point for VR too: every millisecond hides in a different stage of the pipeline, and you only find them by tracing the full path yourself. Great find with the warm TTS websocket pool saving ~300ms, perfect example of this.

    [0]: https://danluu.com/latency-mitigation/

  • docheinestages42 minutes ago
    Does anyone know about a fully offline, open-source project like this voice agent (i.e. STT -> LLM -> TTS)?
  • boznzan hour ago
    "Voice is an orchestration problem" is basically correct. The two takeaways from this for me are

    1. I wonder if it could be optimised more by just having a single language, and

    2. How do we get around the problem of interference, humans are good at conversation discrimination ie listing while multiple conversations, TV, music, etc are going on in the background, I've not had too much success with voice in noisy environments.

  • perelinan hour ago
    Great writeup! For VAD did you use heaphone/mic combo, or an open mic? If open, how did you deal with the agent interupting itself?
    • nicktikhonovan hour ago
      I was using Twilio, and as far as I'm aware they handle any echos that may arise. I'm actually not sure where in the telephony stack this is handled, but I didn't see any issues or have to solve this problem myself luckily.
  • loevborgan hour ago
    Nice write-up, thanks for sharing. How does your hand-vibed python program compare to frameworks like pipecat or livekit agents? Both are also written in python.
    • nicktikhonovan hour ago
      I'm sure LiveKit or similar would be best to use in production. I'm sure these libraries handle a lot of edge cases, or at least let you configure things quite well out of the box. Though maybe that argument will become less and less potent over time. The results I got were genuinely impressive, and of course most of the credit goes to the LLM. I think it's worth building this stuff from scratch, just so that you can be sure you understand what you'll actually be running. I now know how every piece works and can configure/tune things more confidently.
  • MbBrainz2 hours ago
    Love it! Solving the latency problem is essential to making voice ai usable and comfortable. Your point on VAD is interesting - hadn't thought about that.
  • CagedJean36 minutes ago
    Do you have hot talk when you are alone in the shower with HER?
  • jangletown2 hours ago
    impressive