You can also avoid the problem all together. I made a "speculative regeneration" at least thats what i call it pipeline i been using for many months now and it works great. basically as long as you have access to the inference server you are good to go. idea is as such, every n milliseconds of silence (usually 300ms) for me, you send the asr transcript for the llm to process immediately. you let the model start answering , but if the user starts talking again before the end of turn token is reached you cancel the generation and again wait for the 300ms. rinse repeat. there is a safety window you have to set up as well that way the final llm answer gets finalized regardless of its inference speeds. works like a charm. basically the user gets 300ms latency at the highest for that "silence" window. so the only other thing you have to worry about as faras latency is llm inferance speeds and asr and tts latencies. in my pipeline the moment the user stops talking, the model ,is already responding within about 900ms - 1.2 sec.. its my gemma 26b moe model , 4 bit quant thinking on... with mtp getting about 300 tokens per second... asr and tts are both under 350ms for total... anyways, speeds could be improved if turning off thinking but i like my voice agent to be smart as he has tool calling and common sence capability is reduced when thinking is off.... fun times!

This is the 3rd or so time I've posted this -

Run a tiny shitty llm, one that is super fast. Or better yet try out a decision model now days.

Feed it the previous conversation turn and all the STT (ASR output) you have so far and just ask it for a probability that the user is done speaking.

Use that probability to shorten your VAD time.

It isn't as good as a true speech to speech model but this will cut a few hundred ms compared to pure VAD. Oh as a bonus it also lets you wait longer when the user isn't done talking yet.

[flagged]

undefined