Show HN: Flide: Make Jev-like decision models up to 9.5x faster(github.com)
29 pointsby unikcc5 hours ago5 comments
  • yossuf200029 minutes ago
    do i need to host the model to do this ? if yes what model can i host in 3050 card
  • jaryman hour ago
    I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!
  • unikcc5 hours ago
    Hi HN! I'm one of the Flide team.

    We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.

    Here is the source code: https://github.com/OmniJev/Flide

    Look forward to hearing your feedback and ideas!

    • rytill2 hours ago
      Under what specific circumstances is it actually 9.5x faster?

      Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?

      How did you achieve the biggest speed up?

      Are there any tradeoffs? “While keeping exactness” implies not.

      Are there any specifics about how one must structure their context to take full advantage of it?

      • tancopan hour ago
        I looked at their paper and the biggest gains come when you have many questions from one context, which is the way these models are supposed to be used. It's around 1.5-3x faster than vLLM with single questions. That's the worst case where you can't reuse anything.

        The way I understand it most of their speedup comes from caching intermediate results. They freeze the state after processing context then branch each question off that point. It means they can process questions in parallel after the context stage is finished, and don't waste time doing the same work N times.

  • drix1011 minutes ago
    [flagged]
  • yossufyahia39 minutes ago
    [flagged]