Laya (OS Jev) on Mac M4 CoreML Offline (45 decisions per second)(gist.github.com)
97 pointsby putna5 hours ago7 comments
  • imranqan hour ago
    Based on my admittedly limited research, it seems like you should use Laya for much more deterministic tasks where you have some training data. It won't be as good as Jev for zero shot cases.
    • jwpapian hour ago
      I think it can make a lot of sense to make a set with jev and then train laya on that set and do the rest with laya for saving $
    • putnaan hour ago
      agree, too good to be true for one shot cases
  • PaulRobinson4 hours ago
    Local LLMs are the future, and one of the reasons I think the data centre furore is just going to end in a market crash.

    LLMs that can reliably be used for control problems are the future, and I think classic/deep RL has generally been overlooked for years for a whole host of problems by wider industry because it felt inaccessible. The first thing I thought of when I saw Jev (and then Laya), was "this might move the needle in a really, really interesting way".

    Local LLMs that can reliably be used for control problems smash through a lot of barriers I'm interested in, and this intrigues me a lot. Guess I'm about to become a big Laya fan if it can run on this kind of hardware to this performance.

    • ipsi2 hours ago
      The future for whom? The general public? Not a chance, no way, not unless it's able to run on a phone (anywhere from 20-40% of internet users, world-wide, are phone-only).

      For companies? I think that's a lot more plausible, as that's mostly just a question of money - is it cheaper to run and administrate our own models, or outsource that?

      For technically inclined users? I think that's unlikely unless they're able to operate on relatively cheap hardware while still being just as good as the hosted models. And by that I don't mean "a mac studio," that's far more money than I think is reasonable. A single RTX 5080, maybe, once memory prices start to drop.

      • Izmakian hour ago
        Compare the games your average high-end smartphone can run to the AAA titles of the 2010's. It's not a matter of "unless it is able to" but "when it is able to".
    • frag4 hours ago
      that's not a local LLM. If it's local, it doesn't matter in this case. Laya is a System 1 "AI", namely works like a classifier, given a state and questions, it shoots probabilities for each. I publish an episode tomorrow about Laya and Typesafe AI on https://www.youtube.com/@DataScienceatHome

      Stay tuned ;)

      • putnaan hour ago
        cool, will check it
    • bigyabai3 hours ago
      It won't. Laya is a finetuned version of Google's BeRT model, which is almost 10 years old right now.

      If BeRT had any potential to disrupt the datacenter buildout, it already would have.

  • altano2 hours ago
    How much memory does this use of the test machine's (M3 Max) 128 GB unified memory?
    • WASDxan hour ago
      The model is only 0.3B params so probably not much.
    • putnaan hour ago
      Physical footprint: 560.4M Physical footprint (peak): 778.0M
  • frag4 hours ago
    Great job! Did you finetune your own Laya for the snake game or what?
    • putna36 minutes ago
      its just an example how to run it locally. it takes the laya from official repo with the provided snake example. you can try running it in 2 minutes
  • tentacleuno4 hours ago
    This looks like a local AI model playing Snake -- is that correct? The article offers no explanation.
  • oezi4 hours ago
    [dead]
  • aidiveyt3 hours ago
    [dead]