An SLM trained on $8 ESP32-S3(github.com)
36 pointsby pavelai7 hours ago8 comments
  • runtime_lens31 minutes ago
    Sometimes the proof of concept isn't the product. It's the constraints it exposes that end up influencing more practical systems.
    • wuschel13 minutes ago
      I understand that you are alluring that maximum training-state memory, not parameter count, is the key variable here. So you better start with the smallest model, and go to for highest training data quality, with the outlook of coupling systems together?
  • prplxd_nihilist34 minutes ago
    > Which is no small thing.

    I think it is a small thing.

  • chicken-stew5 hours ago
    Would probably be better to demonstrate by exsmple how this approach is used to train on sensor data and then use it (as is hinted by the author) instead of acknowledging that the klingon poc is useless.
  • dannyw6 hours ago
    Very cool project! Sounds like it was a fun challenge :)

    I wonder when we'll start seeing clusters of ESP32-S3s... not sure how interconnects would go though, but I guess the interconnect wouldn't be the bottleneck anyway.

    • fsniper2 hours ago
      Afaik there are already a few videos on YouTube.
    • ReactiveJelly5 hours ago
      Imagine a Beowulf cluster of those
  • pavelai7 hours ago
    This small Klingon speaking language model was trained completely on ESP32. The training took 2 days

    Number of parameters: 319K

    (Disclaimer) The models is tiny and is not a pocket chatbot. It mistakes and is not capable to support conversation, but that's not the goal of the project

    The goals of the project is to bring training to edge devices and it worked out

    How can one use it? By using solar panels such device could be turned into autonomous meteorological station

  • andai5 hours ago
    Very cool project.

    > Backpropagation (gradients derived by hand)

    What does by hand mean in this context?

    Also how did you write the readme? It's a curious blend of human and AI writing.

    • porridgeraisin3 hours ago
      The fact that by hand is emphasized so often and so often (see src/handgpt.h comments as well above backward()) makes me think it's AI.

      Anyways, it looks like gradients derived by hand means they didn't use autograd. They have written out the expression for dL/dW themselves.

      • wikisailoran hour ago
        Correct. No autograd: the expressions for the gradients are written out explicitly in C. There's a gradient check in tests/ that verifies them against centred finite differences on the published header, worst relative error 1.07e-08.
      • 4gotunameagain2 hours ago
        I think the readme is clearly LLM generated. The tone, the short sentences, the em dashes, lists, sections.. It all feels like LLM.
    • wikisailoran hour ago
      [dead]
  • 12373bd5 hours ago
    hIngan motlh puS ruq tuq DujDaq SISwI' nge'vI' vIn SuvwI' yIvwI' qarghtaHvIS SIchoH
  • an hour ago
    undefined