Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning(arxiv.org)
35 pointsby binyu4 hours ago1 comment
  • plastic-enjoyer2 hours ago
    > scaling to 1T parameters significantly enhances sample efficiency and performance ceilings;

    Man, I find SOTA deep learning somewhat hilarious. We scale models to absurd proportions, burning through a shitload of resources just to achieve (slightly above) human intelligence. The human brain has a few billion neurons and uses as much power as a light bulb.

    • pfdietz2 hours ago
      The human brain is estimated to have approaching 10^11 neurons (most of them in the cerebellum).

      However, a neuron is much more than a single parameter. The brain is estimated to have from 10^14 to 5x10^14 synapses.

      • fallingfrog42 minutes ago
        True although a lot of those neurons and synapses are in the cerebellum, responsible for motor coordination and or in the visual cortex and so forth. Only a portion are in the language and reasoning areas. LLM's are comparable to human scale now, i think, and if trends continue will swiftly pass us by in the future.

        If I had a magic button I would not only pause AI development but set it back 10 years. Sadly I have no influence on events and those who do, don't care about the future of humankind or actively wish us dead.

    • aziis982 hours ago
      When for the training part you have to consider brains had like billions of years to develop. Maybe one of the reasons llms seem to be so expensive to train is because we are "compressing" in far less time that learning part
      • ziofillan hour ago
        I think it’s more accurate to say that the mechanism that grows a brain had millions of years to develop.
      • an hour ago
        undefined
    • mapontosevenths20 minutes ago
      This is the worst they will ever be.

      In 1956 a 5mb hard drive shipped on a large truck and took a team of men to unload. It consumed huge amounts of power, and cost about $3,200/month to run. In today's dollars that would be about $160,000 per month.

      Aren't you glad we didnt just give up because it was kind of expensive?

    • esikich2 hours ago
      Similarly, all the components in an audio amplifier are super dumb because you can just have 4 guys play in your living room amirite
    • Eridrus2 hours ago
      Do you find cars similarly hilarious?
    • retinaros41 minutes ago
      everything in LLM world is that right now.
    • yieldcrv2 hours ago
      the active layers concept seems to be experiencing convergent evolution to how synapses in a human function

      huge parameter models with many small but efficient layers can work quickly on low resource hardware

      similar to how neurons experience chemical spiking to activate small portions of the brain at once