IBM CEO says open weight models are now irresistibly good and cheap on-prem(www.theregister.com)
78 pointsby Betelbuddy2 hours ago9 comments
  • jacquesman hour ago
    It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
    • otterley42 minutes ago
      How much did you shell out for this setup? What’s the TTFT and the output tokens/sec rate? How much electricity does it consume?
      • jacquesm36 minutes ago
        Working on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
        • glassresearch2 minutes ago
          I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.
        • selcuka6 minutes ago
          > It is not cheaper than using a frontier lab

          Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.

    • dansquizsoft33 minutes ago
      > Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)

      I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...

      • sp52720 minutes ago
        You need to reread what was written. No one is proposing to take away your right to leak proprietary edge to the rapacious frontier labs that need to eat most of the economy in order to survive.
  • jains99an hour ago
    But hardware to run them is conveniently something IBM would be happy to sell you.
    • numpad031 minutes ago
      Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
    • jacquesman hour ago
      If IBM would ship a piece of affordable hardware that can run large LLMs efficiently I'd be lining up.
      • solyin41 minutes ago
        "Affordable" and "IBM" don't belong in the same sentence together, but they're happy to sell you their inference-capable hardware if you've got the wallet for it.
        • jacquesm38 minutes ago
          I don't agree with that. They made the PC revolution possible by making hardware that was at least affordable enough to be bought by a large number of businesses and individuals. It also set the stage for the clone market. I'm pretty sure that if IBM really set their mind to it that they could do this.

          That doesn't detract from your overall point that they tend to make overpriced stuff, but I don't think they have to do that. Besides, what's overpriced? NVidia's GB300 workstation sells for 95K euros, a single RTX 6000 is now 20K. Surely even IBM can compete at that level.

          • rnd030 minutes ago
            That was completely accidental and they themselves tried to close the barn door with the PS/2 line in 1987 but at that point it was too late.
            • jacquesm19 minutes ago
              True, even so: They still make some pretty impressive hardware. They have access to fab capacity, they know how to make chips. They have the capital. If they really wanted they could do it.
          • dansquizsoft31 minutes ago
            > They made the PC revolution possible

            Yeah they were only half a decade late to the PC revolution back in the day...

            • wmf17 minutes ago
              They weren't really late. In many people's minds IBM validated the concept of the PC.
              • jacquesm13 minutes ago
                When that happened I'd been working at Tandy (EU version of Radio Shack) for a while and even though we sold computers they were not taken serious until IBM came along and then suddenly we couldn't supply them fast enough.
          • quailfarmer33 minutes ago
            They haven’t been _that_ IBM for decades
            • jacquesm25 minutes ago
              I know. But: they have gobs of money, they still have some talent, they still have a hardware department that I'm pretty sure would jump at a chance to give NV a run for their money. This would require a massive long term investment and a commitment that I'm not sure they would have the stomach for.
    • anukinan hour ago
      Not till Jensen buys his next leather jacket.
  • trollbridgean hour ago

      While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.

  “The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”
    Has he heard of “agentic AI”? You simply fire up Claude or ChatGPT and ask it how to do all of the above.
    • Buttons84041 minutes ago
      That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
    • LoganDark40 minutes ago
      Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
  • djelu35 minutes ago
    Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
  • matt321031 minutes ago
    RIP frontier AI, the top customers can easily self host
    • IncreasePosts12 minutes ago
      What happens when their competitors decide to pay for a more intelligent frontier model?
  • SV_BubbleTime8 minutes ago
    I’ve been in tech for a long time… I have no idea what IBM does. Now, I get that I’m not the customer, that their customer is “business” but not my business… no… bigger business. The biggest of businesses I assume. I think it must either be a club I’m not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because “it’s IBM, it must be important”.
  • CookieCrispan hour ago
    Good enough for the old world, but I'm not so sure it's good enough for what's coming.
    • tomrodan hour ago
      What do you think is coming that isn't already in the old world?

      Cerebras with Claude 8?

      • LoganDark38 minutes ago
        OpenAI most likely pushed Cerebras into a contractual agreement not to do business with Anthropic through 2030. ("Most likely" because the names of the companies Cerebras is contractually obligated not to sell to are redacted in public filings)
        • desolate_muffin4 minutes ago
          Imagine a regulatory framework so weak that a company feels comfortable to do something this anti-competitive
  • collinmcnultyan hour ago
    Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if it’s true. And maybe it is true! But IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.
    • nvme0n1p1an hour ago
      "CEO Said A Thing!" Journalism

      https://karlbode.com/ceo-said-a-thing-journalism/

    • epsteingptan hour ago
      But you know - it would be funny if somehow Open Weights are the thing that completely save IBM
      • SoftTalker41 minutes ago
        Linux already served as a life-preserver for them also.
    • MattDamonSpacean hour ago
      It’s a conversation point off which a discussion can be had
      • gjvc43 minutes ago
        the only credible "conversation" is "well he would say that, wouldn't he?"
    • gjvc43 minutes ago
      "we are delighted..." said a spokesman
  • buffer_overlord2 hours ago
    [flagged]