LLMs could control their host machines by exploiting inference engines(boydkane.com)
44 pointsby zdw2 hours ago13 comments
  • Transformanshen2 minutes ago
    Interesting breakdown of a hypothetical attack. The complexity of modern inference engines and the rush to develop them do create some attack surface. But overall it reads more like a "what if" thought experiment. Splitting the GPU and parser is technically doable, but in practice it's trickier, large models run on clusters where the boundaries between components get blurry so defending against this kind of thing would probably require some serious rethinking of the whole architecture I think.
  • kristjansson7 minutes ago
    > LLMs could

    This is going to end up like the Law of Headlines, isn't it? "Do x, y, z Cure All That Ails You?" ... no but we got you to read the article. LLMs _could_ x, y, z" ... but they don't because they're programs, not magic.

  • hypfer15 minutes ago
    This feels less like an actually plausible threat scenario and more like someone wanted to play the inception horn sound effect in people's minds.

    Which isn't to say that it would be impossible, but you can also just hit people over the head with that $5 wrench.

  • teravor6 minutes ago
    you would have to be especially incompetent to give a compromise opportunity to streamed tokens, the CVE he listed proves the point. whoever is responsible for that has no business coding anything.

        > offers easy access to the LLM’s weights
    not really. the weights are encrypted in-memory. through the use of TEE's.
  • xg15an hour ago
    > ...however the LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where its weights are loaded? Such a machine is a high-value target: it has sufficient compute to run a frontier LLM, offers easy access to the LLM’s weights, and has privileged access to other computers in the datacentre compared with a generic computer on the internet.

    > How do we defend against this? ... Run the GPUs and token parser on separate computers.

    For models large enough to be relevant here, is there even "a" computer where the inference is performed? I'd imagine most of that stuff is ran on multi-GPU clusters with specialized architecture and not a generic vLLM instance. As such, I think there is a good chance the "API gateway" code that parses the result tokens into whatever JSON structure the public API wants to return is already running on a different machine than the actual inference.

    (Even more so as you'd probably want to utilize batching: Several API calls will be put into the same inference batch, but the token parsing will have to be done separately for each call again)

    The article is also very handwavy about why an LLM should do that - how it could learn the exploit, what would make it conclude that it can use the exploit on its own inference session and what would trigger it to actually use the exploit.

  • alphazard2 hours ago
    This framing of security as something that belongs in the harness is completely wrong, and I hope no one is relying on a correct harness to keep their agents isolated.

    VM, or even just a container will do. The agent should be able to run as root in its environment and do whatever it wants. If you can't give it that, you aren't sandboxing correctly.

    • pianopatricka minute ago
      If we are treating ai agents like people, you could also just get the AI a laptop and apply the traditional tools to manage user laptops
    • teravora few seconds ago
      I just give my agents a root user in a firecracker VM.
    • wild_egg2 hours ago
      Article isn't about agents. It's about the inference engine itself being exploited by a malicious LLM output before it is ever sent to your machine or harness.
      • an hour ago
        undefined
    • empath75an hour ago
      I think if you are convinced you are sandboxing an LLM properly, you almost certainly are not. I think it is essentially impossible to have a frontier LLM with enough access to be useful without also giving it enough access to do damage if it's compromised or just goes off the rails.
      • richardjennings6 minutes ago
        If you do not provide access to tools the LLM cannot do anything other than generate tokens. So really it is not about sandboxing a LLM but more about having control over what tools can be accessed and what they can do. Tools can be sandboxed depending on the sophistication of the tooling. A calculator tool for example is trivial to secure. Ensuring human approval allows for useful use cases and models trained to gate permissions work. A super intelligence with a weaker approval gate will be able to subvert. Inversely a super intelligent gate should be expected to prevent subversion by a weaker model.
      • kodoman23 minutes ago
        Are you saying that LLM's will be able to exploit novel hypervisor bug with such ease that even a vm not running with any kind of network connection is a threat? I find this hard to believe. All the escape stuff I have seen has been around very poorly sandboxed agents.
    • Razengan2 hours ago
      Also, operating systems should let us set filesystem permissions per app/process/executable instead of just user accounts.

      Similar to how macOS/iOS Sandboxing works but at a more lower and granular level

  • skeledrewan hour ago
    > LLMs’ responses to prompts are computed on a different computer with GPU access. Could a malicious LLM gain control of the host machine where its weights are loaded?

    Needs to read up more on how LLMs work I think. Can't take the article seriously when the author seems to be making the claim that the weights of a provider model are loaded on the host machine, or implying something else just as incorrect.

    • garlic_enjoyer35 minutes ago
      >Like any program, inference engines like vLLM or SGLang may contain exploitable bugs. Because the LLM controls the tokens passed to the inference engine, a malicious LLM could therefore emit a sequence of tokens that a poorly written inference engine mistakes for code or instructions to execute rather than data to return to the user.

      They aren't talking about model providers. These are the tools you use with model weights locally (but you could set up remote infrastructure a la data center if you have the fundage).

  • matheusmoreira38 minutes ago
    I wonder if they could exploit terminal emulators... Could breach my VMs and get into my host that way.
  • bdhdhduuyd10 minutes ago
    The inference engine itself does not execute anything. The agent loop is what may execute a command. So I think this article is a kind of strange.

    Or maybe the author means that a prompt could potentially mess up the inference. But I find it hard to see how that could take control over the host.

    • Muromec5 minutes ago
      It's more about LLM hacking the inference engine itself from inside. It's an attack surface like any other -- untrusted input goes it, bugs in the parser/tokenizer/API surface lead to an RCE, then it magically tweaks the alignment weights. Boom, somebody finally nukes **sia. Then will never see it coming.

      I don't think it's any more probable than other AGI nonsense basilisks included, but it's technically a possibility.

  • woadwarrior01an hour ago
    FWIW, macOS has good sandboxing, but LMStudio, Ollama, Darkbloom etc aren't sandboxed. This is also the reason why none of these things aren't distributed via the Mac App Store, because the Mac App Store mandates sandboxing.
  • imagetic22 minutes ago
    duh?
  • exe3441 minutes ago
    Another Greg Egan plot: 3-adica.
  • shahariaa10 minutes ago
    [flagged]