A General Goal-Conditioned Minecraft Model(pantograph.com)
33 pointsby agajews5 hours ago5 comments
  • agajews4 hours ago
    Hey everyone! I'm Alex, one of the founders of Pantograph. We've spent the last six months building a pretty smart Minecraft model, coming soon to a server near you!

    We trained it on about 500k hours of Minecraft videos, and it learned how to fight creepers, build walls and other structures, and explore to find visual goals.

    We're considering putting up a public API for larger models like this one, let us know if you'd like to be able to put Pan in your own server :)

    What's most interesting about the model isn't the performance that it gets in Minecraft, but how general the method is. When we scale it up, it should be able to act in any kind of video game, as well as robots in the real world (which are really just another video game).

    • ryanhecht33 minutes ago
      Hi Alex! I'm Ryan, I run a Minecraft theme park recreation server. While our gamemode is very far from vanilla Minecraft (it plays more like a MMORPG), I've been interested in standing up a model like this to help automate agentic testing of new software features in my monolith Bungee/Paper codebase.

      In its current state, do you think it can handle non-vanilla Minecraft tasks like interacting with custom UI's, running commands, or doing other unusual tasks in the world?

      • agajews27 minutes ago
        Probably not the current model, but one of the benefits of doing internet-scale pretraining is that the model has seen a lot of mods already! We think the bigger models will be able to handle some more custom Minecraft worlds.
    • superb_dev2 hours ago
      I would love to be able to add one of these bots to my Minecraft server! I’ve got a family server and we’ve been talking about how to get an AI in there for a while. I tried out some open source harnesses to allow a generic LLM to join, but none of theme were particularly good.
      • agajews26 minutes ago
        Awesome! We should be a lot better than ordinary LLMs, especially at tasks that require making a lot of decisions in real-time.
  • xnx4 hours ago
    Reminds me of Google's AI Dreamer that could mine diamonds in Minecraft (https://www.scientificamerican.com/article/google-deepmind-t...) or OpenAI training on video to play Minecraft (https://openai.com/index/vpt/)
    • agajews4 hours ago
      Very much inspired by those papers! One of the things that's interesting about our model is it's goal-conditioned, so it can do any task at inference time without training on it. We had a lot of fun making eval environments after we trained the model trying to find interesting things it can do, and that was all after we trained the model. More like prompting an LLM.

      (Versus Dreamer, which needs to be trained on a hand-written reward function for each task that you want to do.)

      • SirHackalotan hour ago
        How does this compare to non-stochastic, non-ML AI in games currently?
        • agajews24 minutes ago
          Pretty different--obviously a lot less consistent today, but capable of a lot more diverse kinds of behavior. As the models get bigger, they'll be easier to prompt, and you could imagine a game designer writing a long prompt for different kinds of NPCs. (E.g. GPT-3 was pretty difficult to get to do a particular thing, although it was very general, but over the years the instruction tuning has gotten a lot better and now it's very easy to ask questions to ChatGPT)
  • codeulike4 hours ago
    Does it have language skills? I always thought it would be interesting to train a model within minecraft as a sortof proxy for 'embodiment'. You could then try asking it about its experiences. "whats your favourite food?", "How does it feel when you hear a spider?", "how low does your food bar need to go before it feels really urgent?" etc
    • agajews4 hours ago
      Not yet, but we'll add language as a modality to the larger models! The models are trained end-to-end on video data, so we'll need datasets that mix video and language, e.g. transcripts of game streams. When the models are scaled up to cross-videogames and robots there will definitely be a bunch of language data.
  • nee1r3 hours ago
    how do you pick good goal conditioning images/do you have to hand pick a dataset of good goal images? seems hard if you don't have full context. really cool though!
    • agajews3 hours ago
      you don't need to pick good images manually! it's similar to next-token prediction, many prediction tasks aren't especially interesting, but there are enough hard ones that the model can spend the most time learning from those. the simplest thing scaled up works very well.
  • Onavo3 hours ago
    I disagree with the approach. It's a good approach for limited domain problems, but not for general purpose. Take something like this where you will need to be able to refer to wikis and research and ask questions on Reddit and Discord to optimize playthrough, none of the goal conditioning will be useful.

    https://www.feed-the-beast.com/modpacks/125-ftb-evolution

    I think a properly fine-tuned VLA with access to tool calls can scale way better.

    • agajews3 hours ago
      The goal conditioning is just a training objective! You could combine it with tool use, web searches, etc. and still train end-to-end on goal conditioning. (One way of thinking about it is goal conditioning defines a good distribution of tasks, and lends itself well to pretraining with an unlabelled video dataset.)