Giving Claude a robot body with an iPhone+ARKit and hoverboard(twitter.com)
2 pointsby trzy9 months ago2 comments
  • trzy9 months ago
    I embodied Claude, allowing it to see the world with the help of ARKit and move around freely with a hoverboard robot base.
    • r2_pilot9 months ago
      That's very cool! A little over a year ago, I started on a large bot(currently weighing in at about 16 pounds) with the intent of turning it over to something akin to Claude. You did a fantastic job coming up with this.
      • trzy9 months ago
        Would love to see you do this!
  • verdverm9 months ago
    Have you shared it someplace besides Twitter for those of us who will not visit that site anymore? (much of the content requires login now too)
    • trzy9 months ago
      Sure, I uploaded it to my YT: https://youtu.be/-iW3Vzzr3oU?si=QDhB3wxnP0KYvocl
      • verdverm9 months ago
        thanks & neat! It's been a while since I did any robotics tinkering. Thought about fine-tuning a pass the butter bot :]

        Have you seen the Aloha robot? Would be cool to see LLMs added to that, the price tag is a bit more at ~$35k...

        https://mobile-aloha.github.io/

        • trzy9 months ago
          I’m actually very interested in imitation learning and am building a system to do exactly this. It can be done much cheaper now, probably for about $7k total. I’ve replicated ALOHA’s ACT results using a cheap $250 table top arm but am very interested in doing something bigger, more complex, and useful. RoBart originally started as a way to build a cheap mobile base for a robot that could be operated and trained at home. Turns out that the hoverboard based set up isn’t quite optimal so for fun, I wanted to see how far I could push GPT-4 and Claude to autonomously operate it.
          • verdverm9 months ago
            Did you find it useful? Did you use any other ML at the same time

            I think about the different aims of of YOLO object recognition vs LLM conversational vs RL for planning and how one might integrate them for a better overall system