142 pointsby bemmu4 months ago22 comments
  • pama4 months ago
    This was a nice demonstration of letting these robotic hands learn to play a keyboard. The technique is limited due to constraints of the hands, and is closer to parts of the technique of early keyboard instruments like an organ or a harpsichord, rather than that of a modern grand piano, which requires a lot more control of the body core, shoulders, elbows, arms, and wrists with the fingers doing as minimal a motion as possible. I suppose a similar algorithm learning to play on a grand piano using a full humanoid body could learn a technique that would be exciting to analyze.
    • djtango4 months ago
      Tangential but your comment has just given me a penny drop moment that I can isolate each degree of freedom (finger, wrist, elbow, shoulder) and still play a key to sound a note.

      For context I have been working on playing multiple lines of music with right hand lately (Chopin etudes, much struggle...) and saw a video from a professional recommending playing the main melody "with the arm". This has helped a lot with the visualisation and now I can really have a cleaner technique playing the accompaniment above the escapement (using nice flat fingers) and the melody with a much richer tone (using the arm typically for the pinky).

      I sort of understood in an intuitive nonverbal sense what I am doing but after your comment, the mechanics now makes a lot more sense of what I'm actually doing - thanks!

      • pama4 months ago
        Glad it helps. Claudio Arrau thought of the fingers as “dead” with all the piano playing happening using larger muscles. It is a struggle to internalise this concept in some Chopin etudes (op 10, no 2 especially) but it almost immediately helps in others (op 25, no 1).
        • bondarchuk4 months ago
          Interesting, I've read similar things about calligraphy/writing (from the elbow!), hadn't connected it to piano playing.
      • cbtones4 months ago
        Yes, that's where this model is massively missing, an accomplished pianist plays with the whole arm, the elbow is crucial, it needs to stay very relaxed in order to accommodate the peculiarities of the human hand, the ring fingers and little fingers don't have as many tendons connecting and having a little sway in the elbows is crucial to making those fingers deliver the weight of the arm properly to the keybed. This model appears to ignore rotation of the wrist, and easing the elbows, where at a bare minimum it needs to start at the shoulder, and I'd argue the whole body.
        • djtango4 months ago
          Yeah but as gp pointed out that was the style du jour of baroque and early classical as the plucking keyboards were mostly played with a finger technique
    • vunderba4 months ago
      This. In fact isolating and attempting to play purely from the fingers alone is a great way to develop RSI or any number of soft tissue injuries.

      There's entire schools of thought around proper ergonomics with respect to piano playing - I took lessons in the Taubman technique which (very simplistically) tries to encourage movement farther upstream on the body.

      https://www.ednagolandsky.com/the-taubman-approach-basic-pri...

  • kbouck4 months ago
    Any way we can get the Westworld intro song into the demo list?
  • ericye164 months ago
    This was the basis of a project I did for my deep reinforcement learning class!

    https://ericye16.com/stanford-cs224r

    We were able to make some improvements by tuning how the reward is distributed and also by first pretraining the agent on scales before fine-tuning them on the final pieces.

    Thanks to Kevin Zakka for helping us get started with the RL environment!

    • plaguuuuuu4 months ago
      did you guys ever try having the agents play the song slower at first?
      • ericye164 months ago
        We definitely tried extending the lookahead, but I don't think we tried having a curriculum-style thing where we gradually increased the speed of the song. Great idea though!
  • kittikitti4 months ago
    > UC Berkeley, Google DeepMind, Stanford University

    I've personally experienced how research around this time was being shut down because of AI doomerism. People were getting laid off because of it. It's clear to me that these institutions actively spread AI doomerism so that they have full control over it. They actively called for a stop in AI research so that their personal labs can leap forward ahead. It was a little too on the nose for Big Tech, but they don't understand nuance.

  • criddell4 months ago
    Is this actually buildable? I'd like to hear it on a real piano because (IMHO) the one in the videos sounds bad.

    I'd like to also hear how loud the mechanical noise of the machine playing the piano would be. Does the left hand work harder with the heavier keys? What would the hands be mounted to?

    • dheera4 months ago
      > We use the Musical Instrument Digital Interface (MIDI) standard to represent a musical piece as a sequence of time-stamped messages corresponding to "note-on" or "note-off" events. A message carries additional pieces of information such as the pitch of a note and its velocity.

      I play the piano and I think MIDI does not have enough parameters to describe an acoustic piano. It's not just a single strike velocity that determines each sound. Where you play within the double escapement, how far down you hit the key, they all change the sound a bit.

      That said though there are better MIDI synthesizers, e.g. https://www.modartt.com/ They still don't match an acoustic. I can tell the difference quite easily. I think a neural TTS retrained on piano data could do better.

    • cloudbonsai4 months ago
      > Is this actually buildable

      Yes. The robot used in this study is Dexterous Hand from Shadow Robot. It's a real product, and costs around $200-300k (for a single hand).

      Controlling a real robot using a RL policy (model) trained in a simulation environment is also doable. It's called "Sim2Real" and has been widely experimented in the last decade (with mixed success, though).

    • j_bum4 months ago
      What do you mean “heavier keys”?

      Each key (on a good piano) is weighted to have the same force requirements to be played. Of course varying forces can be used to achieve different decibels, but the curve is the same across the piano.

      • ano-ther4 months ago
        Probably this:

        > Every single key on a grand piano keyboard is weighted differently. This is because the strings for each note are slightly thinner and shorter in the treble register, becoming thicker and longer towards the bass register. As a result, there is greater resistance when playing low notes than when playing high notes. In other words, a heavier touch is required in the left hand and a lighter touch in the right hand. To emulate this in a digital piano, the keys are often individually weighted, with the lower keys heavier than the higher ones — something that’s called graded weighting.

        https://hub.yamaha.com/pianos/p-digital/a-quick-guide-to-wei...

        • cbtones4 months ago
          It's not the strings though, the hammers are smaller as you go to the right in the higher register on the piano. The strings are slightly smaller in diameter, but remarkably shorter in length.
  • rossant4 months ago
    Nice work! And great interactive 3d application. My 6yo had a lot of fun annoying the robot while it's playing by forcefully moving its hands around.
  • 4 months ago
    undefined
  • beardyw4 months ago
    All the right notes but not necessarily at the right time.
    • kzrdude4 months ago
      Which is exactly the wrong way around from what's expected for a pianist :)
  • Tycho4 months ago
    Has anyone tried to get a robot to do oil paintings?
  • fedeb954 months ago
    waiting for Robot Devil.
  • davidanekstein4 months ago
    Funny enough I’m listening to Rhapsody in Blue while browsing HN. I’d like to see it do that for 17 minutes.
  • marbro4 months ago
    How does this research help us win naval battles? Should Elon and his DOGE friends cut it?
  • gtirloni4 months ago
    Imediatelly reminded of the anonymous quote: "I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do laundry and dishes."

    I understand it's a bad argument because 1) people may need assistance to do art/writing and 2) the advancements gained from teaching AI to do art can be applied to other non-artistic endeavors (e.g. piano AI could be really good to operate machines with lots of buttons and no computerized interface).

    However, the cost cutting side of the argument is the one that bothers me because companies/people WILL use AI like that in place of actual humans because they're likely to be cheaper in the future. So that pianist or musician playing in a local restaurant can be sure their job will be automated away by a subpar AI and real humans will be relegated to very expensive locations (an extension of replacing humans with recorded music, in a way).

    My pessismist side thinks greed will be the downfall of humanity.

    • bluGill4 months ago
      The pianist playing in a restaurant was optimized by the player piano (which would only play if you put a nickel in) back in the late 1800s. Muzak put the final nail in that coffin in 1954.

      The player piano technology comes from the barrel organ, which seems to have been on the first pipe organs.

    • throw805214 months ago
      I think if ones perspective of manual art changes, so does that of AI art, sorta like shifting priorities in life. Some priorities are higher than others. And past a certain point, AI art becomes a tempting option beneath a certain motivation level.

      If for example you find manual art more fun to practice than AI art, you'd be less inclined to give AI companies money to generate art, since you could perhaps spend hours and hours practicing and have fun.

      And of course if you don't find manual art enjoyable, but still don't want to give up on the "art" idea entirely, AI art crosses the threshold into enjoyability, so the end product is at least in front of your eyes.

      As for how to learn how to make one type of art more fun than the other, if ones preferences are set in stone, it's pretty tough. The AI/non-art-inclined crowd might invert your quote so the laundry and dishes get prioritized, because different people have different priorities.

    • ebiester4 months ago
      I agree, but live performance in the age of recording is already a luxury good, and it is unlikely an AI would replace live performance any more than a player piano or CD player would (or has.)
      • dingnuts4 months ago
        I can't see how in a world full of recordings how in person, live performance art could get anything but vastly more valuable.

        I saw a stage performance of Clue a few weeks ago. It was lovely. Nothing was AI generated, and couldn't be, because it was people physically making art live. How nice.

        I will continue to pay for those experiences and to prioritize them over recorded ones.

        • gtirloni4 months ago
          No doubt it'll be more valuable but the number of opportunities (to watch and play at those) will decrease globally.
    • numpad04 months ago
      People want robots do (low-value && computationally intensive) tasks - I think it's fair and okay by now to say unfolding wet clumps came out of a washing machine belongs to a class of tasks that are computationally intensive for human brains.
    • fipar4 months ago
      Your comment reminded me of a scene from the movie "El Mariachi". In it, the title character walks into a bar looking for a job as a Mariachi, and is faced with the bar owner telling him something like "Why do I need a Mariachi when I can have a full band?", then pointing to a guy resting in the corner next to an organ (an "electronic organ", of the family type, as opposed to, you know, a hammond or something like that). The resting guy presses the "demo" button and the organ plays Mariachi music on its own ...

      I share your pessimism partially, but I also wonder: if the system keeps optimizing for profit, at some point there just won't be enough people in the world to sell products and services to. I don't know what the answer is.

      Edit: typo

    • MantisShrimp904 months ago
      The only real answer here.
  • iwontberude4 months ago
    Wasted opportunity to not play the Westworld Theme
  • wbakst4 months ago
    the fact that the failure modes come from limitations of the robotic hands is super interesting
  • torusle4 months ago
    Honestly,

    this is really bad. It might be a breakthrough of what you are doing, but when I listen to the output all of the timing and phrasing is aweful.

    • scientaster24 months ago
      Alright then, post your ML simulation of a piano player, I'll wait :)
    • deskr4 months ago
      [flagged]
  • amelius4 months ago
    I mean, keys are always in the place where you expect them to be, so what is the challenge here? (Yes, too lazy to read the article).
    • cloudbonsai4 months ago
      > what is the challenge here?

      If you know the precise state of each motor in a robot hand, you can compute the absolute position of its fingertip, right?

      To make a robot play piano, you need to solve the reverse problem of it i.e. to compute the motor states from the target robot position. This problem is called "Inverse Kinematics" and not fully resolved to this day.

      Reinforcement Learning (RL) people claim that, if you let a robot move randomly, and give it apporopriate rewards depending on the resulting state, eventually it learns to solve the problem by itself (just like a human baby eventually learns how to walk through trial & error).

      Now, you are looking at the state of the art of RL in 2023.

      • upbeat_general4 months ago
        The challenge here is not IK. Sure, you don’t need IK if your policy directly operates in joint space but this isn’t the motivation for doing RL.

        IK to my knowledge is well known in every setting I am aware of.

        • cloudbonsai4 months ago
          > The challenge here is not IK.

          No, a state-based RL research like this is essentially an IK problem. Given a goal position in the world frame, you need to find out the motor configuration to move your EEF to that goal position.

          > IK to my knowledge is well known in every setting I am aware of.

          Really? When I was working in this field, I've actually never seen anyone who used numerical/analytical IK methods on real robots.

          Granted that I was not a Robotics engineer (I was a Deep Learning engineer in the team), but my impression at the time was that no practical IK solution was available for a robot with 8 DOF, let alone a 20 DOF robot like Shadow Hand.

          • upbeat_general4 months ago
            Here’s the simple reason why this is not the case:

            If you provide the 6DoF trajectory (+ gripper joints), a lot of robotics (manipulation in particular) is basically solved. The problem is, we don’t have these good trajectories.

            Sure, joint space is commonly used for learned policies, but cartesian space isn’t uncommon either.

            IK is really just not a major focus on the learning side of robotics because it’s not the problem. The problem is we don’t know what to do even at the slightly higher level.

    • smokel4 months ago
      The idea is that the motor control is pretty hard to program, and by using reinforcement learning, one only has to tell the computer whether it hits the right keys (reward).

      The reinforcement learning algorithm takes in the state of the fingers and piano keys, and outputs the required motor actions to perform well.

      Also, as you are too lazy to read an article but still feel entitled to comment on it, I herby give you -1 reward. Let's see if that helps to improve your character.

    • recursive4 months ago
      Where's yours? That's probably the best way to find all the challenges.
    • rossant4 months ago
      I think the robot learnt to play by itself, right?
  • pacificmaelstrm4 months ago
    "Player Piano"
  • cytocync4 months ago
    [dead]
  • PatronBernard4 months ago
    The thing needs a teacher, terrible technique! /s
    • bongodongobob4 months ago
      I think this is worth mentioning for people who don't play piano. It's technique is terrible and bizarre. It's flicking the keys with its fingernails.
    • rossant4 months ago
      It plays almost as badly as I do.
  • AIFounder4 months ago
    [dead]
  • linwangg4 months ago
    [dead]