Hunting Down a Go Runtime Bug on 32-Bit Embedded Systems(sigma-star.at)
117 pointsby birdculture6 days ago6 comments
  • dlopes73 days ago
    Very interesting read, these bugs that only happen after a lot of time in production are the worse to troubleshoot
  • ReactiveJelly3 days ago
    So a stronger type system in the boundary between Go and Linux would have caught this?
    • fn-mote3 days ago
      My reading is that this is entirely Go’s fault. In Go, there was a union of two different types without metadata to distinguish the two. The disambiguation was done by looking to see if the data looked like a certain pointer.

      I think the reason it is a problem only in 32 bit systems is that the assignment setting the lower half of the data is a 32 bit op, which can overflow the 16 bits used, but won’t affect the other half of a 64 bit value. (Corrections welcome.)

      • inigyou3 days ago
        It's 64 bits on both but only 32 bits were read on 32-bit systems. Two 64-bit values that were different were considered the same because only 32 bits were checked.
  • nasretdinov3 days ago
    Go was born into 64-bit era, I remember seeing many more bugs that only occur on 32-bit systems. I'm pretty certain the observation in the article is correct that no one at Google tests Go on anything apart from 64-bit ARM/x86 Linux
    • typical1823 days ago
      FWIW, every commit for the Go project is tested on multiple 32-bit architectures, including pre-commit tests and post-commit tests.

      You can see a list here:

      https://build.golang.org/

      (That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)

      I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.

      • orphereus3 days ago
        Very cool site. Makes me want to do some system programming project just to make a site with all the tests like this.
      • znpy3 days ago
        interesting that they build for solaris as well
  • budman13 days ago
    Isn't this something that software developers do everyday?
    • apitman3 days ago
      The way I look at these types of difficult debugging problems has completely changed.

      If it's a system I want to understand more deeply and I have the time, I try to force myself to do it by hand. If not, the agent does it.

      The trap is that it's easy to convince myself that I never have the time.

      • budman13 days ago
        Not being a user of agents, my question is 'does the agent always find the root cause and remove the defect'?

        The way I got good at finding the hard ones was finding hard ones. Without that practice, I would be helpless.

        • k4rli3 days ago
          Reason why it's super hard for juniors at the moment. For "agents" to produce good results, the operator must have the experience to direct it in the right direction and properly validate the output.

          I've had some weird bug investigations where it has suggested wrong fixes confidently multiple times before arriving at the actual fix. Only with extra details from manual testing did we arrive at the real bug. The ~5 first suggested fixes would've hidden the root cause even more.

        • apitman2 days ago
          I've found agents very effective at both idebtifying root causes from vague descriptions, and explaining both the problem and the fix to me.

          It's not a panacea though. Sometimes it comes up with bad causes and/or solutions.

  • BlackRabbit13 days ago
    Not directly related to your embedded Go bug, but maybe helpful:

    What really sky-rocketed my debugging skills in embedded devices was learning to utilize the Python plugin interface of 'gdb'.

  • DarmokTanagra3 days ago
    I wonder what other 32 bit bugs are waiting to be discovered.