240 pointsby Hugsun3 days ago15 comments
  • Sharlin2 days ago
    A good way to conceptualize what’s going on is not the idea that balls become "spiky" in high dimensions – like the article says, balls are always perfectly symmetrical by definition. But it’s the box becoming spiky, "caltrop-shaped", its vertices reaching farther and farther out from the origin as the square root of dimension, while the centers of its sides remain at exactly +-1. And the 2^N surrounding balls are also getting farther from the origin, while their radius remains 1/2. Now it should be quite easy to imagine how the center ball gets more and more room until it grows out of the spiky box.
    • ColinWright2 days ago
      But there is another way to think of the high-dimensional balls where "spikey" is the right visualisation.

      Consider the volume of a cap. Take a plane that is 90% of the distance from the centre to the edge, and look at what percentage of the volume is "outside" that plane. When the dimension is high, that volume is negligible.

      And when the dimension is really high you can get quite close to the centre, and still the volume you cut off is very small. In our 3D world the closest thing that has this property is a spike. You can cut off quite close to the centre, and the volume excised is small.

      The sense in which a high-dimensional ball is not a spikey thing is in the symmetry, and the smoothness.

      So when you want to develop an intuition for a high dimensional ball you need to think of it as simultaneously symmetrical, smooth, and spikey.

      Then think of another five impossible things, and you can have breakfast.

      • dawnofdusk2 days ago
        I think our geometric intuition could never be good for a high dimensional object. Consider a sphere in 3D. It's represented by the points which satisfy x^2 + y^2 + z^2 = 1. Because there are only three coordinates, knowing the value of one of them greatly reduces the possibilities for the other two. For example, if I know z is close to 1, then my point is close to the north pole (x and y are both close to zero).

        However, if I have the n-sphere x_1^2 + x_2^2 + ... + x_n^2 = 1, knowing the value of x_1 gives me very little information about all the other coordinates. And humans' interaction with the geometry is reality is somewhat limited to manipulating one coordinate at a time, i.e., our intuition is for built on things like moving our body linearly through space, not dilating the volume or surface area of our bodies.

        • saagarjha2 days ago
          If x_1 is close to one, then the more x_n values there are the closer they are to zero.
      • Hugsun2 days ago
        That's a good point. High dimensional objects can obtain very unintuitive properties, like you describe.

        This to me feels similar in many ways to how a corner in a high dimensional n-cube, although 90 degrees, no matter how you measure it, seems extremely spiky. As the shape does not increase in width, but the corners extend arbitrarily far away from the center. A property reserved for spiky things in 3D.

    • pfortuny2 days ago
      Exactly: a corner of a square covers 1/4 of that part of the plane. A corner of a cube covers 1/8, a corner of a hypercube in dimension n covers just 1/(2^n) of the space. But each side/face/hyperface divides the plane/space/n-dim space just in half.
      • beretguy2 days ago
        Ok, now i started to understand something. Thank you.
    • pfortuny2 days ago
      In some sense, a ball is more “natural” than a cube in Euclidean n-space. Once you add the metric, cubes become artificial constructs (despite being the natural elements in just product spaces).
    • blaufuchs2 days ago
      TFA does say this too, right after the Hamming lecture.

      "So instead of considering n-balls to be spiky, it’s the space around them that outgrows them."

      • Sharlin2 days ago
        Yes, but that doesn’t yet give me much intuition, so I wanted to elaborate on an idea I’ve found somewhat useful.
      • Hugsun2 days ago
        To be clear: the commenter you replied to just seems to be reiterating the idea, so I'm not accusing them of not reading the article.

        You should have seen how few replies read the last article I posted. https://news.ycombinator.com/item?id=40525629 The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.

        • Sharlin2 days ago
          "Space is growing around the ball" just by itself doesn’t give much of an intuition at least to me, so I just wanted to elaborate on a way to think about it.
          • Hugsun2 days ago
            Yep, thanks for the great contribution!
        • ColinWright2 days ago
          > The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.

          I have thoughts about this, but I can find no way to contact you to open a conversation. You might want to think about adding something to your HN profile.

          Meanwhile ... from this and other sources, I feel your pain.

        • gcanyon2 days ago
          I read the article and the threads (to a point). It's hilarious that in the article you describe LLMs as being like hasty people jumping to conclusions, and the commenters on HN do exactly that thing in their comments.
    • petters2 days ago
      Very good to see this as the top-voted comment. I completely agree that this seems like a more natural explanation of what is going on.
    • jheriko2 days ago
      even then the edges do not suddenly curve. its just all round a bad analogy.
  • steventhedev3 days ago
    This is a really good demonstration of the curse of dimensionality[0]

    [0]: https://en.m.wikipedia.org/wiki/Curse_of_dimensionality

    • ocean_moist2 days ago
      Interesting how this relates to LLMs scaling laws.
  • drdeca3 days ago
    Why did I imagine that this would be about two shapes that are merely topologically n-balls, each having part of their boundary be incident with one of the two hemi(n-1)-spheres of the boundary of an n-ball (and otherwise not intersecting it)? (So like, in 3D, if you took some ball and two lumps of clay of different colors, and smooshed each piece of clay over half of the surface of the ball, with each of the two lumps of clay remaining topologically a 3-ball.)

    I don’t know that there would even be anything interesting to say about that.

    • Hugsun3 days ago
      I can't tell you why you imagined that, but that's pretty funny nevertheless.
  • ColinWright2 days ago
    For other HN discussions of this phenomenon you can see some previous submissions of another article on it.

    That article doesn't have the nice animations, but it is from 14 years ago ...

    https://news.ycombinator.com/item?id=12998899

    https://news.ycombinator.com/item?id=3995615

    And from October 29, 2010:

    https://news.ycombinator.com/item?id=1846682

  • robwwilliams3 days ago
    Impressive, helpful, and now time to rebuild my own embeddings so I can grasp that red n-ball with my new n-D hands.
  • bt1a3 days ago
    I am struggling to juggle the balls in my mind. Are there any stepping-stone visual pieces like this to hopefully get me there? Very neat write-up, but I can't wait to share the realized absurdity of the red ball's green box eclipsing in our 3D intersection of the fully diagonalized 10D construct
    • pfortuny2 days ago
      The hypercube is the strange thing, not the red sphere. Placing the blue spheres tangent to the hypercube is an artificial construct which only “bounds” the red sphere in small dimensions. Our intuition is wrong because we think of the problem the wrong way (“the red sphere must be bounded by the box”, but there is no geometrical argument for that in n dimensions).
      • badmintonbaseba2 days ago
        How does the enclosed sphere's radius changes with the number of dimensions, if the enclosing spheres are the following:

        2D: 3 mutually touching 2-spheres (circles)

        3D: 4 mutually touching 3-spheres (or spheres)

        ...

        This variation of the problem doesn't rely on an artificial construct of a hypercube, I wonder if this yields a similarly unintuitive result.

        • badmintonbaseba2 days ago
          If my calculations are correct, then for this variation the enclosed n-sphere's radius converges to sqrt(2)-1 from below, and remains enclosed in the bounding hyper-tetrahedron.
          • pfortuny2 days ago
            Buf, you may be right but I just cannot visualize it. It took me quite a while to do for the cube, imagine a tetrahedron. But you might be right.
  • Imustaskforhelp2 days ago
    Can I just say how my mind is utterly blown by the animations
    • Hugsun2 days ago
      Thank you <3 The trigonometry involved was pretty intense at times.
  • chadhutchins103 days ago
    Anyone else click just to slide some animations?
    • Hugsun2 days ago
      This guy gets it!
  • mbowcut23 days ago
    Numberphile did a video on this a while back. https://youtu.be/mceaM2_zQd8?si=0xcOAoF-Bn1Z8nrO
  • 3 days ago
    undefined
  • joaquincabezas3 days ago
    wow discovering Hamming’s lecture was enough for me! so good
    • Scarblac2 days ago
      The whole series is on Youtube, awesome.

      The course was also the basis for his book _The Art of Doing Science and Engineering_ (1997). At first it takes some getting used to as you have the feeling it may be outdated, but it's about teaching a style of thinking. It's great.

    • blaufuchs2 days ago
      I know, I didn't realize he was alive in the 90s! Hearing him (sarcastically) say "now having 10 parameters isn't unusual, correct?" makes me wish he could've seen the 60B-parameter curve fitting we're doing nowadays.
      • Scarblac2 days ago
        The whole lecture series was about giving students a style of thinking that might hopefully prepare them for the future, without focusing on special knowledge (there were other courses for that).

        Two of the lectures were spent on building intuition for very high dimensionality (this one), and another on neural networks, because he thought there was a big chance they were going to be important. In the early 90s, not bad.

      • f1shy2 days ago
        I think he already knew something about it... it talks about AI. Maybe at the time 100k to 1M dimensions? A bright mind like his, could very good extrapolate to 2024.
    • f1shy2 days ago
      Yes sir! I've just bookmarked the series... Priceless!
  • Asraelite2 days ago
    I never understood the need to distinguish between "ball" and "sphere" in maths. Sure, one is solid and the other hollow, but why is that fact so important that you need to use a completely different word? As I understand it, you could replace every instance of "ball" in this article with "sphere" and it would still be correct.

    We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?

    • Hugsun2 days ago
      > you could replace every instance of "ball" in this article with "sphere" and it would still be correct.

      I completely agree. N-ball is maybe more mathematically precise. It might rule out some strange edge case I couldn't think of. I chose it mostly for stylistic reasons. n-ball is shorter than n-sphere, and ball is a more playful term.

      I do however understand the need to define the two object classes, and see little issue with giving them distinct names.

      • 2 days ago
        undefined
    • bmacho2 days ago
      > We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?

      Topologists don't need them because they already have ball and sphere.

      In analysis, I can imagine them calling full hyperrectangles "brick", and empty hyperrectangles "box", but both words start with "b", so there is no shorthand for them on paper. I^n and ∂I^n are just fine.

      • NeoTar2 days ago
        And, indeed, it’s probably in topology where the distinction between sphere and ball is most important.
    • NeoTar2 days ago
      We do for 2D though - Circle and disc. And in 4D plus we have the 3-sphere and 4-ball, etc.

      And for 3D shapes have Torus, and, well, ‘solid-torus’.

  • WhitneyLand2 days ago
    Both ChatGpt 4.o and Claude failed to answer

    “…At what dimension would the red ball extend outside the box?”

    If anyone has o1-preview it’d be interesting to hear how well it does on this.

    • V__2 days ago
      This was the prompt I gave o1-preview:

      > There is a geometric thought experiment that is often used to demonstrate the counterintuitive shape of high-dimensional phenomena. We start with a 4×4 square. There are four blue circles, with a radius of one, packed into the box. One in each corner. At the center of the box is a red circle. The red circle is as large as it can be, without overlapping the blue circles. When extending the construct to 3D, many things happen. All the circles are now spheres, the red sphere is larger while the blue spheres aren’t, and there are eight spheres while there were only four circles.

      > There are more than one way to extend the construct into higher dimensions, so to make it more rigorous, we will define it like so: An n-dimensional version of the construct consists of an n-cube with a side length of 4. On the midpoint between each vertex and the center of the n-cube, there is an n-ball with a radius of one. In the center of the n-cube there is the largest n-ball that does not intersect any other n-ball.

      > At what dimension would the red ball extend outside the box?

      Response: "[...] Conclusion: The red ball extends outside the cube when n≥10n≥10."

      It calculated it with a step-by-step explanation. This is the first time I'm actually pretty stunned. It analysed the problem, created an outline. Pretty crazy.

      • Hugsun2 days ago
        I'd wager that it's in the training data.
        • 2 days ago
          undefined
  • beretguy2 days ago
    I… can’t.
  • eniwnenahg2 days ago
    Matlock, is that you?