133 pointsby gidellav5 days ago17 comments
  • flpm4 days ago
    And check this one, which is a generalization of the Datasaurus where you can define your own shapes :D

    https://github.com/stefmolin/data-morph

    • moi23884 days ago
      From now on I won’t trust any statistic unless I can transform it into a panda.
  • djoldman4 days ago
    • sunrunner4 days ago
      Content warning: This is a baker’s dozen not a regular dozen, in case anyone clicks through expecting to find twelve and is mildly and briefly perturbed.
    • djoldman4 days ago
      The scary thing is that yea we can see these in 2D and maybe 3D. But ...

      usually there are more than 2 or 3 columns in our data :(

      • imurray4 days ago
        It's clearly hard, but there are tools for doing exploratory visualization of high-dim data. GGobi http://ggobi.org/ and all the ones that arrange points but try to get local neighborhoods correct (t-sne, umap, et al.).
  • jihadjihad4 days ago
    Often there is little or no substitute for plotting the data to see how it is distributed. A scatter plot, histogram, density plot, etc. is almost always going to tell you a "story" about the data that the summary stats will have compressed.

    But sometimes you are at the mercy of the data and your visualization of choice. Box plots, for example, are great at showing more than just how the data is centered, but it is possible to encounter situations where the box plots of the data remain static while the underlying data is clearly changing [0].

    As always it is good to know about these things and continue to add to the arsenal (violin plots, in the example above) of tools and intuition needed to tease out the story behind the data.

    0: https://www.research.autodesk.com/publications/same-stats-di...

  • __mharrison__4 days ago
    I teach curve fitting with this dataset and recently added the fifth dataset. It illustrates Simpsons paradox.

    https://www.linkedin.com/posts/panela_loved-adding-ancombes-...

    • aleyan4 days ago
      That's an amazing addition! Once I read about Simpson's paradox[0], couldn't help but seeing it or suspecting it everywhere. Luckily, it is not a true paradox, and it can resolved if underlying data is available and not just summary statistics.

      I recommend putting together the Quintet in one image, so that the original 4 charts, plus the new one are all visible and interpretable together. It will be learning aid for decades to come.

      [0] https://en.wikipedia.org/wiki/Simpson's_paradox

      • __mharrison__4 days ago
        Yes, not saying the data dinosaur isn't cool. But for real-world applications, the quartet with the addition of this fifth dataset is more useful for pedagogical purposes.
  • joshdavham4 days ago
    During my statistics degree, Anscombe’s Quartet was used as an example of why you should always try to visualize your dataset and not just run your calculations blindly. I’m a bit odd in that I don’t care much for data viz, but Anscombe’s Quartet really shows how important it is in practice.
  • jkyrlach4 days ago
    This dataset is definitely a treasure, and I love visualizing data. That said, i think what's missed when this is used as an argument for visual analysis is the idea of quantitatively identified outliers. If you take the descriptive statistics of p99, they most definitely will not be the same across these four sets. Visual analysis is a valuable dimension for data exploration, but it's a bit of a strawman to infer that "quantitative analysis could go no further, only visual analysis could figure this out"
  • divbzero4 days ago
    I know this is against the main point of Anscombe’s Quartet but just curious: Could skewness or other summary statistics differentiate the four distributions?
    • dccsillag3 days ago
      Take enough moments and you'll be able to differentiate any distributions.
  • padraigf4 days ago
    I love it. I was introduced to it by Edward Tufte's book, 'https://www.amazon.co.uk/Visual-Display-Quantitative-Informa...'.

    And was just thinking about it the other day. I had a bug aggregating sleep-data from an iPhone, which comes in the form of sleep-samples.

    I was trying to fix it, both by prodding Claude Code to fix the problem, and looking at debug logs of the sleep-samples, but we weren't getting anywhere. I asked Claude Code to graph the samples, and BAM, saw it right away. (the problem was that HealthKit returns you sleep-samples from ALL devices, not just the priority one)

    Maybe not exactly the same thing as Anscombe/Tufte were getting at, but I was reminded of it, and the value of visualising data.

  • Mithriil3 days ago
  • efavdb4 days ago
    The example shows that the usual stats aren't enough to pin down the true data. But in practice I imagine / wonder if these stats really are reasonable "sufficient stats" because the probability of seeing data with strong structure is unlikely in most contexts. In other words...

    p(data | stats) = p(stats | data) * p(data) / p(stats).

    and p(data) is only strong for a "blob / cloud" of points, so when there's some correlation the observed stats tell you that you likely have a blob having some degree of correlation.

    • aredox4 days ago
      >But in practice I imagine / wonder if these stats really are reasonable "sufficient stats" because the probability of seeing data with strong structure is unlikely in most contexts.

      We just spent five years since COVID appeared to argue about statistics, with tons of bad analysis of very complicated data fuelling political rage up to this day.

      The US health secretary is currently using data with "strong structure" to deny vaccines and to falsely pin down convenient targets for everything from cancer to autism.

  • throw0101d4 days ago
    Thought this would be about the 'other' Anscombe:

    * https://en.wikipedia.org/wiki/G._E._M._Anscombe

    :)

  • bluesmoon3 days ago
    I did a talk on Cognitive Biases in performance measurement and included Anscombe's Quartet (among other things) in the section on developer bias: https://speakerdeck.com/bluesmoon/we-love-speed-understandin...
  • INGELRII4 days ago
    Always visualize first. Human 'eyballing' is a good pattern detector.

    Linear correlation is just one pattern the data can have.

    Unfortunately many social science publications have reviewers who know only the basics and can't judge or accept statistically valid analysis that is outside their competence. Fit it into line or nothing.

  • ryukoposting4 days ago
    I do STEM mentoring for high school kids. Bookmarking this, because it'll be a great teaching aid at some point.
  • WhitneyLand4 days ago
    This reminds that “visualize while thinking” will probably become an important part of reasoning as we move closer to AGI models.

    This will require improvements to vision models, RL frameworks, etc, but will be interesting to see how much it can broaden current abilities.

  • curtisszmania4 days ago
    [dead]