Sad to Say: An AI Creativity Test (The Billy Joel Test)
2 pointsby daly5 hours ago1 comment
  • codingdave4 hours ago
    Being able to follow a formula to generate specific types of content is kind of the opposite of creativity. If you really want a creative benchmark, you need to have an LLM devise a completely new structure/formula, create works with it, and benchmark whether or not audiences respond positively. Copying an existing formula is not going to prove a thing.
    • ben_w3 hours ago
      Half agree. Also, there's already some tests like this:

      • https://www.sciencedaily.com/releases/2026/01/260125083356.h...

      • https://creativitybenchmark.ai

      But the "half" is that in practice, "creativity" is finding the border zone between novelty and familiarity. Every time I've been praised for it in my life I've had the inside view on what inspired me, and once I'd seen enough culture I started to be able to spot the sources of inspiration in much other work too. How Star Trek is inspired by mixing cold war naval manoeuvres, the age of exploration, and John W. Campbell's "Islands of Space", and how Islands of Space itself feels like Jules Verne with less autism (Verne had a lot of lists) and more civilising-mission smugness.

      Go too far outside the border zone and you get the same initial reaction as "Danse macabre" by Saint-Saëns ("horrible screeching from solo violin" causing widespread feelings of anxiety). It took familiarity for it to be seen as it is now, "one of Saint-Saëns' masterpieces, widely regarded and reproduced in both high and popular culture" to quote Wikipedia.