157 pointsby g0xA52A2A4 days ago18 comments
  • mfworks4 days ago
    The most magical part of this transform is the search! First learned about this in a bioalgorithms course, and the really cool property is that for a string length l, you can search for the string in O(l) time. It has the same search time complexity of a suffix tree with O(n) space complexity (with a very low constant multiple). To this day it may be the coolest algorithm I've encountered.
    • dgacmu4 days ago
      I encountered the search version of this, which is turned suffix arrays, in grad school and was so taken by them I incorporated them as the primary search mechanism for the pi searcher. 25 years later it's still the best way to do it. Incredible insight behind bwt and suffix arrays.
    • dcl4 days ago
      A friend doing bioinformatics told me about this at uni, it was definitely one of those "i can't believe this is doable" sort of things.
  • pixelpoet4 days ago
    I remember randomly reading about this in the Hugi demoscene diskmag as a young teen, and it completely blew my mind (notably the reverse transform, apparently not covered in OP's article): https://hugi.scene.org/online/coding/hugi%2013%20-%20cobwt.h...

    The author later wrote many now-famous articles, including A Trip Through the Graphics Pipeline.

    • sequin2 days ago
      Same; any time I see BWT I think of Fabian Giesen. I think he was only 16 or 17 when he wrote that article.

      By chance I once sat near him and the rest of Farbrausch at a demoparty, but I was too shy to say hi.

    • rollulus3 days ago
      Oh wow, I clicked that link without reading the last section of your comment, and was like “Fabian Giesen!”, an absolute HN favourite: https://hn.algolia.com/?q=fgiesen
  • hinkley4 days ago
    Noteworthy that the paper that was published describing this technique came at a time when IP lawyers had begun to destroy the world wrt to small business innovation. And that they released it unencumbered is a huge debt of gratitude that we haven’t really honored.
  • Imnimo4 days ago
    An interesting bit of trivia about the Burrows-Wheeler transform is that it was rejected when submitted to a conference for publication, and so citations just point to a DEC technical report, rather than a journal or conference article.
    • Qem3 days ago
      What were the reasons given, if any? Are they publicly known?
      • Imnimo3 days ago
        I only know the story from word-of-mouth. My understanding is that the submitted paper wasn't written/presented well, and reviewers had trouble understanding the significance of what was being proposed. But take that story with a big grain of salt.
  • moribvndvs3 days ago
    For some reason I was really getting lost on step 2, “sort” it. They mean lexicographically or “sort it like each rotation is a word in the dictionary, where $ has the lowest character value”. Now that I see it, I’m a little embarrassed I got stuck on that step, but hopefully it helps someone else.
    • tetris113 days ago
      That trips up a lot of people, plus most examples include a '^' character at the beginning which is assigned the highest value
  • lancefisher4 days ago
    This is a good article and a fun algorithm. Google made a great series of compression videos a while ago - albeit sometimes silly. For the Burrows-Wheeler one they interviewed Mike Burrows for some of the history. https://youtu.be/4WRANhDiSHM
    • emmelaich4 days ago
      I wonder if he came up with it when pondering permuted indexes. Which used to be a feature of the man pages system.
  • username2234 days ago
    Does anyone else remember when Dr. Dobb's Journal wrote about stuff like this? https://web.archive.org/web/20040217202950/http://www.ddj.co...
  • foobarian4 days ago
    The unintuitive part of BWT to me was always the reverse operation, which was the only thing the post didn't give intuition for :-)
    • hinkley4 days ago
      Most BWT descriptions: now draw the rest of the owl.

      Once upon a time I found one that did the entire round trip, I neglected to bookmark it. So when I eventually forgot how it worked, or wanted to show others, I was stuck. Never did find it.

    • SimplyUnknown3 days ago
      I'm still not sure I get it. I think it is

      1. Put the BWT string in the right-most empty column

      2. Sort the rows of the matrix such that the strings read along the columns of the matrix are in lexicographical order starting from the top-row????

      3. Repeat step 1 and 2 until matrix is full

      4. Extract the row of the matrix that has the end-delimiter in the final column

      It's the "sort matrix" step that seems under-explained to me.

      • foobarian3 days ago
        I think what did it for me is to recognize that the last column has characters that, for each row, come before the characters in the first column (since they are rotations). So we can view the last column as coming "before" the first one.

        1. We sorted the first column to get the BWT column. Thereby we created the structure where the BWT column comes before the sorted column.

        2. Therefore if we insert the BWT column before a sorted column, the order of row elements is preserved

        3. If we now sort again, the order of characters across individual rows is again preserved

        4. Going to step 2 again preserves row order

        5. Once all columns are populated, therefore all rows are in the correct original order. And thanks to the end marker we can get the original string from any row.

    • raboukhalil4 days ago
      Good point, thanks! I'll add a subsection about the intuition for that.
    • kragen3 days ago
      The algorithm given in this page for the reverse transform is not usably efficient; the algorithm given in the original paper is, and I illustrated it in my explorable explanation of it linked in my top-level comment here. It was a pretty surprising insight!
  • raboukhalil4 days ago
    Author here, nice to see the article posted here! I'm currently looking for ideas for other interactive articles, so let me know if think of other interesting/weird algorithms that are hard to wrap your head around.
  • kvark3 days ago
    I've spent years playing with BWT and suffix sorting at school (some work can be found be names of archon and dark). It's a beautiful domain to learn!

    Now I'm wondering: in the era of software 2.0, everything is figured out by AI. What are the chances AI would discover this algorithm at all?

    • mrkeen3 days ago
      No need to speculate about this algorithm in particular. It's been a few years since every programmer has had access to LLMs (that's a huge sample size!). Some LLMs are even branded as 'Research'. Plus they never get tired like humans do. We should be swimming in algorithms like this.
  • kragen3 days ago
    This page looks very nice indeed!

    This doesn't follow the original paper very closely, although I think the device of putting an end-of-string marker into the data instead of including the start index as a separate datum is an improvement over the original algorithm. My own "explorable explanation" of the algorithm, at http://canonical.org/~kragen/sw/bwt, is visually uglier, and is very similar to this page in many ways, but it may still be of some interest: it follows the original paper more closely, which is likely to be useful if you're trying to understand the paper. Also, I used an inverse-transform algorithm that was efficient enough that you could imagine actually using in practice, while still being easy to explain (from the original paper)—but, like the page linked, I didn't use an efficient suffix-array-construction algorithm to do the forward transform. I did link to the three that have been discovered.

    I forget when I did this but apparently sometime around 02010: https://web.archive.org/web/20100616075622/http://canonical....

    I think my JS code in that page is a lot more readable than the minified app.js served up with this page.

  • zahlman3 days ago
    BWT is how I first got to interact with Tim Peters on the Internet: https://stackoverflow.com/questions/21297887
  • jansan4 days ago
    For an article describing a compression algorithm this was very digestible and entertaining.
  • fair_enough4 days ago
    Thanks, man! This helps a lot, because as you say the algorithm is not intuitive. The description of the type of data it is very good for is the best value-adding part of this writeup. Greatly appreciated!
  • DmitryOlshansky3 days ago
    And I’ve just implemented BWT and Inverse BWT in D, earlier today! https://github.com/DmitryOlshansky/compd/blob/master/source/...
  • feanaro3 days ago
    This is essentially a group theoretical result about permutation groups. Would be nice to see a treatment from this angle.
  • loloquwowndueo4 days ago
    I was once mentioning this transform to someone and they were like “what? The Bruce Willis transform?” - I guess my pronunciation sucked that much.
    • Bukhmanizer4 days ago
      Once in a lecture one of my friends asked “You keep mentioning Cooper Netties, who is Cooper Netties?” Everyone was very confused until someone figured out she was asking about Kubernetes.
  • embit4 days ago
    This is great. The best article I have read on BWT and experimented was in Dr Dobbs Journal around 1996-97.