Show HN: Crystal Unified – Search compressed archives without decompressing(github.com)
1 pointby danebl10 hours ago1 comment
  • theamk9 hours ago
    Protip: if what's shown in README is your "Traditional workflow"

        # Traditional workflow: decompress, grep, wait, clean up
    zstd -d huge_logs.zst && grep "error" huge_logs && rm huge_logs
    STOP NOW! There is never need to decompress files to disk, every compressor supports streaming decompression just fine:

        # Traditional workflow: decompress + grep
    zstd -dc huge_logs.zst | grep "error"
    (and as a bonus, no need to install any third-party tools)
    • danebl8 hours ago
      You are right, that was poor wording. If we compare apples to apples:

      Traditional: must decompress entire file (even when streaming) - zstd -dc huge_logs.zst | grep "error" # 709 MB decompressed through memory

      Crystal: indexed search, skip to matches - cuz search huge_logs.cuz "error" # Only decompress matching blocks

      Thanks for pointing this up. We will update the readme file.