Garage S3: LMDB Corruption Post-Mortem(me.bertof.net)
3 pointsby bertof4 hours ago2 comments
  • 4 hours ago
    undefined
  • magicalhippo2 hours ago
    TL;DR: LMDB is known to be vulnerable to corruption after an unclean shutdown. Garage provides a built-in snapshot mechanism (metadata_auto_snapshot_interval) exactly for this scenario. I hadn’t enabled it. I also had no Btrfs snapshots on the metadata partition. If you run Garage on LMDB, enable snapshots today.

    I had a bunch of data disappear from a SyncThing share recently because SyncThing doesn't enable keeping backups of deleted files by default, and some bug caused it to think one client had deleted all the files and promptly propagated that "change".

    I'm quite annoyed by such non-safe defaults. Make the defaults as safe as reasonable. If people want performance they can dig through the settings and turn off such settings.