104 pointsby onurkanbkrc5 hours ago12 comments
  • amstan2 hours ago
    Missing (probably because of the date of the article): `mv --exchange` aka renameat2+RENAME_EXCHANGE. It atomically swaps 2 file paths.
    • rustyboltan hour ago
      I tried using this a while back and found it was not widely available. You need coreutils version 9.1 or later for this, many distros do not ship this.

      I made https://github.com/rubenvannieuwpoort/atomic-exchange for my usecase.

    • oguz-ismail22 hours ago
      Title says Unix, renameat2 is Linux-only.
      • jasode2 hours ago
        >Title says Unix,

        You're misinterpreting the title. The author didn't intend "Unix" to literally mean only the official AT&T/TheOpenGroup UNIX® System to the exclusion of Linux.

        The first sentence of "UNIX-like" makes that clear : >This is a catalog of things UNIX-like/POSIX-compliant operating systems can do atomically,

        Further down, he then mentions some Linux specifics : >fcntl(fd, F_GETLK, &lock), fcntl(fd, F_SETLK, &lock), and fcntl(fd, F_SETLKW, &lock) . [...] There is a “mandatory locking” mode but Linux’s implementation is unreliable as it’s subject to a race condition.

  • 0xbadcafebee4 hours ago
    You can use `ln` atomicity for a simple, portable(ish) locking system: https://gist.github.com/pwillis-els/b01b22f1b967a228c31db3cf...
    • akoboldfryingan hour ago
      Really nice explanation of a useful pattern. I was surprised to discover that even the famously broken NFS honours atomicity of hardlink creation.
  • Igrom36 minutes ago
    >fcntl(fd, F_GETLK, &lock), fcntl(fd, F_SETLK, &lock), and fcntl(fd, F_SETLKW, &lock)

    There's also `flock`, the CLI utility in util-linux, that allows using flocks in shell scripts.

    • cachius30 minutes ago
      What are flocks in this context? Surely not a number of sheep...
  • ncruces25 minutes ago
    I use several of these to implement alternative SQLite locking protocols.

    POSIX file locking semantics really are broken beyond repair: https://news.ycombinator.com/item?id=46542247

  • zzo38computer3 hours ago
    Even though it can do some things atomically, it only does with one file at a time, and race conditions are still possible because it only does one operation at a time (even if you are only need one file). Some of these are helpful anyways, such as O_EXCL, but it is still only one thing at a time which can cause problems in some cases.

    What else it does not do is a transaction with multiple objects. That is why, I would design a operating system, that you can do a transaction with multiple objects.

    • ptx2 hours ago
      Windows had APIs for this sort of thing added in Vista, but they're now deprecating it "due to its complexity and various nuances which developers need to consider":

      https://learn.microsoft.com/en-us/windows/win32/fileio/about...

    • akoboldfryingan hour ago
      I don't follow, sorry. Are you saying that if we run:

          mv a b
          mv c d
      
      We could observe a state where a and d exist? I would find such "out of order execution" shocking.

      If that's not what you're saying, could you give an example of something you want to be able to do but can't?

      • jstimpflean hour ago
        I don't think that's happening in practice, but 1) it may not be specified and 2) What you say could well be the persisted state after a machine crash or power loss. In particular if those files live in different directories.

        You can remedy 2) by doing fsync() on the parent directory in between. I just asked ChatGPT which directory you need to fsync. It says it's both, the source and the target directory. Which "makes sense" and simplifies implementations, but it means the rename operation is atomic only at runtime, not if there's a crash in between. It think you might end up with 0 or 2 entries after a crash if you're unlucky.

        If that's true, then for safety maybe one should never rename across directories, but instead do a coordinated link(source, target), fsync(target_dir), unlink(source), fsync(source_dir)

  • MintPaw3 hours ago
    Not much apparently, although I didn't know about changing symlinks, that could be very useful.
  • sega_sai4 hours ago
    rename() is certainly the easiest to use for any sort of file-system based synchronization.
  • andrewstuart25 minutes ago
    Anywhere there is atomic capability you can build a queuing application.
  • ta89033 hours ago
    Not technically related to atomicity, but I was looking for a way to do arbitrary filesystem operations based on some condition (like adding a file to a directory, and having some operation be performed on it). The usual recommendation for this is to use inotify/watchman, but something about it seems clunky to me. I want to write a virtual filesystem, where you pass it a trigger condition and a function, and it applies the function to all files based on the trigger condition. Does something like this exist?
    • direwolf2022 minutes ago
      are you asking for if statements?

      if(condition) {do the thing;}

    • Brian_K_White3 hours ago
      incron
  • maximgeorge2 hours ago
    [dead]
  • exac4 hours ago
    Sorry, there is zero chance I will ever deploy new code by changing a symlink to point to the new directory.
    • sholladay4 hours ago
      Why? What do you prefer to do instead?
      • gib4443 hours ago
        Anything less than an entire new k8s cluster and switching over is just amateur hour obviously
    • iberator4 hours ago
      why? it works and its super clever. Simple command instead some shit written in JS with docker trash
    • bandrami3 hours ago
      Works pretty well for Nix
      • atmosx2 hours ago
        Worked pretty well in production systems, serving huge amount of RPS (like ~5-10k/s) running on a LAMP stack monolith in five different geographical regions.

        Just git branch (one branch per region because of compliance requirements) -> branch creates "tar.gz" with predefined name -> automated system downloads the new "tar.gz", checks release date, revision, etc. -> new symlink & php (serverles!!!) graceful restart and ka-b00m.

        Rollbacks worked by pointing back to the old dir & restart.

        Worked like a charm :-)

      • mananaysiempre3 hours ago
        And for Stow[1] before it, and for its inspiration Depot[2] before even that. It’s an old idea.

        [1] https://www.gnu.org/software/stow/

        [2] http://ftp.gregor.com/download/dgregor/depot.pdf

        • bandrami2 hours ago
          I really liked stow. My toy distro back in the day was based on it.
    • slopusila3 hours ago
      that's how some phone OSes update the system (by having 2 read only fs)

      that's how Chrome updates itself, but without the symlink part

      • dizhn2 hours ago
        No snapshotting at all? Thinking about it.. The filesystem does not support it I suppose.
      • x41322 hours ago
        not surprised about the chrome part, but pretty shocked at the phone OS part. I know APFS migration was done in this way, but wouldn't storage considerations for this be massive?
        • slopusila2 hours ago
          what would be more massive would be phones not booting up because of a botched update. this way you can just switch back to the old partition
        • marmaramaan hour ago
          Not really, because only the OS core is swapped in this way. Apps and data live in their own partitions/subvolumes, which are mutable and shared between OS versions.

          The OS core is deployed as a single unit and is a few GB in size, pretty small when internal storage is into the hundreds of GB.

    • alpb3 hours ago
      Nobody's saying you should deploy code with this, but symlinks are a very common filesystem locking method.
  • klempneran hour ago
    This document being from 2010 is, of course, missing the C11/C++11 atomics that replaced the need for compiler intrinsics or non portable inline asm when "operating on virtual memory".

    With that said, at least for C and C++, the behavior of (std::)atomic when dealing with interprocess interactions is slightly outside the scope of the standard, but in practice (and at least recommended by the C++ standard) (atomic_)is_lock_free() atomics are generally usable between processes.