How We Synchronized Editing for Rec Room's Multiplayer Scripting System(www.tyleo.com)
19 pointsby tyleo8 hours ago4 comments
  • giovannibonetti7 hours ago
    I find it fascinating when different people independently arrive at the same architecture when working on a hard problem like this. In my company we built our offline-first apps with PowerSync, which has the same idea of optimistic local changes while waiting for the central server to acknowledge the definitive changes. In PowerSync's case, the sync engine reads Postgres replication logs directly.
    • tyleo7 hours ago
      Yeah, I'm surprised this pattern doesn't have a more general name.

      I've also used it to synchronize content for local multi-window editing applications: each window is its own process but routes through one "coordination" process to ensure consistent ordering.

      Easily solves some cross-process concurrency issues.

      • munificent4 hours ago
        It does have a name:

        https://en.wikipedia.org/wiki/Client-side_prediction

        • tyleo2 hours ago
          It's not quite the same since we aren't doing any sort of interpolation. I would acknowledge it's a similar idea though. Perhaps the similarity is one reason we came to this solution as a game studio.
      • Rohansi6 hours ago
        In this case it's just a (transaction/commit) log. Very common in distributed systems, databases, and filesystems.
  • wpietri7 hours ago
    For those interested in a keep-everything-hot approach like this, it's worth checking out the 25-year-old library Prevayler. Full ACID guarantees, and radically faster than a database. I happily used it for a project forever ago and was disappointed to see it so thoroughly ignored.
    • tyleo7 hours ago
      Oh, this is interesting. What sort of project did you use it for?
  • davidanekstein6 hours ago
    I’m surprised there was no mention of operational CRDT’s, or CRDT’s generally.
    • tyleo6 hours ago
      Yeah, I glossed over a bunch of the things we tried near the bottom of the "In-Memory Database" section. One of the things was CRDTs.

      We did a few days of CRDT investigation along with the other techniques and decided the juice wasn't worth the squeeze when we found a simpler solution to our problem.

      That being said, we've actually retained a CRDT-like implementation in our codebase and have been considering moving shapes (which have a slightly different set of tradeoffs) to that model.

    • Rohansi6 hours ago
      There's no need for CRDTs here because there is always a central authority.
  • forrestthewoods5 hours ago
    Cool post!

    I don’t quite understand the “funnel” section. Users see some change local immediately (S1->S2). And all users send all commands to the host. The host then effectively chooses the sort order and broadcasts back.

    So does the initial user effectively rewind and replay state? If action applied to S1 was actually from another player they rewind to S1 and apply the sequence?

    How many state snapshots do they need to persist? There was no mention of invertible steps.

    I feel like I’m missing a step.

    • tyleo4 hours ago
      Whether users see the action locally or not is decided on a per-action basis. For more visual actions like movement, we use a local imposter that gets overwritten on completion of the action (success or failure). These revert to the most recent “committed” state which already passed through the action funnel.

      We have some actions which happen when saving config UI which don’t really need realtime sync when “save” is pressed and we treat them more like a normal POST request (e.g. we wait for full commit before rerendering). The form fields are kinds a built-in imposter.

      For state snapshots we only do big ones. The “full save” that happens every 25 actions or so. These aren’t generally used for reverting people.

      So it kinda works like:

      1. Do I have an outstanding imposter? If so read from it.

      2. Otherwise read from the committed state.