46 pointsby sacs0nia month ago2 comments
  • sacs0nia month ago
    I built this to have a dedicated wire-protocol client for postgres logical replication. General-purpose SQL clients either don't implement the replication protocol at all, or bury it behind abstractions designed for other use cases. Replication has a bit different mechanics - it's a stateful binary stream requiring LSN tracking, standby heartbeats, and feedback to prevent WAL bloat. Bolting that onto a query-focused client has its own challenges.

    This is just the transport - raw XLogData frames and LSNs. Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure.

    What it does:

    - Explicit LSN control - start/stop at specific WAL positions for deterministic recovery

    - Automatic standby feedback - no more forgotten heartbeats filling your disk with WAL

    - Bounded channels - backpressure propagates to Postgres naturally

    - Pure Rust, no libpq

    What it doesn't do: pgoutput decoding (intentionally). That belongs in a higher layer. Simplest way of using this:

    while let Some(event) = client.recv().await? { match event { ReplicationEvent::XLogData { wal_end, data, .. } => { process(&data); client.update_applied_lsn(wal_end); } _ => {} } }

    • nkmnz22 days ago
      > Use pg_replicate, as an example, if you need "replicate to BigQuery." Use this if you're building replication infrastructure.

      Would I use this if I host my own postgres and want to use replication for „real time backups“ into a hot standby?

      • radimm22 days ago
        Not OP - but definitely no. In such a case use just physical or - if you have special use case - logical replication that’s built in.
        • sacs0ni22 days ago
          I agree. pgwire-replication is useful when you need to build a customized and closely controlled pipeline. In fact, it will give you the first part of handling the data (reading from the source), you still need to implement the rest yourself.
    • malodyets25 days ago
      I learned about this tonight when Claude Code picked up your library for my application that uses logical replication. Looking forward to putting it through its paces.
      • sacs0ni24 days ago
        nice! would appreciate any feedback.
    • jsjfusua22 days ago
      [flagged]
  • gunnarmorling22 days ago
    Nice one, great to see this addition to the Rust ecosystem!

    Reading through the README, this piqued my curiosity:

    > Small or fast transactions may share the same WAL position.

    I don't think that's true; each data change and each commit (whether explicit or not) has its own dedicated LSN.

    > LSNs should be treated as monotonic but not dense.

    That's not correct; commit LSNs are monotonically increasing, and within a transaction, event LSNs are monotonically increasing. I.e. the tuple commit-LSN/event-LSN is monotonically increasing, but not LSNs per se. You can run multiple concurrent transactions to observe this.

    • sacs0ni22 days ago
      Good catch, you are correct. I did mix a few things there and the statements were incorrect or at least very misleading.

      To demo your point I created a gist, for myself and others to see the (commit-LSN, event-LSN) ordering in action:

      https://gist.github.com/vnvo/a8cf59fc3cd8719dbea56d3bb5201f9...

      I'll update the readme to reflect this more accurately. Appreciate you taking the time to point it out.