73 pointsby jcalabro3 hours ago8 comments
  • tapoxia minute ago
    I don't really understand this architecture, but I thought Bluesky was distributed like Mastodon? How can it have an outage?
  • threecheese3 hours ago
    > What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request.

    That’ll do it.

    • 98codes2 hours ago
      Ahh, the three relevant numbers in development: 0, 1, and infinity.
    • bombcar2 hours ago
      Zero, one, many, many thousands.
    • htx80nerd12 minutes ago
      less than ideal if I had to be frank.
  • goekjcloan hour ago
    > The timing of these log spikes lined up with drops in user-facing traffic, which makes sense. Our data plane heavily uses memcached to keep load off our main Scylla database, and if we're exhausting ports, that's a huge problem.

    I expect this is common.

  • electrondood18 minutes ago
    Great write up... curious about the RCA. Thanks!
  • rvzan hour ago
    Thank you for the post mortem on this outage.
  • templar_snow2 hours ago
    [flagged]
  • jmclnxan hour ago
    Lite Blue on a dark Blue background. That is a new one, I have seen grey text on lite grey, but blue on blue ?

    The article does work in lynx, at least I can read it.

  • jonstaab39 minutes ago
    nostr never goes down
    • pfraze37 minutes ago
      All support to other decentralizers but nothing never goes down.
      • jonstaab13 minutes ago
        1000x redundancy makes it vanishingly unlikely. Although I know we're due for a pole shift so all bets are off I suppose.