Ask HN: Do you still run Redis and workers just for background jobs?
1 pointby sergF2 hours ago1 comment
  • figassis2 hours ago
    Use Go, it has built in go routines and likely libraries that let you implement your own workers.

    If you’re running a single instance, you don’t even need any synchronization. If you’re running multiple instances of your app, try implementing locking (this actually works in any language, not just go. Go jsut helps with the multiple long running workers part. With other languages, just run multiple instances.

    Process:

    1. Each worker can startup with their own id, it can be a random uuid.

    2. When you need to create a task, add it to the tasks table, do nothing else and exit.

    3. Each worker running on some loop or cron, would set a lock on a subset of the tasks. Like:

    update tasks set workerId = myUUID, lockUntil = now() +10minutes where (workerId is null or lockUntil < now()) and completed = false

    Or you can do a select for update or w/e helps you keep other workers from setting their ids at the same time.

    4. When this is done, pull all tasks assigned to your worker, execute, then clear the lock, and set to completed.

    5. If your worker crashes, another will be able to pick it up after the lock expires.

    No redis, no additional libraries, still distributed

    • sergFan hour ago
      Yeah, this is pretty much what I end up doing as well.

      It works, but I keep rewriting the same task table / locking / retry logic in every project, which is why I'm wondering if it makes sense to move this out into a separate service.

      Not sure if it's actually a real problem or just my workflow though.

      • figassis27 minutes ago
        I would create a library, make some logic more generic, create a generic table (task id, taskType, workerId, etc), store task metadata as jsonb so it can be pulled and marshalled into typed data by users.

        Import it into your projects.

        Make the library work standalone. But also build a task manager service that people can deploy if they want to run it outside their code.

        Then offer a hosted solution that does the webhooks.

        I’m sure someone will want to pay for it.

        • sergF4 minutes ago
          That makes sense, and this is actually close to what I keep ending up with in different projects.

          I usually start with something simple, then add a task table, then locking, retries, then some kind of worker process, and eventually it turns into a small job system anyway.

          At some point it starts feeling like I'm rebuilding the same queue/worker setup over and over, which is why I'm wondering if this should live outside the app entirely.

          Thanks, this discussion is really helpful.