Persisting task queue

[baugarten]

Hey there,

I was wondering if the queue of tasks to be performed was persisted to any kind of reliable store. Reading through the source it looks like all pending tasks would be lost on app restart or whatnot.

[jamesdabbs]

Hey! Yeah, at the moment everything's just stored in memory. I haven't touched this project in a while, but I have just been working on a Redis-aware Slackbot framework (https://github.com/jamesdabbs/replicant) that will need some sort of persistent worker queue storage in the not-too-distant future. I'd love to 1) write up a Sidekiq-compatable Redis interface and 2) extract that back out to here. It's not tops on my priority list at the moment, but if you have a particular need / though / are interested in contributing, let me know.

[baugarten]

Yeah, I would be into that. I'm really just learning, but if you're willing to entertain some back and forth, I would be interested in contributing

[jamesdabbs]

K - I've got a first draft in progress at https://github.com/jamesdabbs/keenser

Some things to be mulling over:

• I've got this registration DSL thing going on - https://github.com/jamesdabbs/keenser/blob/0c31be789785857b81d8fac8c6c67466eeb1817a/app/Main.hs#L43 - largely because the worker types include their argument type, and so you can't make a well-typed list of all workers. I don't love that, and would appreciate ideas about making it better
• Other worker registration-time improvements: verifying that worker names are unique, warning (or auto-fixing) if your configured queue plan doesn't incude the queue for a registered worker, any kind of auto-discovery of workers.
• My next order of business is going to be to get the Redis interface consistent with Sidekiq, so that the Sidekiq-web Gem will work for monitoring. If you're interested, you might want to poke around the sidekiq api.
• The hard part is going to be gracefully handling worker shutdowns and crashes and making sure that incomplete jobs get re-enqueued in redis to retry later.

Any of that seem like something you'd like to chew on?

[danpalmer]

The hard part is going to be gracefully handling worker shutdowns and crashes and making sure that incomplete jobs get re-enqueued in redis to retry later.

I realise this is a very old thread, but Redis has a nice primitive that helps build queues that are resilient to crashing workers. The process is roughly...

1. BRPOPLPUSH off the queue into a "processing" queue.
2. Process... crash!
3. Worker checks "processing" queue on startup and RPUSHes onto the main queue before continuing to (1)

We use this architecture at my place of work and it has been very successful.