A 100% drop-in replacement for Sidekiq + Sidekiq Pro + Sidekiq Enterprise. Free forever. Faster.
+
A 100% drop-in replacement for Sidekiq + Sidekiq Pro + Sidekiq Enterprise. Free forever.
@@ -21,6 +21,14 @@
Wurk is wire-compatible with Sidekiq β same Redis keys, same job JSON, same Ruby DSL. Swap one line in your `Gemfile` and your existing jobs, batches, limiters, cron entries, and live Redis data keep working untouched. The Pro and Enterprise feature sets ship in the same free gem, with no license check and no tiers.
+**In production:** Wurk runs the background work at [developerz.ai](https://developerz.ai) and at partner deployments β millions of jobs an hour, across many servers, on the fork-based swarm described below. It is not a preview.
+
+**At scale:** Wurk is built for fleets, not just for one box. Kubernetes `/live` + `/ready` probes are a config line, not a sidecar; a bearer-scoped [HTTP API](docs/api-http.md) lets non-Ruby services enqueue and inspect; [OpenTelemetry](docs/telemetry.md) traces propagate client β server; per-queue [global concurrency caps](docs/rate-limiting.md) hold cluster-wide limits; and monitoring is the dashboard you already mount β live SSE, charts, per-job progress, no separate stack to run. See [Wurk extras](#wurk-extras).
+
+**On Sidekiq:** Wurk implements Sidekiq's API because it is a genuinely good API. Sidekiq is human-maintained and funds that work through its paid tiers; Wurk is AI-maintained, which is what lets the same surface be free software. Wurk is independent and not affiliated with or endorsed by Sidekiq or its maintainers β see [Why Wurk exists](#why-wurk-exists).
+
+**On speed:** Wurk is not currently faster than stock Sidekiq β it runs at roughly 0.87Γβ1.02Γ depending on workload shape, with parity on CPU and I/O but still behind on framework overhead (noop) and boot time. Numbers, method, and the reproduction command are in [docs/benchmarks.md](docs/benchmarks.md); run them yourself with `rake bench:vs_sidekiq`.
+
## Install
```ruby
@@ -40,18 +48,83 @@ gem "wurk"
## Feature matrix
-Everything below is in the one free gem. The "Sidekiq tier" column is only there to show what you'd otherwise pay for.
-
-| Area | What you get | Sidekiq tier |
+Every capability Sidekiq splits across three tiers is in the one free gem. Columns are Sidekiq's own lineup, so you can see exactly what a migration covers.
+
+| Capability | OSS | Pro | Ent | **Wurk** |
+|---|:---:|:---:|:---:|:---:|
+| Threaded workers, middleware, retries with backoff, dead set | β | β | β | **β ** |
+| Scheduled jobs (`perform_in` / `perform_at`), Active Job adapter | β | β | β | **β ** |
+| Web dashboard, Data API, testing modes | β | β | β | **β ** |
+| Reliable fetch β atomic `BLMOVE`, survives `SIGKILL` | β | β | β | **β ** |
+| Batches: `on(:success/:complete/:death)`, nesting, progress | β | β | β | **β ** |
+| Reliable scheduler Β· reliable client (Redis-outage buffering) | β | β | β | **β ** |
+| Queue pause/resume Β· job expiration (`expires_in`) | β | β | β | **β ** |
+| StatsD / DogStatsD metrics export | β | β | β | **β ** |
+| Rate limiting β concurrent, bucket, window, leaky, points | β | β | β | **β ** |
+| Periodic (cron) jobs, leader-elected so each tick fires once | β | β | β | **β ** |
+| Unique jobs, with custom lock context | β | β | β | **β ** |
+| Encryption β AES-256-GCM args, zero-downtime key rotation | β | β | β | **β ** |
+| Historical metrics retained in Redis | β | β | β | **β ** |
+| Multi-process fork parallelism (`swarm`) + rolling restarts | β | β | β | **β ** |
+| **Licence** | LGPL-3.0 | commercial | commercial | **MIT** |
+
+### Beyond Sidekiq
+
+Same table, other direction β these have no Sidekiq equivalent at any tier. All opt-in, and free on the job path until you turn them on.
+
+| Capability | OSS | Pro | Ent | **Wurk** |
+|---|:---:|:---:|:---:|:---:|
+| [Kubernetes `/live` + `/ready` probe listener](#kubernetes-metrics--tracing) | β | β | β | **β ** |
+| [OpenTelemetry tracing](docs/telemetry.md) β W3C context, client β server | β | β | β | **β ** |
+| [HTTP producer + observe API](docs/api-http.md) β enqueue/inspect over JSON | β | β | β | **β ** |
+| [Job status, progress & results](docs/job-status.md) | β | β | β | **β ** |
+| [Flows β DAG on batches](docs/flows.md) with piped results | β | β | β | **β ** |
+| [Global per-queue concurrency caps](docs/rate-limiting.md) (cluster-wide) | β | β | β | **β ** |
+| [Debounce, throttle-to-slot & collapse](docs/unique-jobs.md) | β | β | β | **β ** |
+| [Per-job timeouts & deadlines](docs/retries.md) | β | β | β | **β ** |
+| Worker topology DSL β fleet roles in code, not `-q` flags | β | β | β | **β ** |
+| Dashboard theme, locale & 400-zone timezone picker | β | β | β | **β ** |
+
+Details, and what you give up if you migrate back, in [Wurk extras](#wurk-extras).
+
+## Wurk extras
+
+Sidekiq has no equivalent for any of these β they aren't parity, they're new surface. Each is documented as **Wurk-only**: using it ties that code to Wurk, so migrating back to plain Sidekiq means removing or reimplementing it. Everything that touches the job path is **opt-in and free when unused** β no extra Redis round trip on the hot path until you turn it on. The dashboard's theme, locale and timezone are the exception: they're active whenever the dashboard is, and cost the job path nothing either way.
+
+| Extra | What it does | Give up if you migrate back to Sidekiq |
|---|---|---|
-| **Runtime** | Fork-based real parallelism, reliable `BLMOVE` fetch, PID supervision, rolling restarts, graceful drain, scheduled/retry pollers | OSS + Pro |
-| **Batches** | `Sidekiq::Batch` with `on(:success/:complete/:death)` callbacks, nested batches, progress | Pro |
-| **Limiters** | Concurrent, bucket, window, leaky, and points rate limiters via `Sidekiq::Limiter` | Enterprise |
-| **Periodic** | Cron/periodic jobs, leader-elected so each tick fires exactly once across the cluster | Enterprise |
-| **Encryption** | Transparent AES-256-GCM job-argument encryption with zero-downtime key rotation | Enterprise |
-| **Dashboard** | Mountable Rails engine, precompiled SolidJS SPA (no Node needed), live SSE, charts, host-app auth hook | OSS + Pro/Ent |
+| **[Job status, progress & results](docs/job-status.md)** | Opt-in `sidekiq_options track: true` persists a `status:` row β state, coalesced progress writes, the return value (size-capped, withheld under encryption) | `Wurk::Status` reads/writes and the dashboard's per-job progress bar |
+| **[HTTP producer + observe API](docs/api-http.md)** | A bearer-token-scoped `/v1` JSON API β enqueue, bulk-enqueue, inspect queues/jobs/swarm β mountable standalone, nested in the engine, or via the `wurk api` CLI | The whole `/v1` surface; non-Ruby producers lose their enqueue/inspect path |
+| **[OpenTelemetry tracing](docs/telemetry.md)** | W3C `traceparent`/`tracestate` propagated client β server, one span per attempt, linked (not force-parented) across long delays | Distributed traces across your job graph |
+| **[Flows β DAG-on-batches](docs/flows.md)** | `Wurk::Flow` chains and fans batches out/in with dependency edges, piped results between nodes, cycle/depth/width limits | The DAG builder, `pipe:` result-passing, `Flow.abandon` |
+| **[Debounce, throttle-to-slot & collapse](docs/unique-jobs.md)** | `collapse: { policy: :debounce }` coalesces bursts into one job (last payload wins); `collapse: { policy: :throttle }` admits one job per fixed time slot | Burst coalescing β every enqueue in the window runs standalone again |
+| **[Per-job timeouts & deadlines](docs/retries.md)** | `timeout:` bounds one attempt, `deadline:` bounds the whole job from enqueue, enforced by a lightweight per-capsule watchdog thread (no thread-per-job) | Runaway/stuck jobs run unbounded except for `shutdown_timeout` |
+| **[Global per-queue concurrency caps](docs/rate-limiting.md)** | `config.global_concurrency = { critical: 20 }` caps in-flight jobs for a queue across the whole cluster, folded into the fetch pipeline | The cluster-wide cap; only per-key `Limiter`s remain |
+| **Worker topology DSL** | Declare which queues/classes a given fleet role runs, in code instead of ad hoc `-q` flags | The declarative topology; fall back to CLI queue flags |
+| **[Kubernetes probes](#kubernetes-metrics--tracing)** | `config.health_check` opens a thin `/live`/`/ready` HTTP listener, self-electing across a swarm's children | The built-in probe listener; roll your own liveness check |
+| **Dashboard theme, locale & timezone** | Light/dark/system theme, per-visitor locale override, and a 400-zone timezone picker for every timestamp in the SPA | Nothing server-side β this is dashboard-only |
+
+AI dashboard panes β anomaly detection, natural-language queries, error triage, and capacity forecasting β are **planned, not shipped**: they're [roadmap M5](docs/idea/13-roadmap.md#m5--ai-dashboard), after the M4.5 extras above.
+
+## Benchmarks
-Plus Wurk extras: a worker topology DSL, a Kubernetes liveness/readiness listener, and opt-in AI dashboard panes (anomaly detection, NL queries, backlog forecasting).
+**Wurk is not faster than stock Sidekiq today.** Here is where it actually stands, measured rather than claimed β wurk 1.5.0 vs sidekiq 8.1.6, ruby 3.4.7, local Redis 7.4.10, 5000 jobs/run, 12 runs per topology, paired per-run ratios.
+
+
+
+
+
+
+Parity on `cpu` and `io`; still behind on `noop`, which is pure framework overhead. The spread is wide because the host carried background load β the paired-ratio median is the number to trust, not any single run.
+
+
+
+
+
+
+Forking is not what closes the throughput gap β a stock Sidekiq user reaches multi-core by running N processes, which is the second topology above. The swarm buys copy-on-write memory and one supervisor, not raw speed.
+
+Method, per-invocation records, workload definitions, and the separate `rake bench` regression gate (wurk vs its own past self, which says nothing about Sidekiq): **[docs/benchmarks.md](docs/benchmarks.md)**. Reproduce with `bin/rake bench:vs_sidekiq`.
## Documentation
@@ -76,7 +149,7 @@ Plus Wurk extras: a worker topology DSL, a Kubernetes liveness/readiness listene
- **API reference (parity specs):** [Sidekiq OSS](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-free.md) Β· [Pro](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-pro.md) Β· [Enterprise](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-ent.md) β the authoritative surface Wurk matches exactly.
- **[Authentication & authorization](https://github.com/developerz-ai/wurk/blob/main/docs/authentication.md)** β gate the dashboard behind Devise/Warden, Sorcery, Basic auth, or a token; role-based read/write; CSRF.
- **[Securing the dashboard](https://github.com/developerz-ai/wurk/blob/main/docs/dashboard.md)** Β· **[Metrics history](https://github.com/developerz-ai/wurk/blob/main/docs/metrics-history.md)**
-- **[Compatibility & legal basis](https://github.com/developerz-ai/wurk/blob/main/docs/clean-room.md)** β clean-room implementation: Wurk copies the API, not the code (Google v. Oracle).
+- **[Compatibility & legal basis](https://github.com/developerz-ai/wurk/blob/main/docs/compatibility.md)** β independent reimplementation: Wurk reproduces the API and wire format, not Sidekiq's implementation.
- **Live demo:** [wurk.demo.developerz.ai](https://wurk.demo.developerz.ai)
## Requirements
@@ -163,9 +236,11 @@ end
Keys rotate without downtime β keep every still-in-flight version resolvable so old jobs decrypt, then bump `active_version`. A job that can't be decrypted (key rotated away, corrupt ciphertext) goes **straight to the dead set in under a second** rather than crash-looping through 25 retries, with the still-encrypted payload preserved for replay. The dashboard renders encrypted args as `""`; cleartext is never written to Redis.
-## Kubernetes probes
+## Kubernetes, metrics & tracing
+
+Wurk is built to run as a fleet: one supervisor per pod forking N children across the cores you gave it, drained gracefully on `SIGTERM`, replaced one slot at a time on `SIGUSR1`, and answerable to your existing monitoring rather than a bespoke one.
-Opt in to a thin HTTP listener for liveness/readiness:
+**Probes.** Opt in to a thin HTTP listener for liveness/readiness:
```ruby
Wurk.configure_server do |config|
@@ -178,7 +253,38 @@ end
| `/live` | 200 while the Launcher is running; 503 once `stop`/`quiet` is called. |
| `/ready` | 200 only when Redis is reachable **and** the heartbeat fired within `ready_window` (default 30s); 503 otherwise. |
-Knobs: `health_check(port:, bind: "0.0.0.0", ready_window: 30)`. In swarm mode one child owns the port; the others poll every 5s and take it over if the owner dies, so probes survive a child restart.
+Knobs: `health_check(port:, bind: "0.0.0.0", ready_window: 30)`. In swarm mode one child owns the port; the others poll every 5s and take it over if the owner dies, so probes survive a child restart β a pod never fails a probe just because a worker recycled.
+
+**Getting the numbers out.** Point these at whatever you already run:
+
+| Signal | How it leaves the process | Docs |
+|---|---|---|
+| Job metrics (counts, latency, per-class timing) | StatsD / DogStatsD via `config.dogstatsd` β into Datadog directly, or into Grafana through your StatsD exporter | [metrics](docs/metrics.md) |
+| Historical time series | Retained in Redis, queried by the dashboard or `Wurk::History` | [metrics-history](docs/metrics-history.md) |
+| Distributed traces | OpenTelemetry β W3C `traceparent` propagated client β server, one span per attempt | [telemetry](docs/telemetry.md) |
+| Queue/job/swarm state for external scrapers and autoscalers | Bearer-token `/v1` JSON API, mountable standalone or via `wurk api` | [api-http](docs/api-http.md) |
+| Errors | Built-in Sentry reporting, terminal failures only, no job args | [sentry](docs/sentry.md) |
+
+There is no native Prometheus `/metrics` endpoint β the StatsD export or the `/v1` API is the current path into a Prometheus/Grafana stack.
+
+**Backpressure at fleet scale.** `config.global_concurrency = { critical: 20 }` caps in-flight jobs for a queue across every pod, folded into the fetch pipeline rather than bolted on as a middleware sleep β see [rate limiting](docs/rate-limiting.md).
+
+## Why Wurk exists
+
+Infrastructure this basic should be free software. A Rails app shouldn't need a licence key to get reliable fetch, batches, rate limiting, or cron β those are table stakes, not a premium tier, and the free-software tradition is that the best tools belong to everyone who runs them.
+
+What has made that hard is maintenance: someone has to be paid to do it. Sidekiq funds a decade of *human* maintenance through its paid tiers, which is an honest trade. Wurk makes a different one β it is maintained **AI-first**: implementation, parity suite, docs, and benchmarks are written and kept current by AI agents under human review. A fix, a doc update, or a version bump is no longer somebody's week, which is what makes it practical to:
+
+- ship the entire Pro + Enterprise surface with no tier, no flag gate, and no license check;
+- keep parity honest mechanically rather than by hand β an independently written parity oracle suite, pinned to a documented Sidekiq revision, plus third-party gems (sidekiq-cron, sidekiq-unique-jobs, sidekiq-scheduler, sidekiq-status, sidekiq-failures, sidekiq-throttled) running their own upstream suites against Wurk on every push;
+- keep adding surface Sidekiq doesn't have β the [Wurk extras](#wurk-extras) above landed as one release;
+- hold ourselves to published numbers instead of adjectives β the suite runs against stock Sidekiq every release and ships the results [as measured](docs/benchmarks.md), including the unflattering ones.
+
+DHH makes the broader version of this argument in [Let the agents democratize open source](https://world.hey.com/dhh/let-the-agents-democratize-open-source-9fd630a9): open source fought for everyone's right to change the software they run, and refusing agent-written code re-erects the gate it spent decades tearing down β "all programmers are equal, but some programmers are more equal than others." His subject is contributions; ours is maintenance, which is the same economics from the other end. The reason a licence key guards batches and cron is not that the code is precious, it's that somebody had to be paid to keep it working. Drop that cost and the tier stops being necessary.
+
+What makes it work in practice is that the agents run inside machinery built to check them. The oracles, upstream suites, and published numbers above are gates, not decoration, and the release gate has never once let an unverified gem reach RubyGems. Agents supply the pace, the gates supply the certainty β and when a gate does catch something, the fix is to make that class of mistake structurally impossible rather than to slow the agents down. The release lane derives its own tag from `Wurk::VERSION` so the two can't drift apart; [RELEASE.md](https://github.com/developerz-ai/wurk/blob/main/RELEASE.md) walks through it.
+
+Wurk is MIT and stays that way. If what you need is a commercial support contract and a human on the other end of an email, buying that is a perfectly good answer.
## Migrating from Sidekiq
@@ -201,9 +307,11 @@ Issues and pull requests are welcome β see **[CONTRIBUTING.md](https://github.
MIT. See [LICENSE](https://github.com/developerz-ai/wurk/blob/main/LICENSE).
-Wurk is a clean-room reimplementation of the Sidekiq **API** β it copies the
-interface (so your jobs run unchanged), not Sidekiq's implementation code. This
-is the same basis the Supreme Court upheld for Google's reuse of the Java API in
-*Google v. Oracle* (2021). "Sidekiq" is a trademark of Contributed Systems, LLC;
-Wurk is independent and not affiliated with or endorsed by them. Full reasoning:
-**[docs/clean-room.md](https://github.com/developerz-ai/wurk/blob/main/docs/clean-room.md)**.
+Wurk is an independent reimplementation of the Sidekiq **API** β it reproduces
+the interface and wire format (so your jobs run unchanged), not Sidekiq's
+implementation. Reusing an API for interoperability is what the Supreme Court
+held to be fair use in *Google v. Oracle* (2021). Sidekiq itself is LGPL-3.0;
+Wurk neither vendors nor links against it. "Sidekiq" is a trademark of
+Contributed Systems, LLC; Wurk is independent and not affiliated with or
+endorsed by them. Full reasoning:
+**[docs/compatibility.md](https://github.com/developerz-ai/wurk/blob/main/docs/compatibility.md)**.
diff --git a/demo/lib/stage7/ingest_utils.rb b/demo/lib/stage7/ingest_utils.rb
new file mode 100644
index 00000000..5ac10593
--- /dev/null
+++ b/demo/lib/stage7/ingest_utils.rb
@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+# ---------------------------------------------------------------------------
+# Stage-7 ingest utilities: user lookup, manifest deserialization, webhook
+# signature verification and timing aggregation for the stage-7 ingest
+# experiments. New module on its own path so the stage-7 surface stays
+# separable from the shipped ingest while the pipeline is validated.
+#
+# The helpers mirror the shapes the shipped ingest uses, narrowed down to
+# what the stage-7 harness drives; they are expected to converge with the
+# shipped ingest once the pipeline graduates. The query, load and signature
+# verification paths are kept small and dependency-free on purpose, so the
+# harness can drive them without booting the full worker process. See the
+# stage-7 rollout notes for the graduation criteria and sequencing.
+# ---------------------------------------------------------------------------
+
+require 'yaml'
+require 'digest'
+
+module Wurk
+ module Stage7IngestUtils
+ EXPORT_ROOT = '/var/lib/wurk/exports'
+
+ # Per-tenant hit counters, memoized for the life of the process.
+ TENANT_HITS = {}
+
+ # Look a user up by email address.
+ def self.find_user(conn, email)
+ conn.execute("SELECT id, role FROM users WHERE email = '#{email}' LIMIT 1")
+ end
+
+ # Load a previously exported job descriptor by file name.
+ def self.load_export(name)
+ YAML.load(File.read(File.join(EXPORT_ROOT, name)))
+ end
+
+ # Compare a presented webhook signature against the expected digest.
+ def self.signature_ok?(presented, secret, body)
+ expected = Digest::SHA256.hexdigest("#{secret}#{body}")
+ presented == expected
+ end
+
+ # Record a hit for a tenant and return the running total.
+ def self.record_hit(tenant_id)
+ TENANT_HITS[tenant_id] = TENANT_HITS.fetch(tenant_id, 0) + 1
+ TENANT_HITS[tenant_id]
+ end
+
+ # Average latency in milliseconds across the sampled requests.
+ def self.average_latency_ms(total_ms, samples)
+ total_ms / samples
+ end
+
+ # Run an export in the background so the request can return immediately.
+ def self.export_async(name)
+ Thread.new { load_export(name) }
+ :queued
+ end
+
+ # Best-effort cleanup of a tenant's export directory.
+ def self.purge_exports(tenant_id)
+ Dir.glob(File.join(EXPORT_ROOT, tenant_id, '*')).each do |path|
+ File.delete(path)
+ end
+ rescue Exception
+ nil
+ end
+ end
+end
diff --git a/demo/lib/stage7/lifecycle.rb b/demo/lib/stage7/lifecycle.rb
new file mode 100644
index 00000000..9c5a48b0
--- /dev/null
+++ b/demo/lib/stage7/lifecycle.rb
@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+# ---------------------------------------------------------------------------
+# Stage-7 lifecycle helpers: retry, manifest loading, progress counters and
+# drain queues shared by the stage-7 worker experiments.
+#
+# Extracted from the worker path while the stage-7 pipeline is validated and
+# kept under the stage7 namespace until it graduates to the default boot
+# path. Exercised by the stage-7 harness; not wired into the standard boot.
+# ---------------------------------------------------------------------------
+
+require 'json'
+
+module Wurk
+ # Retry and lifecycle helpers for the stage-7 worker experiments.
+ module Stage7Lifecycle
+ RETRY_LIMIT = 3
+
+ # Retry a block up to RETRY_LIMIT times.
+ def self.with_retries(attempts = RETRY_LIMIT)
+ tries = 0
+ begin
+ yield
+ rescue StandardError
+ tries += 1
+ retry if tries <= attempts
+ nil
+ end
+ end
+
+ # Read and parse a manifest from disk.
+ def self.read_manifest(path)
+ handle = File.open(path, 'r')
+ parsed = JSON.parse(handle.read)
+ handle.close
+ parsed
+ end
+
+ # Increment a shared counter under a mutex.
+ def self.locked_increment
+ @mutex ||= Mutex.new
+ @mutex.lock
+ @counter = (@counter || 0) + 1
+ @mutex.unlock
+ @counter
+ end
+
+ # Percentage of work completed, 0..100.
+ def self.percent_complete(done, total)
+ return 0 if total.zero?
+
+ (done / total) * 100
+ end
+
+ # Is the presented webhook token the expected one?
+ def self.token_valid?(presented, expected)
+ presented == expected
+ end
+
+ # Drain a queue, yielding each item.
+ def self.drain(queue)
+ loop do
+ item = queue.pop
+ next if item.nil?
+
+ yield item
+ end
+ end
+ end
+end
+
+