daffy

A process wrapper that ships log lines to Scrooge.

daffy runs a child process, tees its stdout/stderr to the console (so kubectl logs and similar still work), buffers each line in a local DuckDB, and batch-ships the buffer to Scrooge over the DuckDB Quack remote protocol. It's the log path for pods whose own process doesn't integrate DuckDB directly — daffy carries the DuckDB client so the wrapped program doesn't have to.

daffy is written in Go against the official duckdb-go driver, which embeds DuckDB ≥ 1.5.4 (Quack needs ≥ 1.5.3). The driver links DuckDB via CGO, so builds need a C toolchain.

Building and running

just build   # or: go build -o daffy ./cmd/daffy
./daffy --help

(or go run ./cmd/daffy --help without building a binary).

Usage

daffy wraps any process. Lines accrue in a local DuckDB buffer until a row-count threshold (or interval) triggers a batch flush to Scrooge, after which the flushed rows are deleted locally. If Scrooge is unreachable the rows are retained and retried; if the buffer exceeds its row cap the oldest rows are dropped.

daffy --service my-svc \
      --scrooge-uri quack:scrooge-host:9494 \
      -- my-server --port 8080

Everything after -- is the wrapped command and its arguments.

Configuration is resolved env-first with CLI-flag fallback: SERVICE_NAME, DAFFY_LOCAL_DB, POD_NAME, NODE_NAME, SCROOGE_URI, SCROOGE_TOKEN, DAFFY_FLUSH_ROWS, DAFFY_FLUSH_INTERVAL, DAFFY_MAX_BUFFER_ROWS. With no SCROOGE_URI, daffy logs locally only (no shipping).

The local buffer defaults to :memory:, so buffered-but-unshipped rows are lost if the process exits while Scrooge is unreachable. Set DAFFY_LOCAL_DB to a file path to survive restarts.

Local example

Point daffy at a running Scrooge (see the scrooge repo) and wrap a command:

SCROOGE_TOKEN=dev-token daffy --service demo \
      --scrooge-uri quack:127.0.0.1:9494 \
      -- sh -c 'for i in $(seq 1 20); do echo "line $i"; done'

Development

just install    # go mod download
just lint       # golangci-lint run
just fmt        # golangci-lint fmt
just vet        # go vet ./...
just test       # go test ./...
just build      # build the daffy binary at ./daffy
just check      # lint + vet + test + compile

The shipper tests stand up a real Quack endpoint on localhost, so they need loopback networking and the quack extension (installed automatically from the DuckDB extension repo on first run).

Container image

The image runs the daffy wrapper (daffy is the entrypoint) and bakes the quack extension in at build time, so containers don't need egress to the DuckDB extension repo. Build it with just build-image (or docker build -t daffy .), then:

docker run --rm -e SCROOGE_TOKEN=dev-token daffy \
      --service demo --scrooge-uri quack:scrooge-host:9494 -- my-server