Skip to content

Latest commit

 

History

History
102 lines (76 loc) · 3.68 KB

README.md

File metadata and controls

102 lines (76 loc) · 3.68 KB

goflow

A small, dependency-light Go library for building directed acyclic task graphs (DAGs) and executing them with maximum parallelism, health-style progress reporting, retries, and Prometheus metrics.

Derived from Gardener's pkg/utils/flow, with Kubernetes- and Gardener-specific dependencies removed.

Why

When you have work that is partly sequential and partly parallel — build → (test ‖ package) → publish — writing the goroutine/sync plumbing by hand is error prone. goflow lets you declare the dependencies and hands you:

  • Automatic parallelism — independent tasks run concurrently; a task starts only once all its dependencies have succeeded.
  • Fail-fast with full reporting — on error, dependents are not run; every task error is collected and retrievable via flow.Errors / flow.Causes.
  • Cancellation — a canceled context.Context stops scheduling new tasks and surfaces a flowCanceled error (check with flow.WasCanceled).
  • Skip conditions — mark a task SkipIf: true and its dependents still run.
  • Task groups — bundle tasks under a shared set of dependencies (Graph.AddGroup).
  • Combinators — Sequential, Parallel, ParallelN, ParallelExitOnError, .Timeout, .RetryUntilTimeout, .Recover.
  • Progress reporting — immediate, delaying (rate-limited), or command-line (dump status on Ctrl+T/SIGINFO on macOS, SIGUSR1 on Linux).
  • Metrics — opt-in Prometheus histograms/counters via flow.RegisterMetrics.

Install

go get github.com/example/goflow/flow

Quick start

g := flow.NewGraph("deploy")

build := g.Add(flow.Task{Name: "build", Fn: buildFn})
test  := g.Add(flow.Task{Name: "test",  Fn: testFn,  Dependencies: flow.NewTaskIDs(build)})
pkg   := g.Add(flow.Task{Name: "package", Fn: pkgFn,  Dependencies: flow.NewTaskIDs(build)})
g.Add(flow.Task{Name: "publish", Fn: publishFn, Dependencies: flow.NewTaskIDs(test, pkg)})

err := g.Compile().Run(context.Background(), flow.Opts{})

test and package run in parallel after build; publish waits for both.

Run the bundled example:

go run ./examples/deploy

Core concepts

Type Role
Task A named unit of work (Fn), optional SkipIf, and Dependencies.
Graph A builder. Add/AddGroup register tasks; Compile validates and produces a Flow.
Flow An immutable, executable graph. Run(ctx, Opts) blocks until completion.
TaskFn func(context.Context) error, with combinators like .Timeout and .RetryUntilTimeout.
TaskGroup A build-time bundle of tasks sharing dependencies; flattened at AddGroup.
ProgressReporter Receives *Stats snapshots during execution.

Add and AddGroup panic on a duplicate task name or an unknown dependency — these are programming errors caught at graph-build time, before anything runs.

Error handling

err := flow.Compile().Run(ctx, flow.Opts{})
if flow.WasCanceled(err) {
    // context was canceled
}
for _, cause := range flow.Causes(err).Errors {
    log.Println(cause)
}

Metrics

flow.RegisterMetrics(prometheus.DefaultRegisterer) // call once

Registers flow_task_delay_seconds, flow_task_duration_seconds, flow_task_results_total, and flow_duration_seconds. If never called, no metrics are collected.

Layout

flow/         the engine (Graph, Flow, TaskFn, groups, reporters, metrics)
flowerrors/   error-ID tracking + ErrorContext (drives ErrorCleaner)
internal/     OS-signal helper for the command-line progress reporter
examples/     runnable example

Provenance & license

Derived from github.com/gardener/gardener's pkg/utils/flow. Apache-2.0.