A small, dependency-light Go library for building directed acyclic task graphs (DAGs) and executing them with maximum parallelism, health-style progress reporting, retries, and Prometheus metrics.
Derived from Gardener's pkg/utils/flow, with Kubernetes- and Gardener-specific dependencies removed.
When you have work that is partly sequential and partly parallel — build → (test ‖ package) →
publish — writing the goroutine/sync plumbing by hand is error prone. goflow lets you
declare the dependencies and hands you:
- Automatic parallelism — independent tasks run concurrently; a task starts only once all its dependencies have succeeded.
- Fail-fast with full reporting — on error, dependents are not run; every task error is
collected and retrievable via
flow.Errors/flow.Causes. - Cancellation — a canceled
context.Contextstops scheduling new tasks and surfaces aflowCancelederror (check withflow.WasCanceled). - Skip conditions — mark a task
SkipIf: trueand its dependents still run. - Task groups — bundle tasks under a shared set of dependencies (
Graph.AddGroup). - Combinators —
Sequential,Parallel,ParallelN,ParallelExitOnError,.Timeout,.RetryUntilTimeout,.Recover. - Progress reporting — immediate, delaying (rate-limited), or command-line (dump status on
Ctrl+T/SIGINFOon macOS,SIGUSR1on Linux). - Metrics — opt-in Prometheus histograms/counters via
flow.RegisterMetrics.
go get github.com/example/goflow/flow
g := flow.NewGraph("deploy")
build := g.Add(flow.Task{Name: "build", Fn: buildFn})
test := g.Add(flow.Task{Name: "test", Fn: testFn, Dependencies: flow.NewTaskIDs(build)})
pkg := g.Add(flow.Task{Name: "package", Fn: pkgFn, Dependencies: flow.NewTaskIDs(build)})
g.Add(flow.Task{Name: "publish", Fn: publishFn, Dependencies: flow.NewTaskIDs(test, pkg)})
err := g.Compile().Run(context.Background(), flow.Opts{})test and package run in parallel after build; publish waits for both.
Run the bundled example:
go run ./examples/deploy
| Type | Role |
|---|---|
Task |
A named unit of work (Fn), optional SkipIf, and Dependencies. |
Graph |
A builder. Add/AddGroup register tasks; Compile validates and produces a Flow. |
Flow |
An immutable, executable graph. Run(ctx, Opts) blocks until completion. |
TaskFn |
func(context.Context) error, with combinators like .Timeout and .RetryUntilTimeout. |
TaskGroup |
A build-time bundle of tasks sharing dependencies; flattened at AddGroup. |
ProgressReporter |
Receives *Stats snapshots during execution. |
Add and AddGroup panic on a duplicate task name or an unknown dependency — these are
programming errors caught at graph-build time, before anything runs.
err := flow.Compile().Run(ctx, flow.Opts{})
if flow.WasCanceled(err) {
// context was canceled
}
for _, cause := range flow.Causes(err).Errors {
log.Println(cause)
}flow.RegisterMetrics(prometheus.DefaultRegisterer) // call onceRegisters flow_task_delay_seconds, flow_task_duration_seconds, flow_task_results_total,
and flow_duration_seconds. If never called, no metrics are collected.
flow/ the engine (Graph, Flow, TaskFn, groups, reporters, metrics)
flowerrors/ error-ID tracking + ErrorContext (drives ErrorCleaner)
internal/ OS-signal helper for the command-line progress reporter
examples/ runnable example
Derived from github.com/gardener/gardener's pkg/utils/flow. Apache-2.0.