feat: add label-based sharding for horizontal scaling

[aashishtomar]

Description

Currently the provider-terraform controller only supports active/passive HA via leader election — a single replica reconciles all Workspaces regardless of replica count. For deployments with hundreds of Workspaces, this becomes a throughput bottleneck since each reconciliation runs terraform init/plan/apply, which can take seconds to minutes per workspace.

This PR introduces a --shard-name flag (also configurable via SHARD_NAME env var) that partitions Workspaces across controller instances using the label terraform.crossplane.io/shard=<name>. Each shard controller only watches and reconciles workspaces matching its label, enabling true horizontal scaling.

Related issues: #212, #269

How it works

Label-based partitioning

Operators label Workspaces with terraform.crossplane.io/shard=<name> and deploy multiple controller instances, each started with --shard-name=<name>. Each instance configures cache.ByObject with a label selector so the controller-runtime informer only watches matching Workspace resources. Workspaces without the label are not reconciled by any shard — operators should either label all workspaces or run an unsharded catch-all instance.

Per-shard leader election

Each shard appends its name to the leader election lease ID (e.g., crossplane-leader-election-provider-terraform-shard-0), allowing multiple shards to be active simultaneously. Without this, all shards would compete for the same lease and only one could be active.

Garbage collection safety

The GC uses mgr.GetAPIReader() (uncached) instead of mgr.GetClient() (cached) to list workspaces. This is critical: in sharded mode, the manager's cache is filtered by shard label, so using the cached client would only return the current shard's workspaces. The GC would then consider other shards' working directories as orphaned and delete them — a data loss bug. By using the uncached API reader, the GC sees all workspaces across all shards and only deletes directories for truly deleted workspaces.

The GarbageCollector.kube field type was changed from client.Client to client.Reader to enforce at compile time that only an uncached reader is passed.

Scheme registration order

All scheme registrations were moved before ctrl.NewManager() creation. This is required because cache.ByObject needs the types registered in the scheme at manager startup to determine whether resources are cluster-scoped or namespaced.

Backward compatibility

When --shard-name is not set (the default), behavior is identical to the existing single-controller mode:

• All three shard-specific code guards (if *shardName != "") are skipped
• Cache has no ByObject filter (watches all resources)
• Leader election ID remains crossplane-leader-election-provider-terraform
• All 9 existing GC unit tests pass unchanged

Changes

cmd/provider/main.go

• Add --shard-name flag with SHARD_NAME env var, default empty
• Move scheme registration before manager creation for cache.ByObject compatibility
• Configure cache.ByObject with label selector on both cluster-scoped and namespaced Workspace types when shard name is set
• Append shard name to leader election lease ID for per-shard leaders
• Add startup INFO messages warning about unlabeled workspaces

internal/controller/gc/gc.go

• Accept shardName parameter in Setup()
• Pass mgr.GetAPIReader() instead of mgr.GetClient() to the GarbageCollector
• Pass WithShardName() option for logging context

internal/workdir/workdir.go

• Add ShardLabel constant (terraform.crossplane.io/shard)
• Change GarbageCollector.kube from client.Client to client.Reader
• Add shardName field and WithShardName() option
• Add comment explaining why uncached reader is required

internal/workdir/workdir_test.go

• Update existing test field types from client.Client to client.Reader
• Add TestCollectWithShardName with 2 table-driven cases
• Add TestWithShardName (option pattern)
• Add TestShardLabel (constant value)

Deployment model

To deploy 3 shards, create 3 Deployments with the same image, each with a different --shard-name:

# Shard 0
args: ["--shard-name=shard-0", "--leader-election", "--max-reconcile-rate=5"]

# Shard 1
args: ["--shard-name=shard-1", "--leader-election", "--max-reconcile-rate=5"]

# Shard 2
args: ["--shard-name=shard-2", "--leader-election", "--max-reconcile-rate=5"]

Label workspaces to assign them:

metadata:
  labels:
    terraform.crossplane.io/shard: shard-0

Testing

Unit tests (13 pass, 4 new)

All tests use the existing table-driven map[string]struct{reason, fields, args, want} pattern with cmp.Diff and test.EquateErrors().

Test	What it verifies
TestCollectWithShardName/ShardedGCListsAllWorkspaces	GC running as shard-0 sees workspaces from all shards via uncached reader. Only deletes directories for truly deleted workspaces.
TestCollectWithShardName/ShardedGCPreservesOtherShardDirs	GC running as shard-1 does NOT delete shard-0's working directories.
TestWithShardName	WithShardName() option correctly sets the field.
TestShardLabel	Constant has expected value terraform.crossplane.io/shard.
9 existing TestCollect/* cases	All pass unchanged, confirming zero regression in unsharded mode.

Static analysis

• go vet ./... — clean
• gofmt — clean
• go build ./... — compiles
• go test ./... — all 4 test packages pass
• make reviewable — passes

End-to-end load test (Kind cluster, 3 nodes, 31 workspaces)

Tested on a Kind cluster with Crossplane 2.2.0 and 3 sharded controller deployments.

Scenario	What was tested	Result
Initial creation	15 workspaces (5 per shard) with time_sleep to simulate real apply time	15/15 SYNCED=True, READY=True. All 3 shards began reconciling within 3 seconds of each other.
Config updates	Updated 3 workspaces in shard-0 with new terraform (variables, locals, multiple outputs)	All re-reconciled successfully. New outputs reflected in Workspace status.
Error isolation	3 intentionally broken workspaces in shard-1 (HCL syntax error, missing provider, bad reference)	Shard-1 marked them SYNCED=False with ReconcileError. Shard-0 and shard-2 workspaces completely unaffected. Shard-1 pod: 0 restarts.
Volume scaling	Added 15 more workspaces (total 30+), 10 per shard	All healthy workspaces reached SYNCED=True. Each shard handling 10 workspaces concurrently.
Relabel	Moved ws-shard1-006 from shard-1 to shard-2	Shard-1 evicted it from cache. Shard-2 picked it up, re-ran terraform init, reconciled successfully.
Delete across shards	Deleted one workspace from each shard	All deleted cleanly. Remaining workspaces unaffected.
Crash recovery	Force-killed shard-2 pod	Deployment restarted it. New pod re-acquired its leader lease, showed startup warnings, resumed reconciling all its workspaces.
Unlabeled workspace	Created workspace with no shard label	Zero shard controllers reconciled it (verified via logs). Only the original unsharded controller handled it.
Cross-shard isolation	Verified across all scenarios	0 violations. Each shard only reconciled its own workspaces.
Status reporting	Checked conditions and outputs on healthy and error workspaces	Each shard correctly wrote Synced/Ready conditions and atProvider.outputs back to the Workspace CR.
Leader election	3 independent leases	Each shard acquired its own lease. Crash of one shard did not affect others' leases.

Operational notes

• Relabel window: When relabeling a workspace, there is a brief period where the old shard evicts it from its cache and the new shard picks it up. During this window the workspace is not actively reconciled. This is expected and typically lasts less than the poll interval.
• Unlabeled workspaces: Workspaces without the shard label are not reconciled by any shard controller. The startup log warns about this. Run an unsharded instance as a catch-all if needed.
• GC safety: The GC intentionally bypasses the shard-filtered cache to list all workspaces. This prevents cross-shard directory deletion but means each shard's GC makes unfiltered API server calls. At typical workspace counts this is negligible.

[cmardonesp]

please we need this!

[cmardonesp]

@ulucinar

[cmardonesp]

@erhancagirici 🙏