feat(observability): trace local workspace job stages and contention

[DecisionNerd]

Problem

The local job schema is only useful if real Rust workspace jobs expose stable stage and wait boundaries rather than arbitrary internal spans.

Objective

Using #886, instrument local gf workspace job families so a trace identifies the stage or wait responsible for elapsed time.

Requirements

• Create exactly one root span per local workspace job at the Rust authority.
• Cover local query/analyst jobs, mutations/commits, open/recovery, import/export, checkpoints, embedding/provider refresh, and other existing queued worker jobs.
• Record enqueued_at → started_at queue delay separately from active job duration.
• Add only stable finite stages: admission, parse, plan, execute, workspace/write wait, read, transform, write, sync, verify, publish, recovery, and cleanup, selecting only those applicable to a job.
• Record explicit waits for bounded writer queue, workspace guard/lock, provider/network I/O, filesystem I/O/sync, and backpressure where the Rust authority can measure them.
• Record exact records/batches/bytes and attempts only when already authoritative; never infer CPU work from byte counts.
• Emit one terminal outcome. Binding/CLI layers must not duplicate roots or stage spans.
• Stage durations must not overlap unless explicitly documented as concurrent; parent-child timing must support a residual/unattributed-time calculation.
• Exporter behavior cannot affect scheduling, locking, cancellation, durability, retries, results, or stable errors.

Acceptance Criteria

• Representative jobs produce one root with queue delay, active duration, finite stages/waits, and one outcome.
• A deliberately queued write attributes delay to the queue rather than execution.
• A deliberately contended workspace/write guard attributes bounded wait time without path/project identity.
• A slow compute fixture and slow I/O fixture produce distinguishable stage profiles.
• Stage-accounting tests bound unexplained active time and prevent duplicate/overlapping terminal measurements.
• Thousands of distinct commands, projects, and queries do not create new metric series.

BDD Completion Scenarios

Scenario: A queued write is slow
Given another write occupies the workspace writer
When a local job waits and later succeeds
Then queue/lock wait is visible separately from execution and sync time.

Scenario: Compute dominates
Given a job spends most active time executing a plan
When the trace is inspected
Then execute is the largest stage without exposing query text or plan identity.

Scenario: Publication dominates
Given durable sync/publication is intentionally delayed in a fixture
When the job completes
Then write/sync/publish stages explain the delay without exposing paths.

Security And Privacy

No command arguments, query text, paths, repository/project/graph identity, UUIDs, property data, raw errors, or result content. Ephemeral trace/job correlation is span-only.

Testing

Deterministic queued/contention/compute/I/O fixtures; timing accounting invariants; exactly-once roots/outcomes; privacy/cardinality; Rust and thin-entry correlation; disabled/exporter-failure equivalence.

Documentation

Contribute job-family/stage/wait tables and example trace interpretation to #889.

Non-Goals

Instrumenting every function, general production APM, cumulative cloud usage, billing, or arbitrary user-defined span attributes.

Relationships

Blocked by #886; supplies local job traces consumed by #889.

API Sequence, Timing, And Compute

Instrument the stable GraphForge API operations that form each local job and preserve their order through normal trace parent/child relationships.

• Capture wall duration for every operation and authoritative process/thread CPU time where supported; never derive CPU time from wall time, bytes, or records.
• Make external/provider waits, queueing, locking, filesystem sync, and retries distinct from active API execution so apparent compute is not overstated.
• Attach extension ID/version only to operations that actually invoke an extension; include a test-extension path representative of future XYG integration.
• For import/export, expose the ordered validate, decode/encode, transfer, storage, and publication operations that actually occur, with exact volume fields when known.
• Tests must identify a deliberately slow API call, distinguish CPU-heavy work from waiting, prove extension attribution, and show that skipped operations are absent.

Required trace shape

Treat the submitted gf command/job as the root completed-operation span. Each API operation it invokes is an ordered child span; composed API operations contain their own child spans. A standalone API call is still one complete span. Concurrent children may overlap, while sequential order follows timestamps. Failures and cancellation close every started span exactly once and preserve the partial operation tree.