Skip to content

feat(observability): summarize local workspace job bottlenecks #888

Description

@DecisionNerd

Problem

A trace shows one job, but developers also need bounded local aggregates to identify recurring bottlenecks across a workspace session without managed-cloud or billing semantics.

Objective

Using #886/#887, expose local session aggregates that answer which job families/stages/waits consume wall time and where throughput or contention degrades.

Requirements

  • Aggregate only finite job family, stage, wait reason, and outcome dimensions from feat(observability): establish Rust-owned local workspace job telemetry #886/feat(observability): trace local workspace job stages and contention #887.
  • Provide counts and histograms for queue delay, active duration, stage duration, explicit wait duration, attempts/retries, and exact bytes/records where available.
  • Provide current queued/active job counts and configured concurrency/capacity where existing worker authorities expose them.
  • Define a versioned in-process local session snapshot so gf tooling can summarize bottlenecks without requiring a collector.
  • Include start time, observation time, sample count, units, histogram boundaries, reset behavior, and a residual/unattributed-time field.
  • Rank only finite job families/stages/waits; retain no command, query, plan fingerprint, path, repository/project identity, or input-keyed top-k structure.
  • Never infer CPU time from wall time. CPU time is reported only if a portable authoritative measurement is implemented; otherwise document it as unavailable.
  • Use checked/saturating arithmetic; snapshot reads are bounded and cannot block job progress.

Acceptance Criteria

  • A local session snapshot and in-memory OTel export reconcile for deterministic queued, contended, compute-heavy, I/O-heavy, and failed job fixtures.
  • The summary identifies the largest finite stage/wait contributor and reports residual time without false precision.
  • Queue depth/active counts reconcile with existing worker authorities where available.
  • Restart/reset, concurrency, overflow, and histogram semantics are deterministic.
  • Distinct projects/queries/paths do not change series count or appear in snapshots.
  • No hosted backend or external exporter is required for local summaries.

BDD Completion Scenarios

Scenario: A session is queue-bound
Given jobs exceed configured local concurrency
When the session summary is read
Then queue delay and queue depth dominate without being counted as execution time.

Scenario: A session is sync-bound
Given repeated durable jobs spend time syncing/publishing
When aggregates are inspected
Then finite sync/publish histograms identify the bottleneck.

Scenario: Inputs remain private and bounded
Given many distinct workspaces and queries
When local aggregates are exported
Then only finite families/stages/waits exist and no identities or inputs are retained.

Testing

Snapshot/export reconciliation; deterministic bottleneck fixtures; reset/concurrency/overflow/histogram tests; series ceiling; privacy mutation tests; disabled mode.

Documentation

Contribute local summary fields, timing math, histogram guidance, and bottleneck interpretation to #889.

Non-Goals

Managed-cloud usage, logical lifetime accounting, billing, quotas, tenants, stored-byte dashboards, project labels, query analytics, or hosted telemetry.

Relationships

Blocked by #886 and #887; local profiling guidance closes through #889.

Local Bottleneck Summary

Summaries answer which API operations dominate wall time, authoritative CPU time, waiting, retries, and import/export volume across local jobs. Preserve operation order in traces rather than encoding sequences as metric dimensions.

Extension usage may be summarized by bounded counts and duration only through a reviewed cardinality policy; arbitrary extension IDs remain trace-only. Acceptance includes slow-API, CPU-heavy, wait-heavy, extension, import-heavy, and export-heavy fixtures whose summaries reconcile with their traces.

Aggregation rule

Aggregate completed operation spans. Do not count annotations as operations, and do not sum nested or overlapping child durations as though they were independent end-to-end time. Report root latency separately from self/child operation time and authoritative CPU measurements.

Workspace statistics snapshots

Sample the fast Rust workspace-statistics operation periodically and retain the latest local snapshot plus bounded history. Surface exact node/edge and other stable counts, GraphForge-owned disk footprint, and separate logical-size and volume-capacity fields where supported.

Show sample time, freshness, collection duration, and failure without substituting stale values as current. Tests prove periodic values equal direct stats reads, collection does not overlap without bound, and sampling performs no graph-data or recursive-directory scan.

Metadata

Metadata

Assignees

No one assigned

    Labels

    coreCore source code changesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions