feat(encord) 4/5: Encord headless curate, system-info, roundtrip smoke spec

[jonlwowski012]

Summary

Fourth of five stacked PRs replacing #339 (prereqs → push → pull → headless curate → verify). Stacked on encord/03-pull; review only the last commit until that merges.

Vision. Encord is the curation-SaaS tool in the npa workbench marketplace. Media already in the customer's Nebius object store is registered in place into Encord (bytes never leave the bucket), curated either headlessly with quality-metric filters or by humans in the Encord app, and pulled back to S3 as curated media + labels + a lineage manifest. Every artifact carries exact identity (npa.source_uri, never a filename) and content checksums, and encord verify proves the roundtrip fail-closed. The tool is in-process (no container) and composes through workbench.encord.* toolRefs in npa.workflow specs.

flowchart LR
    S3[("Your S3 bucket")]
    FOUND["Foundations · PR 1<br/>contracts · exact identity · checksums<br/>Encord SDK seam · credentials + preflight"]
    PUSH["encord push (+ cleanup) · PR 2"]
    ENC["Encord folder / dataset"]
    CUR["encord curate (headless) · this PR"]
    HCUR{{"or: Encord app"}}
    COLL["Curated Collection"]
    PULL["encord pull · PR 3"]
    OUT[("Curated media + labels<br/>+ manifest, in S3")]
    VER["encord verify · PR 5"]
    FOUND -.-> PUSH & CUR & PULL & VER
    S3 --> PUSH --> ENC
    ENC --> CUR --> COLL
    ENC -.-> HCUR -.-> COLL
    COLL --> PULL --> OUT
    ENC --> PULL
    OUT --> VER
    PUSH -. receipt .-> VER
    classDef done fill:#d9ead3,stroke:#38761d,color:#000
    classDef this fill:#fff2cc,stroke:#bf9000,stroke-width:3px,color:#000
    classDef todo fill:#f3f3f3,stroke:#999,stroke-dasharray:4 3,color:#777
    class FOUND,PUSH,PULL done
    class CUR this
    class VER todo

Loading

This PR solves the machine path through the middle of the loop. The dashed "Encord app" alternative stays available, but a workflow can now select a Collection server-side with declared quality filters and no human in the UI. The roundtrip smoke spec (push → curate → pull → pull-curated) is the executable proof and the vehicle that makes the curate toolRef reachable. system-info lands here because it reports both push's media categories and curate's metric allowlist.

Headless encord curate

npa workbench encord curate --folder … --filter brightness:0.2:0.8 --filter width:640:4096 --collection … --output-path s3://… maps workbench-declared filters over Encord's built-in quality metrics onto a run-scoped Encord filter preset and lets Encord evaluate the selection server-side into a Collection: no human in the app, no media movement.

• Pinned filter shape. The per-metric filter JSON is undocumented in the SDK (filters: List[Dict]); a live spike against the real SaaS pinned it, and found that an invalid shape hangs Encord's evaluation indefinitely. The verb therefore allowlists live-verified metrics and fails closed on anything else before any Encord call. Intrinsic metrics (width, height, area, aspect-ratio) work on any folder; computed ones (brightness, sharpness, file-size) match nothing until quality metrics are computed once in the app, and the zero-selection error names that cause.
• Fail-closed selection. Zero selected items is exit 1, never a vacuous success. A --collection that already holds items is refused: Encord's evaluation only adds, so a stale selection could not be told apart from this run's.
• Asynchronous, one-shot evaluation handled. Items pushed moments earlier may not be metric-indexed yet (observed live), so while the selection is empty the verb re-issues the evaluation every 15s until --poll-seconds runs out.
• Receipt and lifecycle. A write-ahead planned receipt lands before the first mutation; the final npa.encord.curate_receipt.v1 records the parsed filters, the exact preset JSON, items_total, items_selected, and preset_deleted. The transient preset (npa-curate-<run-id>, or a timestamped random-suffixed title for ad-hoc runs) is deleted in a finally, after a crash as well as a success.

system-info reports SDK pin, API domain, and configured credential names (never values), plus the supported media and curate metrics: the management verb CONTRIBUTING asks new tools to expose.

Surface

• CLI curate, system-info; SDK curate, system_info; CurateReceipt re-exported; resolve_collection gains create_in_folder_uuid.
• toolRef workbench.encord.curate and encord-roundtrip-smoke.yaml (push → curate → pull → pull-curated, CPU-only), registered as the live CPU case in the submit matrix; push and pull now name it as their executable twin. The fixture set deliberately includes a 64×64 thumbnail the width filter excludes, so a green run proves actual filtering (3 of 4 items curated with the shipped fixtures). The verify stage is added in PR 5.
• Docs: the operator guide gains "Curate headlessly", system-info, the smoke spec, and three troubleshooting rows; docs/workbench/encord-headless-curation.md records the design and the live spike evidence behind the pinned filter shape; catalog-doc row; generated docs/cli/encord.md.

Tests

test_encord_curate.py (filter parsing incl. comma-separated form, the pinned preset JSON, happy path creating collection + preset, planned receipt before the first mutation, reuse of an empty collection, refusal of a populated one, re-issued evaluation until indexing catches up, zero selection fails closed with a receipt, preset deleted when evaluation raises, a failed preset delete recorded, empty folder and unknown metric fail before any scaffolding, missing folder writes the receipt then raises). Two resolver tests join test_encord_client.py. CLI tests: curate help, JSON and text output, path contract, tool error, and system-info making no API call. Workflow tests: smoke step order and schema chain, curate argv, secret hint, CPU-only resources.

Validation

Gate	Result
ruff	clean
PR-4 targeted tests + coupling guardrails (incl. schema chaining, daily coverage)	444 passed, 1 skipped
validate-spec encord-roundtrip-smoke.yaml	valid (states: push, curate, pull, pull-curated)
harness guardrails	2,433 passed
full make test	12,958 passed; the same 73 macOS host failures as PRs 1–3 and clean origin/main, no new failures
docs-drift	up to date
live encord-roundtrip-smoke	not yet run from this branch (needs a cluster + the Encord integration); the prior run on the monolithic branch curated 3 of 4 fixtures with preset_deleted: true

Stack

1. prereqs
2. encord push (+ cleanup)
3. encord pull
4. this PR — headless encord curate (+ system-info), encord-roundtrip-smoke.yaml
5. encord verify, smoke gains its verify stage, skill + docs completion

🤖 Generated with Claude Code