feat(encord): add Encord curation SaaS workbench tool (push/pull)

[jonlwowski012]

Summary

Adds an encord workbench tool: push media from the Nebius object store into
the Encord curation SaaS, curate headlessly with
quality filters (or in the Encord app), pull curated media and labels back
to S3, and verify the roundtrip end-to-end — every artifact carrying exact
identity and content checksums.

flowchart LR
    S3[("Your S3 bucket")]
    ENC["Encord dataset<br/>(items carry exact<br/>npa.source_uri identity)"]
    CUR["encord curate<br/>(headless quality filters,<br/>evaluated server-side)"]
    HCUR{{"or: curate in the<br/>Encord app"}}
    COLL["Curated Collection"]
    OUT[("Curated media + labels<br/>+ lineage manifest, in S3")]
    VER["encord verify<br/>(identity + size + checksum,<br/>fail-closed)"]

    S3 -->|"encord push<br/>(bytes stay in your bucket)"| ENC
    ENC --> CUR --> COLL
    ENC -.-> HCUR -.-> COLL
    COLL -->|"encord pull"| OUT
    OUT --> VER

Loading

Two transfer modes:

• register (the default everywhere: CLI, SDK, seed-demo, and every
  shipped spec but one): Encord references object URLs through an
  S3-compatible integration; source bytes never leave object storage. An
  empty --integration in this mode is rejected before any S3 or Encord I/O.
• upload: media bytes are copied into Encord-hosted storage. The only
  shipped spec that opts in is encord-cosmos3-augment.yaml, whose default
  source is the packaged public CC-BY-4.0 starter clip; its config comment and
  the docs say to switch to register (plus an integration title) when pointing
  it at your own data.

Docs: docs/workbench/encord.md (setup through troubleshooting) and
docs/workbench/encord-headless-curation.md (the headless-curation design and
the live evidence behind the pinned Encord filter shape).

Identity and integrity (adopts the #363 model)

This PR adopts the exact-identity model from #363 onto the live-validated
transport — thanks to @ for the repro that motivated it:

• Identity is never a filename. Every pushed item registers namespaced
  npa.source_uri clientMetadata; receipt lineage and dataset linking resolve
  through that metadata or the item's normalized objectUrl, and conflicting
  signals fail the item closed. Same-basename objects (ep_001/frame_0000.png
  vs ep_007/frame_0000.png) resolve to distinct items and percent-encoding
  aliasing (a%2Fb vs a/b) is handled in URI canonicalization — both pinned
  by regression tests. Resolution indexes the folder inventory once per pass,
  so a push over thousands of items stays linear.
• Content integrity in the durable artifacts. Receipts record per-item
  source size, verbatim ETag, and content checksum (single-part ETag as md5 in
  register mode; streamed sha256 in upload mode); pulls hash downloads
  in-stream and record the destination ETag for zero-egress copies. A
  same-bucket copy that fails falls back to download and records copy_error
  in the manifest row, so paid egress is never silent.
• encord verify joins receipt to manifest by Encord uuid
  (npa.encord.roundtrip_report.v1) and fails closed on: missing or
  unexpected items, size or checksum mismatches, a receipt whose status is not
  done (a write-ahead planned copy, a failed or timeout push), zero
  attributable items (0/0 is a failure, not a vacuous pass), and matched items
  with neither a comparable checksum nor a size on both sides
  (unverifiable). The report's defects list names receipt-level reasons.
  This is the machine-checkable form of any checksum claim; an overwrite
  between push and pull is exactly what it catches.
• Write-ahead receipts. Push and curate land a status: planned receipt
  before the first Encord mutation, so an uncatchable kill (OOM, preemption)
  still leaves a durable record of intent — pinned by tests that assert the
  planned receipt is already durable when the first mutating SDK call happens.
• Caller-side idempotency, both modes. Identities resolve against the
  folder before anything is sent: a retried stage makes zero registration
  round-trips and copies no duplicate bytes. Register mode never falls
  through to upload (pinned by test). Receipt counters (units_done,
  units_error) are one function of the per-item outcomes, so no item is ever
  counted twice or reported done without an attributable uuid.
• Pull never loses a record. One item's failed signed-URL fetch (or any
  other per-item exception) becomes an error row; the manifest keeps every
  item that landed, so resume and cleanup can see what is in the prefix.

Headless curation

npa workbench encord curate --folder ... --filter brightness:0.2:0.8 --filter width:640:4096 --collection ... maps workbench-declared filters
over Encord's built-in quality metrics onto a run-scoped Encord filter preset
and lets Encord evaluate the selection server-side into a Collection — no
human in the app, no media movement. The undocumented per-metric filter JSON
was pinned by a live spike against the real SaaS (an invalid shape hangs
Encord's evaluation; the verb allowlists live-verified metrics and fails
closed on anything else). Zero selections fail closed with a diagnostic that
names the likely cause. A --collection that already holds items is refused:
Encord's evaluation only adds, so a stale selection could not be told apart
from this run's. The receipt records items_total and items_selected; the
transient preset (npa-curate-<run-id>, or a timestamped random-suffixed
title for ad-hoc runs) is deleted in a finally, after a crash as well as a
success, and preset_deleted records whether that delete went through.

What's included

• npa.workbench.encord implementation (one module per verb, a SaaS seam that
  returns ResolvedRefs from one shared title-or-id resolution contract, and
  an S3-only artifact writer), the npa workbench encord push|curate|pull|verify|cleanup|seed-demo|system-info CLI on the shared
  workbench CLI helpers, and SDK wrappers for each verb. system-info is the
  management verb CONTRIBUTING asks new tools to expose (SDK pin, API domain,
  configured credential names — never values).
• Every CLI verb carries @json_stdout_contract: --output json prints
  exactly one JSON document on stdout, including on failure. Only
  EncordToolError and path-contract violations are client errors (exit 1);
  anything else propagates to app_entry as exit 2 with the NPA_DEBUG
  traceback path intact. The contract decorator now reads enum-typed
  output_format values, which also activates it for the other enum-typed
  commands in the repo.
• toolRefs workbench.encord.{push,curate,pull,verify,seed_demo}, credential
  hints (verify needs no Encord secret and its setup guard says so), and
  npa[encord] dependency support for image-less stages. The
  spec-declared-outputs guardrail pins push, curate, pull, and verify outputs
  to the tool's own *_uri_for helpers.
• Encord credential preflight via health preflight --checks encord. Exactly
  two credential transports: ENCORD_SSH_KEY_B64 (pods) and
  ENCORD_SSH_KEY_FILE (laptops) — the truncation-prone raw-PEM paste is not
  accepted (pinned by test). A credential with no encord package installed
  is a WARN with a pip remedy, not a failed credential.
• Pull media transfers run in a bounded parallel pool.
• encord cleanup --title-prefix <npa-...-> tears down run-scoped Encord
  state (folders, collections, presets; datasets are reported — the SDK
  exposes no dataset deletion). All created titles embed the run id, which
  embeds a UTC timestamp.
• Workflows: encord-push.yaml, encord-pull.yaml,
  encord-roundtrip-smoke.yaml (push → curate → pull → pull-curated →
  verify), encord-cosmos3-augment.yaml, and
  encord-cosmos3-groot-finetune.yaml (push → headless curate → pull →
  Cosmos 3 augment → materialize LeRobot episodes → GR00T fine-tune). All
  five are registered in the live submit matrix; groot-finetune needs
  --var encord_integration=<title> because it handles customer data in
  register mode.
• Operator documentation, the headless-curation design note (next to the
  operator doc, since it applies to one tool), tool-catalog rows, skill
  guidance, and the skill-index lines in CLAUDE.md / AGENTS.md.

Also in this PR (small, cross-cutting)

Two fixes surfaced while running the Encord workflows live and were kept
here rather than split out, because each blocked the runs above:

• StorageClient now bounds every boto network wait (connect_timeout=10,
  read_timeout=60, max_pool_connections=16): a stalled TCP connection
  during workflow-preflight source staging blocked a submit indefinitely.
  Pinned by a unit test; affects every tool that uses the shared client.
• scripts/build_docs.sh guards an empty-array expansion that bash 3.2 (the
  macOS default) rejects under set -u.

parse_bucket_uri in npa.clients.storage is now public (its two private
importers migrated). The SkyPilot procfs probe on this branch is byte-identical
to main's #367 fix, so merging main is a no-op there.

Validation

Everything below is a checked-in test or a durable S3 artifact from a live
run on Nebius (eu-north1 managed Kubernetes, SkyPilot managed jobs):

• Encord unit, CLI, and workflow tests cover identity regression (shared
  basenames, percent-encoding aliasing, conflict fail-closed), integrity,
  every verify fail-closed path (receipt status, zero items, unverifiable
  items, mismatches), write-ahead receipts landing before the first mutation,
  raw-PEM rejection, curate's preset lifecycle and populated-collection
  refusal, pull's per-item error rows and copy-fallback reason, push counter
  consistency and early integration validation, idempotency, cleanup,
  seed-demo, and system-info. Catalog argv, doc-sync, three-tier, spec-output,
  live-matrix, and skills-index guardrails are green.
• Live roundtrip on encord-roundtrip-smoke, run
  encord-roundtrip-smoke-20260902T162808351118Z-ab1bfb (managed jobs
  100–104, all SUCCEEDED, on the code in this PR):
  • push registered 4 fixtures in place, each resolved by the metadata
    identity signal, md5 checksums recorded;
  • curate selected 3 of 4 (items_total 4) — the width filter excluded
    the deliberately low-res fixture, proving server-side filtering; the
    transient preset was deleted (preset_deleted: true);
  • dataset pull returned 4 items and curated pull 3, all zero-egress
    server-side copies, no copy_error;
  • verify passed: 4/4 matched, 4/4 checksums verified, 0 mismatched, 0
    unavailable, 0 unverifiable, no defects (verify/roundtrip_report.json
    in the run prefix);
  • encord cleanup then removed the run's folder (4 items) and collection
    and reported the one dataset the SDK cannot delete.
• Register-mode re-push verified idempotent live (no registration
  round-trips, stable uuids).

Note on earlier evidence wording: a prior commit message claimed checksum
verification before the mechanism existed in-tree; this PR makes that claim
real and machine-checkable — the roundtrip report is the artifact, and the
claim now lives nowhere except where CI and the smoke can enforce it.

Boundaries

• .mp4, .png, .jpg, and .jpeg are supported push inputs. MCAP cloud
  registration is not supported by the pinned Encord SDK upload format;
  .mcap inputs are recorded as receipt-visible experimental_error rows and
  fail closed rather than sending an inferred schema.
• Composite Encord items (image groups, DICOM) have no single signed URL;
  pull records them as per-item errors.
• Curate's computed quality metrics (brightness, sharpness, file-size) match
  nothing until quality metrics are computed for the folder — a one-time
  action in the Encord app with no public API. Intrinsic metrics (width,
  height, area, aspect-ratio) work on any folder immediately; the
  zero-selection error names the cause.
• Curate populates only an empty or freshly created Collection; use run-scoped
  titles (the shipped specs do) or cleanup a stale one first.
• The Encord SDK cannot delete datasets; cleanup deletes everything else
  and reports them.

[timothy-le7]

My main workflow comment is that a user should be able to leverage headless encord capabilities from workbench. It makes sense to have the ability to operate in Encord, but we should think from how an agent would drive this and minimize human in the loop