feat(encord) 3/5: Pull from Encord

[jonlwowski012]

Summary

Third of five stacked PRs replacing #339 (prereqs → push → pull → headless curate → verify). Stacked on encord/02-push; review only the last commit until that merges.

Vision. Encord is the curation-SaaS tool in the npa workbench marketplace. Media already in the customer's Nebius object store is registered in place into Encord (bytes never leave the bucket), curated either headlessly with quality-metric filters or by humans in the Encord app, and pulled back to S3 as curated media + labels + a lineage manifest. Every artifact carries exact identity (npa.source_uri, never a filename) and content checksums, and encord verify proves the roundtrip fail-closed. The tool is in-process (no container) and composes through workbench.encord.* toolRefs in npa.workflow specs.

flowchart LR
    S3[("Your S3 bucket")]
    FOUND["Foundations · PR 1<br/>contracts · exact identity · checksums<br/>Encord SDK seam · credentials + preflight"]
    PUSH["encord push (+ cleanup) · PR 2"]
    ENC["Encord folder / dataset"]
    CUR["encord curate (headless) · PR 4"]
    HCUR{{"or: Encord app"}}
    COLL["Curated Collection"]
    PULL["encord pull · this PR"]
    OUT[("Curated media + labels<br/>+ manifest, in S3")]
    VER["encord verify · PR 5"]
    FOUND -.-> PUSH & CUR & PULL & VER
    S3 --> PUSH --> ENC
    ENC --> CUR --> COLL
    ENC -.-> HCUR -.-> COLL
    COLL --> PULL --> OUT
    ENC --> PULL
    OUT --> VER
    PUSH -. receipt .-> VER
    classDef done fill:#d9ead3,stroke:#38761d,color:#000
    classDef this fill:#fff2cc,stroke:#bf9000,stroke-width:3px,color:#000
    classDef todo fill:#f3f3f3,stroke:#999,stroke-dasharray:4 3,color:#777
    class FOUND,PUSH done
    class PULL this
    class CUR,VER todo

Loading

This PR solves the Encord → S3 edge, from either a dataset or a curated Collection. With push and pull both merged the loop closes for the first time through the human path: register in place, curate in the Encord app, pull the Collection back to S3 with a lineage manifest.

encord pull

npa workbench encord pull --source {collection|dataset|project} --source-id <id-or-title> --output-path s3://… materializes media, per-item JSON, and (for projects) label rows into an S3 prefix with an npa.encord.pull_manifest.v1 manifest.

• Zero-egress by default. When the signed URL points at the configured endpoint the item is copied server-side bucket-to-bucket and the destination ETag is recorded. A failed copy falls back to a signed-URL download and the manifest row records copy_error, so paid egress is never silent.
• Hashed in-stream. Downloads are sha256'd as they stream; registered copies carry the source's md5-as-ETag. Both land in the manifest, so the later verify verb has something to compare.
• Never loses a record. One item's failed signed-URL fetch (or any per-item exception) becomes an error row; the manifest keeps every item that landed, so resume and cleanup can see what is in the prefix. The manifest is written before any failure exit and any failed item fails the command closed.
• Composite items (image groups, DICOM) have no single signed URL and are recorded as per-item errors. Transfers run in a bounded thread pool.

Surface

• CLI pull and SDK pull on the group and module PR 2 created; PullManifest re-exported.
• toolRef workbench.encord.pull with encord-pull.yaml (CPU-only, terminal at the manifest; plan-only twin in the live matrix, because no standalone submit can supply a human-curated id).
• Operator doc gains the "curate in the Encord app" table, the Pull section, SDK pull, and the pull spec; catalog-doc row; generated docs/cli/encord.md.

Tests

test_encord_pull.py (same-endpoint detection for path-style and virtual-hosted URLs, server-side copy, composite-item error, cross-origin download through a stubbed httpx.stream, copy-fallback reason, per-source enumeration, manifest on errors, one raising signed-URL fetch keeps every record, labels crash still writes the manifest, expired-URL retry, empty source fails closed). The shared fakes gain stub_httpx_stream. CLI tests: pull help, text and JSON output, path contract, bad --source, missing-credential path, EncordToolError exit. Workflow tests: pull spec structure, argv rendering, CPU-only resources, secret hint.

Validation

Gate	Result
ruff	clean
PR-3 targeted tests + coupling guardrails	386 passed
validate-spec encord-pull.yaml	valid
harness guardrails	2,442 passed
full make test	12,924 passed; the same 73 macOS host failures as PRs 1 and 2 and clean origin/main, no new failures
docs-drift	up to date

Stack

1. prereqs
2. encord push (+ cleanup)
3. this PR — encord pull
4. headless encord curate (+ system-info), encord-roundtrip-smoke.yaml
5. encord verify, skill + docs completion

🤖 Generated with Claude Code