Skip to content

cli: implement remote submission uploads Issue 1486 - #1518

Open
kbuma wants to merge 6 commits into
materialsproject:mainfrom
kbuma:issue-1486-cli-remote-archive-upload
Open

cli: implement remote submission uploads Issue 1486#1518
kbuma wants to merge 6 commits into
materialsproject:mainfrom
kbuma:issue-1486-cli-remote-archive-upload

Conversation

@kbuma

@kbuma kbuma commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Implements remote archive upload for emmet submit push, resolving #1486.

The CLI now stages submission changes, creates RawArchive HDF5 objects and a complete snapshot manifest, uploads them through presigned URLs, and finalizes the upload before advancing local submission history.

This targets the submission API introduced in #1515

Changes

  • Represent added, changed, unchanged, and removed calculations explicitly.
  • Generate content-addressed HDF5 archives and a canonical snapshot manifest.
  • Implement the prepare → PUT → finalize upload workflow.
  • Authenticate control-plane requests using X-API-KEY.
    • Prefer MP_API_KEY.
    • Temporarily support EMMET_API_TOKEN with a deprecation warning.
  • Cache upload sessions and checksums so interrupted or expired sessions can resume safely.
  • Validate server responses before committing local submission history.
  • Add:
    • emmet submit contributor-status
    • emmet submit status TARGET
  • Make CLI state updates atomic and protect the state directory and files.
  • Add httpx and archival dependencies and update generated requirements and CI coverage.
  • Document remote upload configuration and behavior.

Failure and retry behavior

  • Local history advances only after the server reports a completed session.
  • Failed uploads retain resumable state.
  • Expired presigned URLs are refreshed without re-uploading confirmed objects.
  • Object sizes and SHA-256 checksums are retained for finalization.
  • Credentials and presigned URLs are not persisted or included in errors.
  • Changes made to source files after staging abort the push.

Contributor Checklist

  • I have broken down my PR scope into the following TODO tasks
    • Model complete submission change sets.
    • Implement archive and manifest uploads.
    • Add resumable, atomic local state handling.
    • Align authentication and responses with [WIP] mock core submission route #1515.
    • Add contributor and submission status commands.
  • I have run the tests locally and they passed.
  • I have added tests, or extended existing tests, to cover new features and bugs fixed in this PR.

Tests

  • python -m pytest -q emmet-cli/tests — 110 passed
  • python -m mypy --config-file .mypy.ini emmet-cli/emmet
  • Ruff checks passed.
  • Black checks passed.
  • git diff --check passed.

@kbuma
kbuma requested a review from tsmathis September 4, 2026 20:36
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.95248% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.87%. Comparing base (17a87d0) to head (966ece3).

Files with missing lines Patch % Lines
emmet-cli/emmet/cli/upload.py 92.85% 24 Missing ⚠️
emmet-cli/emmet/cli/submit.py 91.83% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1518      +/-   ##
==========================================
+ Coverage   73.14%   73.87%   +0.73%     
==========================================
  Files         152      153       +1     
  Lines       13151    13576     +425     
==========================================
+ Hits         9619    10029     +410     
- Misses       3532     3547      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants