Skip to content

[Bug]: curate writes a manifest with a typo'd output scheme to a local directory #721

Open

Description

@chiruu12

Version or commit

main at 4ac0a7b

Environment

macOS, Python 3.12, uv sync --locked --all-extras

Minimal reproduction

from hflow.curation import curate

curate(catalog_root, "SELECT episode_id FROM episodes", output="gcs://my-bucket/manifest.parquet")

or hflow curate -o gcs://my-bucket/manifest.parquet "SELECT episode_id FROM episodes".

Expected behavior

A ValueError naming the unsupported scheme, the same as parse_storage_root gives for a data root. Its docstring says a typo'd scheme "is refused loudly rather than treated as a strange directory name".

Actual behavior

The manifest is written to ./gcs:/my-bucket/manifest.parquet and the report says it worked:

CurationReport(manifest_path=PosixPath('gcs:/my-bucket/manifest.parquet'), row_count=1, ...)

The CLI prints manifest: gcs:/my-bucket/manifest.parquet (1 rows, from 1 cataloged episodes) and exits 0. FILE:///abs/manifest.parquet does the same and lands under ./FILE:/abs/.

Additional context

curate() treats output as a bucket only when is_bucket_url() says so, and as a file URL only when it starts with a lowercase file://. Everything else goes to Path(output). The comment above the file:// branch says it matches parse_storage_root's convention, but that function lowercases the scheme and refuses anything outside file and BUCKET_URL_SCHEMES.

Proposed fix: resolve a string output through parse_storage_root and match on the result, so the manifest follows the same scheme rules as a data root. Plain paths and file:///abs keep working, and an unknown scheme fails before the query runs. Tests for the typo'd scheme and the uppercase FILE:// case.

I found this reading curate(). Happy to take it if you want to assign it to me.

Activity

  1. kstonekuan commented on Oct 13, 2026

    @kstonekuan
    Contributor

    Confirmed, thanks. Your proposed fix is right: resolve a string output through parse_storage_root so the manifest follows the same scheme rules as a data root. It's yours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions