Skip to content

docs: document batch Zstandard archive generation #1138

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xiangfu0 merged 1 commit into from
Oct 23, 2026
Merged

docs: document batch Zstandard archive generation #1138

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions reference/configuration-reference/job-specification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,18 @@ pinot-admin.sh LaunchDataIngestionJob \
-propertyFile /path/to/job.properties
```

## Segment archive compression

Standalone, Spark and Hadoop batch segment generation accept the top-level `segmentTarCompressionCodec` property. The default is `gzip` (`.tar.gz`); set `zstd` to produce Zstandard archives (`.tar.zst`):

```yaml
segmentTarCompressionCodec: zstd
```

Enable Zstandard only after all consumers of these archives, including controllers and servers, support the codec. Keep `gzip` during mixed-version upgrades if any consumer cannot read Zstandard. Metadata sidecars remain `.metadata.tar.gz`, regardless of the segment codec.

Generation accepts only `gzip` and `zstd`. Batch push and backfill archive discovery also recognize existing `.tar.lz4` files; this does not enable `lz4` as a generation setting. Discovery excludes metadata sidecars. Update any `pushFileNamePattern` restricted to `*.tar.gz` when switching to Zstandard; a pattern selecting no archives results in no push or consistent-replacement action.

## Template your job spec file

Users are allowed to define some variables in the job spec file to make it a template then passing the variables at runtime.
Expand Down
Loading