Radio Kills The Video Star -- Content-addressed file delivery for hostile networks.
Satellite links, maritime VSAT, hotel WiFi, 3G roaming, field radios. These networks are expensive, slow, intermittent, and sometimes half-duplex. Existing tools (rsync, rclone, scp) assume a stable connection and fail hard when the link drops at byte 490,000,000 of a 500 MB transfer.
- Content-addressed storage. Files are split into variable-size chunks with FastCDC, hashed with BLAKE3, compressed with zstd. Identical content is stored once regardless of filename or location.
- Catalog-first. Metadata syncs first into a local SQLite catalog. Browse remote file trees offline. Estimate transfer cost before a single byte of payload moves.
- Chunk-level resume. No file-level resume. If a 500 MB transfer (125 chunks) drops at chunk 120, resume at 121.
- Cost-aware scheduling. A Grade x CostTier decision matrix controls when jobs run. URGENT always transfers. BACKGROUND waits for a free link. Estimate first, fetch when the price is right.
- Tar interface.
tar cf - /local/ | rk ingestandrk tar <tape>/<path> | tar xf -. Unix composability -- no proprietary archive formats. - QUIC transport. Connection migration (roaming WiFi to 4G), 0-RTT reconnect, multiplexed streams, BBR congestion control built for lossy links.
Inspired by UUCP (1978): store-and-forward semantics, priority grades, cost-aware scheduling. Modern guts.
Requires Rust 1.85+ (edition 2024) and a C compiler (for SQLite).
git clone https://github.com/gustaveiffel/rktvs.git
cd rk
cargo build --releaseThe binary is at target/release/rk. Data is stored in ~/.rk by default. Override with --data-dir or the RK_DATA_DIR environment variable.
Ingest files from a tar stream:
tar cf - /path/to/files/ | rk ingest --tape myprojectOr index files in-place (zero-copy, no data copying):
rk index --tape myproject /path/to/files/List what was stored:
rk ls myproject/Export back to tar:
rk tar myproject/some/subdir | tar xf - -C /tmp/restore/Verify integrity of indexed files:
rk verify --tape myproject
rk verify --tape myproject --blake3 # full hash verificationInitialize a hub (generates TLS certificate). Use --san for every IP or
hostname satellites will connect to:
rk --data-dir /tmp/rk-hub hub init --san 192.168.1.10Ingest some content on the hub, then start serving:
tar cf - /data/ | rk --data-dir /tmp/rk-hub ingest --tape demo
rk --data-dir /tmp/rk-hub hub serveOn another machine (or another terminal):
rk library add myhub 127.0.0.1:4443 --cert /tmp/rk-hub/hub.cert.der
rk library ping myhub
rk library listSync the catalog (fetch file metadata from the hub):
rk library sync myhub
rk ls myhub:demo/Fetch files:
rk fetch myhub:demo/path/to/file --grade normalrk estimate myproject/large-dataset.binrk jobs
rk jobs --status pending| Command | Description |
|---|---|
rk ingest --tape <name> |
Read a tar archive from stdin, chunk and store it |
rk tar <tape>/<path> |
Export files as a tar archive to stdout |
rk ls <tape>/<path> |
List local files in the catalog |
rk ls <library>:<tape>/<path> |
List remote library files (requires catalog sync) |
rk --version |
Show version and protocol version |
rk index --tape <name> <dir> |
Index files in-place (zero-copy by-reference chunking) |
rk verify [--tape <name>] [--blake3] |
Verify integrity of indexed files |
rk estimate <tape>/<path> |
Estimate transfer cost (chunks needed, bytes to fetch) |
rk jobs [--status <s>] |
List transfer jobs, optionally filtered by status |
rk hub init [--listen <addr>] [--san <ip-or-host>]... |
Initialize a hub (generates TLS certificate) |
rk hub serve [--listen <addr>] |
Start the hub QUIC server |
rk library add <id> <endpoint> --cert <path> [--force] |
Register a remote library with cert pinning |
rk library list |
List known libraries |
rk library remove <id> |
Remove a library |
rk library ping <id> |
Ping a library to check connectivity |
rk library sync <id> [--tape <name>] |
Sync catalog metadata from a remote library |
rk fetch <library>:<tape>/<path> [--grade <g>] |
Fetch a file from a remote library (supports glob patterns: *.mov, src/**/*.rs) |
Grades: urgent (P0), normal (P1), batch (P2), background (P3).
The project is a Cargo workspace with four crates:
| Crate | Purpose |
|---|---|
rk-core |
ChunkStore (BLAKE3 + zstd), FastCDC chunker, SQLite catalog, Manifest (zero-copy), Indexer, Verifier, ChunkResolver |
rk-tar |
Tar archive ingest and export via streaming |
rk-transport |
QUIC hub server and satellite client (quinn), TLS cert management, protobuf wire protocol |
rk-scheduler |
Grade/CostTier decision matrix, job queue, transfer estimation, resumable chunk fetcher |
The top-level src/main.rs provides the rk CLI built with clap.
Key concepts:
- Library -- an rk server (the hub). Satellites connect to libraries to sync catalogs and fetch chunks. Trust is established via certificate pinning.
- Tape -- a named logical volume grouping files with ACLs, sync policy, and versioning. Chunks are shared across tapes.
- Chunk -- a variable-size data block (~4 MB avg) identified by its BLAKE3 hash, compressed with zstd.
- Manifest -- a
.rkmfile mapping chunk hashes to file offsets for zero-copy by-reference chunking. - Grade -- transfer priority: Urgent, Normal, Batch, Background.
- CostTier -- link classification: Free, Cheap, Metered, Expensive.
See doc/ARCHITECTURE.md, doc/PROTOCOL.md, doc/TAPES.md, doc/COMPARED.md, doc/GUIDE.md, and doc/API.md for details.
Alpha -- active development.
What works today:
- Content-addressed chunk store with BLAKE3 hashing, zstd compression, and integrity verification on read
- FastCDC content-defined chunking with native deduplication
- SQLite catalog with 9 tables (libraries, tapes, files, chunks, jobs, ACLs, versioning)
- Tar ingest from stdin and tar export to stdout
- Zero-copy by-reference chunking (Manifest, Indexer, Verifier, ChunkResolver)
- QUIC transport layer with hub server and satellite client
- TLS with self-signed certificates and cert pinning (TOFU model)
- Hub init/serve and library add/list/remove/ping CLI commands
- Remote chunk fetch with hash verification and chunk-level resume
- Compressed wire transfer (chunks sent zstd-compressed from store, no decompress/recompress)
- Connection/stream limits (256 streams, 1024 connections) and timeouts (10s connect, 30s chunk, 300s idle)
- Endpoint validation at library add, relative cert path storage
- Cascade delete on library remove (files, chunks, tapes, jobs)
- Schema migrations for upgrading from earlier database versions
- Protobuf wire protocol
- Grade x CostTier scheduling matrix
- Transfer cost estimation without data transfer
- Resumable chunk-level fetching with job tracking
- Catalog sync between hub and satellite (
rk library sync)
What is planned:
- Connection reuse and 0-RTT reconnect
- Link cost auto-detection
- Tape ACL enforcement
- Cache eviction policies
- Mutual TLS authentication
Contributions are welcome.
- Open an issue to report bugs or discuss features before writing code.
- Fork the repo, create a branch, submit a pull request.
- Run
cargo test --workspacebefore submitting. All tests must pass. - Follow existing code style. No
unsafewithout justification.
Requires cargo-release and git-cliff.
cargo release patch # 0.1.0 → 0.1.1 (dry-run by default)
cargo release minor # 0.1.0 → 0.2.0
cargo release --execute # actually do itThis bumps the version in all crates (workspace inheritance), generates CHANGELOG.md via git-cliff, commits, and tags. No push — review locally first.
AGPL-3.0. See LICENSE for the full text.