All settings live in one TOML file. The doc comments on Config in
crates/pipette-core/src/config.rs are
the key reference, examples/config.toml the
annotated example, and cli.md says how the file is
found and where credentials come from. A missing evals_server_url fails the
load.
Every environment needs a [database] section, because the records and the job
queue are in PostgreSQL. A config without [database] fails to load. A bare
[database] section is valid: the libpq PG* environment variables supply the
connection.
[todo_storage], todo_attempt_timeout_secs and todo_operation_timeout_secs
are retired keys. A config that has one of them fails to load. Delete each one
from every deployed config.
serve holds no TLS and binds every interface by default. See
§5.6 for the deployment shape that requires.
# HTTP server
pipette-mgmt serve
# Fast submission pass
pipette-mgmt process-submissions
# Slow eval scorer
pipette-mgmt score-evalserve shuts down gracefully on SIGTERM (what a container runtime sends
before it kills the process) or SIGINT (^C). It stops accepting new
connections and lets the requests already in progress finish, so a rolling
restart does not cut a client off mid-submission. The log records
shutdown requested with the signal, then server stopped once the last
request completes.
The wait is capped at 20 seconds. If requests are still running then, the
server logs a
connections still open at shutdown are abandoned when the process exits
warning; they keep running while the server spends up to 10 more seconds
writing the audit events in flight, and the process then exits and abandons
them.
Set the runtime's grace period above 30 seconds so it never kills the server
before that.
serve speaks HTTP/1.1 only, as the server library bounds no HTTP/2
connection's idle time. It closes a connection that does not finish its request
headers within read_timeout_secs (default 75 seconds), and fails a request
whose body sends nothing for that long (408). The header timeout covers an
idle keep-alive connection too, so set read_timeout_secs above the idle
timeout of the load balancer in front (the ALB default is 60 seconds): the
balancer must close an idle connection first, or a request it sends on one the
server is closing fails with 502. Raise both together when you raise the idle
timeout for long imports
(§5.7).
A body that trickles in, or an answer the client never reads, has no time
bound. serve instead holds at most max_connections (default 512)
connections at once; a further one waits in the listen backlog until one
closes. Keep it below the process's open-file limit. The cap is shared by all
clients, so one host holding idle or slow connections can fill it: the listener
must only be reachable from the load balancer or proxy
(§5.6), which completes request headers before it
opens a connection to the server.
Scoring runs as two crons. process-submissions is the fast pass and
score-eval the slow one, which eval benchmarks need
(cli.md). score-eval
holds its own lock, so an overlapping tick exits rather than double-scoring, and
never blocks the fast pass.
# /etc/cron.d/pipette-mgmt-score
*/5 * * * * app /usr/local/bin/pipette-mgmt --config /etc/pipette-mgmt/config.toml process-submissions 2>&1 | logger -t pipette-mgmt-process-submissions
# required for eval benchmarks
*/5 * * * * app /usr/local/bin/pipette-mgmt --config /etc/pipette-mgmt/config.toml score-eval 2>&1 | logger -t pipette-mgmt-score-eval
On S3 backends the cron environment needs AWS credentials (AWS_*, or an
EC2/ECS role).
Run process-submissions, score-eval, and queue-maintenance once per
schedule interval, regardless of server replica count. Use one cron host or one
Kubernetes CronJob. See queue.md §Processes.
Two recurring tasks keep GET /datasets/{benchmark_id} servable
(cli.md). Schedule them
however the deployment schedules recurring work: a scheduled container task in
production, cron on a single host.
pipette-mgmt --config /etc/pipette-mgmt/config.toml dataset-cache probe # every minute
pipette-mgmt --config /etc/pipette-mgmt/config.toml dataset-cache build # every 15 minutes
probemust run every minute. Its verdict is all the route reads, and a verdict older thandataset_probe_max_age_secs(default 180) refuses with503. A stopped probe therefore stops serving within three minutes rather than serving on a staleok.buildonly acts on a missing artifact, so its cadence bounds how long a new catalog pin waits to become servable. It never rebuilds or deletes.- An eval corpus is the scores service's package. The scores service packs
every
vlm_evalandasr_evalcorpus intoscores-packages/tars/of[eval_packages_storage](default[storage]), andGET /datasetsstreams it from there.buildregisters such a pin without downloading it: the package must have the pinned size and a manifest naming the pin, and the evals server must still serve the pinned samples. A pin whose package the scores service has not written fails every build until it is there. mgmt only readsscores-packages/tars/; on S3 its role also needss3:ListBucketon that prefix, so an absent package reads as absent, not as access denied. - The two hold separate locks, so an 80 MB build never delays the minute probe, and a second run of either exits rather than overlapping.
- A crashed build keeps its lock until the lease expires. On
s3thebuildlease is the run's worst case: every pinned artifact atdataset_build_timeout_secs, plus a minute, where an eval corpus counts only while it is unregistered. So a build killed mid-run (out of memory, an evicted task) blocks the next ones for up to that long. A registered eval corpus costs nothing. To clear it sooner, first make sure nobuildis still running, then delete<storage-root>/locks/dataset-cache-build.lock(the lease object; its body names the holder, host, pid andexpires_at).unlockclears only the mutate lock. Onlocal_fsthe lock is aflock(2)that dies with its process, so there is nothing to clear. - A build checks its corpus against its entry. An asr-eval corpus must
hold its entry's
parameter_sample_countsamples, as its package's manifest counts them, or the build fails andGET /datasetsrefuses the entry, rather than every device downloading a corpus it then refuses. A corpus already registered is held to the same count, so an entry restated with another count fails the next build too. - Only git corpora need build scratch, tens of megabytes each. After an
evals server restart, a set's first probe may read
unavailableuntil the server has its digests again. - Both shell out to
git: the host or image needs it onPATH, and outbound HTTPS to each upstream. - Both call
evals_server_urlfor an eval corpus, for/samples/digestalone:probeevery minute,buildonce per registration.
Apply the SQL migration before you run the importer. Set GITHUB_TOKEN in
the cron environment. The token needs read access to public GitHub releases.
Run one import each day at 03:00 UTC. Use flock to stop a second run from
starting before the first run ends:
# /etc/cron.d/pipette-mgmt-catalog
CRON_TZ=UTC
0 3 * * * app flock -n /run/lock/pipette-mgmt-catalog.lock /usr/local/bin/pipette-mgmt --config /etc/pipette-mgmt/config.toml catalog import 2>&1 | logger -t pipette-mgmt-catalog
A daily entry takes the newest complete build that is older than
descriptor_catalog.llamacpp_min_build_age_secs. The importer skips a younger
build. The entry never changes after the importer creates it. So a run that starts
less than the minimum age after 00:00 UTC can name the day after an older
build. Keep the cron at least the minimum age after 00:00 UTC. The default age
is 10800 seconds, which fits the 03:00 UTC cron. If you increase the age, move
the cron later by the same amount. Do not run a manual import before that time.
Each run applies automatic expiry to the imported entries
(cli.md). An imported entry with an old release
starts hidden. To import older daily entries, run
catalog import --since YYYY-MM-DD with the first day to check. The importer also checks all stable releases.
Alert for a stale catalog importer. The importer writes one line to the
pipette-mgmt-catalog log at the end of each run:
INFOcatalog import run finished: the run succeeded. The line has the fieldsrun_id,command,entries_created,members_added,entries_hidden, andunclassified_assets.ERRORcatalog import run failed: the run recorded an error. The line has the same fields, pluserror.
Alert when no catalog import run finished line with command=import appears
in 36 hours. Set the alarm to fire on missing data, because a stopped schedule
writes no line. Enable the alarm after you enable the daily import schedule.
A run that stops before it records its end writes no line. Its row stays
running, and the query below reports it as interrupted after 6 hours.
When the alert fires, run this query against PostgreSQL. The
latest_import_outcome column uses the same rule as the outcome field of the
run history:
SELECT success.last_successful_import_at,
success.last_successful_import_at IS NULL
OR success.last_successful_import_at < now() - interval '36 hours' AS stale,
CASE
WHEN latest.id IS NULL THEN 'never_run'
WHEN latest.result = 'running'
AND latest.started_at < now() - interval '6 hours' THEN 'interrupted'
WHEN latest.result = 'running' THEN 'running'
WHEN latest.result = 'success' THEN 'success'
ELSE 'failed'
END AS latest_import_outcome,
latest.error_summary
FROM (
SELECT MAX(finished_at) AS last_successful_import_at
FROM catalog_import_runs
WHERE command = 'import' AND result = 'success'
) AS success
LEFT JOIN LATERAL (
SELECT id, started_at, result, error_summary
FROM catalog_import_runs
WHERE command = 'import'
ORDER BY started_at DESC, id DESC
LIMIT 1
) AS latest ON true;The alert detects a stopped daily import schedule, even if the catalog still
lists active entries. A successful refresh does not reset it.
Unclassified assets. An asset that the flavor table does not know becomes
no member, but the run still succeeds. The importer logs each asset at INFO
with the message catalog asset is not classified and the fields
release_tag and asset. The run history lists the assets of each run. If a
build variant is necessary:
- Find the asset names in the run history.
- Add the flavor to the flavor table.
- Run
catalog refreshfor each affected entry that is not active. The next import adds the member to the active entries.
For an upstream or storage error, inspect the run's error_summary and the
pipette-mgmt-catalog logs.
queue-maintenance reconciles everything stale in the job queue; what each
run does is cli.md, and the protocol it
maintains queue.md.
Run every 1 to 5 minutes. The interval is a latency knob, not a correctness
requirement: every pass is idempotent, and the handlers enforce lease and expiry
rules themselves. But a crashed device's job becomes claimable again only at
lease expiry plus one interval, so if you shorten plan_lease_duration_secs
to speed up dead-device turnaround, tighten the cron to match. The eligible
index reflects new jobs, profile changes, approvals and permission changes
within one interval, and a new job is claimable only after the run that indexes
it. Every minute is the right default when claim latency matters:
# /etc/cron.d/pipette-mgmt-queue
* * * * * app flock -n /run/lock/pipette-mgmt-queue.lock /usr/local/bin/pipette-mgmt --config /etc/pipette-mgmt/config.toml queue-maintenance 2>&1 | logger -t pipette-mgmt-queue
queue-maintenance takes no lock of its own, so serializing runs is a
correctness requirement, not just an optimization: the flock -n above
makes a tick exit immediately while the previous run is still going, rather
than queue behind it. A skipped tick loses no work: each run reconciles
everything outstanding at that moment.
Overlapping runs can be destructive. The plan-index GC deletes a content key,
a membership marker or a pending-source marker only after two consecutive runs
find it orphaned. Each run records its first sightings in the
plan_index_gc_candidates table for the next run. Each run clears that table
as it starts, so a run that fails partway leaves no stale sightings. It only
delays GC by one interval. The rule is safe only because the gap between two
sightings is a full cron interval. In that time, an ingest that was in
progress writes the records that make the key live. Two overlapping runs can
count two sightings seconds apart and delete a live key. A missing membership
marker lets teardown retire a job that another plan still needs.
If you deploy as a Kubernetes CronJob, set concurrencyPolicy: Forbid to
get the same serialization the flock -n provides. An EventBridge schedule
that starts an ECS task starts a new task at each scheduled time. Put a
launcher between the schedule and the task. The launcher checks the task
family for a running task. It starts a task only if no task runs.
Lost leases. A job whose lease lapses plan_lost_lease_limit times (default
3) is retired as a lease_lost failure, with a WARN
queue-maintenance: retired a job whose lease was lost {n} times. Alert on it:
the job is likely crashing or disconnecting the devices that run it. To give a
retired job another chance, campaigns rerun --job it; to lift a count, delete
the job's rows from queue_lost_leases. A serve outage longer than the lease adds one loss to every
leased job, so before long planned serve downtime pause this cron or set the
limit to 0, which turns the count off.
Stranded ingests. A manifest still creating past
plan_creating_max_age_secs (default 6 hours) marks a crashed ingest, and its
jobs are torn down. Set it above the longest expected ingest
(plan-ingestion.md §8).
Campaign group policies. Pass 11, the last, applies the
policies that are on. It reads
their results from pipette-duckdb, so while a policy is on the host needs a
[duckdb] section
(storage.md); without
one every run fails with a WARN that names it. The pass starts no member
once the run is more than 60 s past its start, the cron's cadence, so a run
holds back the next run's lease recycling by at most one member's read and
revision. A revision of a plan with thousands of jobs can still take minutes.
A run takes the members by when the pass last revised each or failed to read
its rows, longest ago first, so a member it just revised, or whose rows
pipette-duckdb did not answer, goes behind the others. When the pass reports a
member:
- A revision that did not advance (
filings_not_advanced): the campaign does not run the plan the pass filed. The pass cancels that plan, in the same run or the next, and logs aWARN. Only a revision whose campaign write has an unknown outcome waits for you. Each run then fails with theWARNqueue-maintenance: a campaign group filing whose head write is unknown still runs; cancel it, or confirm the campaign runs it, which names the campaign and the plan (filed). Read the campaign withcampaigns show. If it runs the plan, the write landed, and the next run goes on from it. Otherwise cancel the plan withplans cancel, but not right after the failed write, which can still land (§3.5, step 1). - A revision whose replaced plan still runs: the
WARNqueue-maintenance: a campaign group filing advanced, and the plan it replaced still runsfails one run. The campaign runs the new plan (filed), and the error names the plan that was not cancelled after "but plan". No later run cancels it, and while it runs, stopping the campaign's plan leaves the jobs both plans hold running. Cancel that replaced plan withplans cancel <replaced plan_id>, never the plan the campaign runs. - A campaign that waits for devices (
members_waiting_for_devices): devices have not claimed every job the policies' last revision of it queued, so nothing of it is read. TheINFOqueue-maintenance: the jobs a campaign group filing queued wait for devices, so its campaign is not readnames the campaign, the revision's plan (filed) and how many of its jobs wait (waiting), and the policy preview shows the same.plans status <plan_id> --explainshows why a job waits. The hold has no time limit, so a job no device can take holds the campaign until its deadline: mark it failed with the campaign's re-run route (fail), whose revision drops the job. - A member or group that does not read (
members_unreadable,members_skipped,groups_skipped): each is a failed item, and the next run reads it again. When pipette-duckdb fails or times out on a member's rows, theWARNpipette-duckdb did not answer the campaign's rows; it reads unreadablenames the campaign, the run reads no more rows, and that member goes behind the others. A store read that fails logs its error with the member or group. - A member state that does not decode: the
WARNqueue-maintenance: unreadable campaign group member state, skippednames the campaign and its row (key), and every run fails until the state reads. Take the campaign out of the group and add it back, on the group's page or withDELETE …/members/{campaign_id}andPOST …/members: the new row keeps no state, and the next run reads the campaign's plan as new. While a policy of the group other thanmodel_paramsis on, adding it back takesread_plans,cancel_plansandread_clients(the_any_grants for another owner's campaign), so have someone who holds them do both steps. - A plan that cannot be filed again (
refused_heads): the plan's document is gone, no longer matches the plan, or the server refuses it, such as for a deadline that has passed. The run that first meets it fails. Later runs of the same release log theWARNqueue-maintenance: the campaign group policies cannot file into this plan again; it is read again once it or the policies change, or after a new releasewith the reason, and do not fail. The first run of a new release tries the plan once more, and fails again if the plan is still refused. To have the policies act on the campaign again, file a revision with a document the server takes, such as one with a laterexpires_at. - A job that waits for rows (
jobs_waiting_for_rows): an answered job with no rows in pipette-duckdb yet, read again on every run. Its rows land once a scoring run has scored the result, every 5 minutes with the crons above, and pipette-duckdb has published them. An eval that failed every scoring pass is parked, and gets no rows until it is retried (storage.md §4): listsubmissions/score-queue/parked/, then move the body back toto_do/and delete itsfailures/entry. - Result rows that lack model params (
models_missing_params), while the group'smodel_paramspolicy is on: theINFOqueue-maintenance: result rows lack model paramsnames the campaign, thekeyan entry for the model would take, and therowsandjobsthat lack it. It is no failed item. Add the entry (model params); the pages read the table, so they show the counts at once. - Re-run results (
reruns_recorded,reruns_judged): a revision the campaign moves onto records each re-run it queued, one row per policy that re-runs it, and the run that reads a re-run's rows records its outcome, ordroppedfor one with neither rows nor a record that the campaign's plan no longer holds (re-run results). Each run reads each member's re-runs that wait, one database read, and of those its plan no longer holds, whether each has a record, one storage read each, and their rows, one pipette-duckdb read. A write that fails is a failed item. After theWARNqueue-maintenance: failed to record the re-runs a campaign group filing queued, which names the campaign and the plan (filed), rundb backfill-rerun-resultsfor the group (§3.6). After theWARNqueue-maintenance: failed to record how a campaign group member's re-runs answered, the run files nothing for the member and keeps none of its state, so the next run reads those re-runs again and records their outcome.
Clock synchronization. The expiry pass judges deadlines by the maintenance
host's clock, while serve's claim/reclaim gates use their own. A
maintenance clock ahead of a serve host's opens a window where serve still
hands a job out while queue-maintenance expires it, writing a synthetic
failure for a job a client is running. It also shortens the stranded-ingest
deadline, which compares the filing host's created_at. Keep all hosts
NTP-synced; a lagging maintenance clock is harmless, so single-host or
shared-clock deployments are inherently safe.
Log lines. Alarms and dashboards can match these INFO lines, each
carrying its figures as fields:
-
queue-maintenance: run finished, once per completed run: the run's and each phase's duration, the stored data it read (plans, probes, content keys, client records), the CPU time, the slowest plan, andfailed_itemsfor a run that skipped items and then exits non-zero. A run that stops on an error logs no summary.deadlines_restampedcounts the waiting jobs the run moved to the earliest horizon of their plans (plan-ingestion.md §7).bodies_readcounts full job bodies that the run fetches from the SQL queue. The count includes these reads:- expiration
- lease retirement
- corrupt bodies, which each run reads again until the row is repaired
Cache hits and reads of matcher projections do not count.
-
queue-maintenance: policy pass finished, once per run with a[database], also after queue passes that stopped on an error: thegroupswith a policy on and thegroups_skipped, theirmembers, themembers_readand themembers_not_reachedwithin the 60 s, themembers_unreadableandmembers_skipped, therevisionswith theirrerun_jobsandmarked_jobs, thereruns_recordedfor them and thereruns_judged, whose outcome the run recorded,jobs_waiting_for_rows,members_waiting_for_devices,filings_not_advanced, thecancel_markers_writtenon them,filings_stoppedby someone stopping the plan they revise,refused_heads,other_owner_members,jobs_left_for_good,models_missing_params, the keys the result rows it read lack a model params entry under,failed_items, the pass'sduration_ms, andrun_duration_msfrom the run's start, the queue passes included. A pass that cannot read the policies logs no line. Alert whenrun_duration_mspasses 5 minutes: such a run holds back the next one's lease recycling. -
queue-maintenance: result rows lack model params, once per key and member campaign while the group'smodel_paramspolicy is on: thecampaign_id, thekeyan entry would take, and therowsandjobsread this run that lack it. -
queue-maintenance: S3 and CPU work of a phase, before the summary, once per phase (queue_passes,reconcile_plans,plan_index_gc): HTTP attempts of both S3 clients by outcome (throttledcounts S3'sSlowDown), the peak in flight, the time waited, and the bodies decoded. -
queue-maintenance: refreshed a plan's progress snapshot, per plan whose jobs the pass probed or whose status changed, andqueue-maintenance: plan status changedwithfromandto. A plan that fails gets theWARNqueue-maintenance: failed to reconcile a plan.
S3 request attempts that reach the HTTP connector without an answer log
S3 request attempt got no response, with the client, method, path, host,
wait time and error. The SDK's per-attempt timeout can end an attempt before
that connector emits a log, so the request log does not count every slow
attempt. The object_store client waits 5 s to connect and 30 s for a
request. To fail a stalled maintenance read sooner, set
queue_maintenance_read_timeout_secs.
Each filing emits plan job promotion latency with its plan ID, completed
promotion count, failure count, p50, p95, p99 and maximum call duration in
milliseconds. The sample includes completed promotions before a filing error;
in-flight promotions dropped on that error have no duration.
The eligible index is maintained by queue-maintenance alone; serve
replicas only read it (queue.md). No operator
action is needed after clients approve, a permission change or a profile
update: the next run reflects it. The console counts each worker's backlog from
the index at each request (queue.md).
flag-reindex forces a client's rebuild when the automatic path did not
converge. When to use it, and the work gate it imposes, are
cli.md.
The audit log keeps two years of events, and only audit prune deletes them
(cli.md). Run audit prune daily from
one host:
# /etc/cron.d/pipette-mgmt-audit
30 3 * * * app /usr/local/bin/pipette-mgmt --config /etc/pipette-mgmt/config.toml audit prune 2>&1 | logger -t pipette-mgmt-audit-prune
If a run does not happen, the audit log keeps the old events until the next run deletes them.
A filing writes its plan and jobs before it writes the campaign. If it fails
while staging, the plan can remain creating with promoted jobs in the job
queue.
Queue maintenance leaves those jobs out of the eligible index while no final
plan also holds them (queue.md). A job already
leased to a device can continue running. Refile before you cancel or wait
for teardown if you want to preserve those jobs and results.
- Find the plan and campaign. Get their IDs from the error or the server
log's
request failedline (promoting job <job_id> for <plan_id>). A campaign-write error with an unknown outcome names both IDs. Read the campaign withpipette-mgmt campaigns show <campaign_id>. If its current plan or history includes<plan_id>, the filing completed. Stop here. A negative read immediately after a timed-out campaign write does not prove that the write will never land. Do not cancel or refile that plan until its write outcome is reconciled. The remaining steps cover a staging failure or a confirmed orphan. - Read the plan and record its work. Run
pipette-mgmt plans status <plan_id> --explain --leasesand note its status, available and leased job counts, and the clients under Running now. Save recorded results withpipette-mgmt plans results <plan_id> --json. Check how much time remains beforeplan_creating_max_age_secs(default 6 hours) tears down a plan stuck increating. - Refile the same document before that deadline. Use the same owner and
filing operation (
campaigns createfor a failed creation), not--dry-run. Keep duplicate merging enabled. A live job with the same content key joins the new plan; a completed cell can reuse its result for the same owner. A job already gone from both queue stages must be minted again. Do not cancel the old plan before the new filing succeeds: teardown would retire its exclusive jobs and end their leases. - Verify the new filing. Its final response must report success (
201over HTTP). Comparemerged_job_countwith the old plan's available plus leased jobs from step 2. The count includes finished runs reused, so it should be at least that live count when those jobs remain in the queue. Read the new campaign and plan to confirm they exist. If the filing fails, retry with the same document before the old plan's teardown deadline. - Let the old plan finish cleanup. A
creatingplan past its age limit is torn down by queue maintenance. It releases jobs that the new plan now holds and retires only jobs with no other member. No manual cancel is needed. If the orphan was already final, cancel it only after the new filing succeeds, then verify that its shared jobs remain with the new plan.
A failed retry can leave extra membership markers even if it wrote no new manifest. Check the new campaign and plan, not only its HTTP status. Keep the original plan uncancelled while you retry; a later maintenance pass can then release its joined jobs to the successful filing.
From the release that adds migration 0037, the policy pass records each
re-run it files (§3.1). The re-runs it filed
before have no row until db backfill-rerun-results records them
(cli.md). Run the
command once, after the deploy, not before: until then the older binary still
files re-runs and records none.
-
Run
db migratebefore the deploy, as for every migration (§5). -
Deploy the release.
-
With the deployed release and its config, name each group whose policies filed re-runs (
GET /api/v1/campaign-groupslists the groups):pipette-mgmt db backfill-rerun-results <group_id>...The command reads pipette-duckdb, so its config needs a
[duckdb]section; without one it exits naming it. It prints the filings it read and the rows it recorded. A run that fails can run again: a second run records only the rows still missing.
Whole warehouse. To re-score everything: move files from
submissions/processed/ back to submissions/incoming/, clear all
scorer-owned output directories (warehouse/results/ and
warehouse/eval_sample_results/), then run cron. On S3, use aws s3 mv
to move objects between the processed/ and incoming/ prefixes, and
delete the relevant warehouse and eval sample result objects.
One eval benchmark (e.g. after a scorer fix). Use
pipette-mgmt requeue-eval, then run the
scoring passes. It copies, so scope a repeat run with --submitted-before.
A campaign's evals. To re-score the evals of the plan a campaign runs,
file a recalculation
(POST /api/v1/campaigns/{id}/recalculate,
or the app's Recalculate buttons). It copies each named job's record
the same way, under a new job of a revision the campaign moves onto, so the
fresh scores read on the campaign's pages. Copies from either path name the job
they copy in rescored_from. They keep the time of the device run in
run_submitted_at. The next process-submissions and
score-eval score the copies.
The liquidai/pipette-mgmt image (multi-arch: linux/amd64, linux/arm64) ships
the pipette-mgmt binary. The default CMD is serve. Override with
process-submissions, score-eval, or clients <action> to run other
subcommands. The image runs as
non-root uid 10001 and reads its config from /etc/pipette-mgmt/config.toml
(override with PIPETTE_MGMT_CONFIG).
The only env vars consumed at runtime are PIPETTE_MGMT_CONFIG, RUST_LOG,
PIPETTE_MGMT_LOG_FORMAT, the AWS_* family (for an S3-compatible backend),
libpq's PG* family (for [database]), and GITHUB_TOKEN (for catalog import and catalog refresh; see §3.0.1).
HOSTNAME, SUDO_USER, USER and LOGNAME only label audit records and
storage locks. Everything else comes from the TOML config. Set
PIPETTE_MGMT_LOG_FORMAT=json where a collector reads the container's log; see
cli.md § Logging.
A deployment keeps its records (plans, campaigns, identities, vault records)
and the job queue on PostgreSQL through [database]
(design/sql-record-store.md); run db migrate
before a binary that needs a newer schema
(cli.md).
The examples below show [database] and the object-store sections, which keep
submissions, the warehouse, blobs and the dataset cache. Each example reads the
database connection from the libpq PG* environment variables.
config.toml:
evals_server_url = "http://evals:8080"
listen_addr = "0.0.0.0:3000"
authorities = ["mgmt.example"] # the hosts clients reach serve by (§5.6)
[storage]
backend = "local_fs"
data_dir = "/data"
[auth_storage]
backend = "local_fs"
data_dir = "/data"
[database]docker run -d --name pipette-mgmt \
-p 3000:3000 \
-v "$(pwd)/config.toml:/etc/pipette-mgmt/config.toml:ro" \
-v "$(pwd)/data:/data" \
liquidai/pipette-mgmt:<version>The /data mount must be writable by uid 10001 (the image's /data is
already mode 1777, so a fresh bind mount works as long as the host directory
itself permits writes from that uid, chmod 1777 ./data or
chown 10001:10001 ./data on the host).
config.toml:
evals_server_url = "http://evals:8080"
listen_addr = "0.0.0.0:3000"
authorities = ["mgmt.example"] # the hosts clients reach serve by (§5.6)
[storage]
backend = "s3"
bucket = "my-pipette-data"
prefix = "v1/"
region = "us-east-1"
[auth_storage]
backend = "s3"
bucket = "my-pipette-auth"
region = "us-east-1"
# The records and the job queue; the libpq PG* environment variables supply
# the connection.
[database]
# The eval corpora the scores service packs, which mgmt only reads (§3.0).
[eval_packages_storage]
backend = "s3"
bucket = "my-eval-datasets"
region = "us-east-1"
# Vault blobs and staged uploads, apart from [storage]. Copy vault-resources/
# here before adding this section, or the existing blobs read as missing.
[vault_storage]
backend = "s3"
bucket = "my-pipette-vault"
region = "us-east-1"docker run -d --name pipette-mgmt \
-p 3000:3000 \
-v "$(pwd)/config.toml:/etc/pipette-mgmt/config.toml:ro" \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_REGION=us-east-1 \
liquidai/pipette-mgmt:<version>On EC2/ECS/EKS, omit the access-key env vars and let the standard AWS
credential chain pick up the instance role / task role / IRSA token:
object_store resolves the same env vars and metadata endpoints as the AWS
SDK (AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, AWS_WEB_IDENTITY_TOKEN_FILE,
EC2 IMDSv2). Use separate buckets for [storage] and [auth_storage] so
benchmark/warehouse data and private client identities can have different IAM
policies. The memberships, device ownerships and campaign and vault owners that
decide access (storage.md §3.2) live in
PostgreSQL, and the database credentials are that authority.
[storage] and [auth_storage] are probed for conditional PUT, which the
storage mutate lock is built on. A backend that answers 200 to a write it should have refused refuses to
start the process. The locked batch commands (process-submissions,
score-eval, fix-*, requeue-eval) probe [storage] for both a conflicting
If-None-Match: * create and an If-Match compare-and-swap against a superseded
version, since the lock needs the first to acquire and the second to take over a
crashed holder's lease. serve probes [auth_storage] for the create alone.
AWS S3 honors both; this is aimed at S3-compatible endpoints, which vary. Two
operational consequences: leave AWS_CONDITIONAL_PUT unset (setting it to
disabled fails the probe), and give the credentials s3:DeleteObject so each
probe can clean up the key it wrote under probe/.
On AWS (or any other backend confirmed to honor conditional PUT), the probe can
be switched off with probe_backend_capabilities = false, which skips its writes
and leaves nothing under probe/. Keep it on for an S3-compatible endpoint: it is
the only thing standing between a non-conformant one and a silently dropped row.
With it on against a versioned bucket, add a lifecycle rule on
<prefix>/probe/ expiring noncurrent versions and expired delete markers.
DeleteObject there only writes a marker, so the probe's own objects accumulate,
a few per run of every locked command, unread by anything. See
storage.md §8.
vault-resources/ under [storage] holds the private artifacts mgmt serves
(storage.md §3.4). mgmt is the only writer: a client
uploads each file over PUT /api/v1/vault/blobs/{principal_id}/{sha256}, and mgmt
keeps it at vault-resources/{principal_id}/blobs/{sha256} only if its bytes
hash to that digest. Nothing else needs credentials for the prefix. These points
apply to an operator:
- Bytes pass through the service both ways. An upload streams through a
request handler into a multipart upload of 32 MiB parts, as a download
streams back out. Each upload buffers up to ~190 MiB of parts, so mgmt runs
at most four at once (
VAULT_UPLOADS_MAX) and answers429past that. - The bucket needs an incomplete-multipart-upload lifecycle rule on
vault-resources/(AbortIncompleteMultipartUpload, a day is plenty). A refused upload is aborted, but one cut off by a deploy or a crash is not, and its parts are billed and invisible to a listing until something aborts them. An import that a client disconnect stops also leaves its upload open, so the rule covers imports too. - The bucket needs a lifecycle rule on
vault-staging/with the same one-dayAbortIncompleteMultipartUpload, and also anExpirationof one day. An import of a public model streams each archive tovault-staging/{principal_id}/{uuid}. Then mgmt copies the archive tovault-resources/and deletes the staging object. A deploy or a crash between these steps leaves the staging object, and the rule deletes it. Give the credentialss3:DeleteObjectonvault-staging/. mgmt copies an archive over 5 GiB with a multipartUploadPartCopy. An S3-compatible backend must support a rangedUploadPartCopy. - Set
endpointandregionin[storage]. mgmt copies a staged archive over 5 GiB through anaws-sdk-s3client, and sends every other vault request throughobject_store. The two clients read the environment in different ways. For example,object_storereadsAWS_ENDPOINTand the SDK does not. The[storage]keys set both clients to the same bucket. - mgmt deletes a file only when no record names it: a resource delete
removes the files no remaining record of its owner names, and
pipette-mgmt vault pruneremoves older orphans. It learns a file went missing only when asked:pipette-mgmt vault showreports a missing or resized file, ingestion refuses a plan naming a record whose file is gone, and a device fetching one gets a404with avault_object_missingevent in the log. Uploading a missing file again restores every record naming it. A resized one must be removed from the bucket first, since an upload keeps whatever already sits at its digest.
R2 is S3-compatible. Set region = "auto" and point endpoint at the
account-scoped R2 URL. Generate an R2 API token (Account → R2 → Manage R2 API
Tokens) and pass it as the AWS keys.
config.toml:
evals_server_url = "http://evals:8080"
listen_addr = "0.0.0.0:3000"
authorities = ["mgmt.example"] # the hosts clients reach serve by (§5.6)
[storage]
backend = "s3"
bucket = "pipette-data"
prefix = "v1/"
region = "auto"
endpoint = "https://<account_id>.r2.cloudflarestorage.com"
[auth_storage]
backend = "s3"
bucket = "pipette-auth"
region = "auto"
endpoint = "https://<account_id>.r2.cloudflarestorage.com"
[database]docker run -d --name pipette-mgmt \
-p 3000:3000 \
-v "$(pwd)/config.toml:/etc/pipette-mgmt/config.toml:ro" \
-e AWS_ACCESS_KEY_ID="<r2_access_key_id>" \
-e AWS_SECRET_ACCESS_KEY="<r2_secret_access_key>" \
liquidai/pipette-mgmt:<version>R2 ignores region content but the AWS sigv4 signer requires some value;
"auto" is the conventional choice.
Point endpoint at the MinIO URL. HTTP endpoints are accepted because the
S3 client enables allow_http whenever endpoint is set in the TOML.
config.toml:
evals_server_url = "http://evals:8080"
listen_addr = "0.0.0.0:3000"
authorities = ["mgmt.example"] # the hosts clients reach serve by (§5.6)
[storage]
backend = "s3"
bucket = "pipette-data"
prefix = "v1/"
region = "us-east-1"
endpoint = "http://minio:9000"
[auth_storage]
backend = "s3"
bucket = "pipette-auth"
region = "us-east-1"
endpoint = "http://minio:9000"
[database]docker run -d --name pipette-mgmt \
--network minio_net \
-p 3000:3000 \
-v "$(pwd)/config.toml:/etc/pipette-mgmt/config.toml:ro" \
-e AWS_ACCESS_KEY_ID="<minio_access_key>" \
-e AWS_SECRET_ACCESS_KEY="<minio_secret_key>" \
-e AWS_REGION=us-east-1 \
liquidai/pipette-mgmt:<version>Both buckets must exist before startup (mc mb local/pipette-data,
mc mb local/pipette-auth). MinIO doesn't enforce a real region but the
sigv4 signer needs one. Match whatever MINIO_REGION is configured with
(defaults to us-east-1).
Override the default CMD to run a cron pass or an admin command against the
same image and config. In a containerized deployment the §3 crons run this way,
as scheduled jobs:
docker run --rm \
-v "$(pwd)/config.toml:/etc/pipette-mgmt/config.toml:ro" \
-e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_REGION \
liquidai/pipette-mgmt:<version> process-submissionsThe server speaks plaintext HTTP; TLS terminates in the proxy placed in front of
it. listen_addr defaults to 0.0.0.0:3000, so a container started from the
examples above is reachable on every interface it has.
The required shape is a TLS-terminating proxy in front of the listen port, with a network control that admits only that proxy. Both halves are load bearing. TLS on the public hop protects clients; the network control is what keeps the plaintext hop between the proxy and this process from being reachable directly, which would otherwise offer the same traffic unencrypted on a different port.
What rides on it:
POST /clients/registerwithgenerate_key: truereturns a freshly generated private key in the response body: the one exchange in the API that carries a long-lived credential. (The supported clients generate their own keypair and sendpublic_key, so this is a convenience path rather than the normal one, but it is reachable and unauthenticated.)- Registration accepts a pre-auth key as a bearer secret, and every authenticated request carries its signature headers in the clear.
The server cannot see what sits in front of it, so it cannot check any of this. What it does report is the moment a credential actually leaves it:
WARN client_id=ev1_a3f8... returned a server-generated private key in the registration response
Since the supported clients generate their own keypair, this names a caller taking the one path where a long-lived secret transits.
How much weight a single line carries depends on who can reach the endpoint.
Under require_client_preauth_key, registration takes a minted secret, so each
occurrence is worth reading on its own. Under the default open policy, any
anonymous caller can produce these at request rate. Read a burst as probing
rather than as many separate exposures, and note that each one also leaves a
durable pending client record behind it. POST /api/v1/users/register leaves the same
kind of residue on the other keyspace when it is asked for an account: a pending
user record and the claim on the address it asked with, one per distinct
address. require_user_preauth_key = true closes that path the way
require_client_preauth_key closes this one.
Each process admits at most 20 registrations without a pre-auth key per
minute, devices and accounts together, and answers the rest 429, logging
refused: too many registrations without a pre-auth key. A registration with a
pre-auth key, and a device retrying a registration already on record, are not
counted; an account request retried with the same keypair is. The count is per
process, so a deployment of several instances admits that many per instance.
Of those 20, one client address gets 1, an IPv6 one counted by its /64, since
a device or an account registers once; clients behind one NAT share it. The
address is the X-Forwarded-For entry trusted_proxy_hops from the right
(default 1, the entry one load balancer appends); 0 ignores the header and
takes the peer. A header with fewer entries names no address, and only the
ceiling of 20 holds. This reads right only when exactly that many proxies
stand in front and each appends the address it received from: the reference
load balancer does, as its deployment template configures it, but nothing
here checks that. Count every appending proxy, or the limit falls on one of
them and refuses every keyless registration behind it. A proxy that does not
append lets a caller name its own address. The 429 carries Retry-After,
one second past the end of the minute. Enroll more than one device a minute
from one site with a multi-use pre-auth key.
The ceiling slows the residue rather than stopping it: a caller holding the
window full still leaves up to 28,800 pending records a day on each instance.
A flood from 20 addresses or more still fills the shared budget, and then
refuses every keyless registration, genuine ones included, for as long as it
lasts. An
internet-facing deployment should set require_client_preauth_key = true and
require_user_preauth_key = true, which close both paths.
A reference deployment on AWS runs an Application Load Balancer holding the certificate, listening on 443 with 80 redirecting to it, forwarding to the container's port over plaintext HTTP inside the VPC. The tasks sit in a public subnet with public IPs, so the security group admitting the load balancer's security group on that port is the only thing keeping the plaintext port off the internet: widening that rule exposes it directly, with no second control behind it.
The proxy must forward the client's Host unchanged, and authorities must
name each host clients reach this deployment by (mgmt.example, or
mgmt.example:8443 off the default port). An RFC 9421 or device v3 signature
over any other host is refused
(authentication.md §2.4),
and with the list empty serve warns at startup and refuses every one, the
console's key sign-in included.
Ingestion imports a gated HuggingFace model into the vault when a model in a
plan document carries an auth_token (plan-ingestion.md §6.2). For
this import, mgmt makes outbound HTTPS requests to these hosts:
huggingface.co, or the host ofhf_import_endpoint, for the Hub API and each download request.cdn-lfs.huggingface.co, where a download of a file in Large File Storage (LFS) redirects.cas-bridge.xethub.hf.co, where a download from a repository on Xet storage redirects.
Allow all three in the egress rules of the task or the host. If mgmt cannot
reach them, an ingest that imports answers 502 and stages nothing. A plan
with no token makes no outbound request.
The import runs inside the request. A client disconnect stops the import and files no plan. Thus:
- A client that sends
Accept: application/x-ndjsonkeeps the connection open. mgmt then sends a line at intervals of 15 seconds or less (Campaigns: Response201 Created). This interval is shorter than the ALB default idle timeout of 60 seconds. - For a client that does not send the header, set the idle timeout of the
load balancer above the longest import that you expect, and
read_timeout_secsabove that (§2.2). Without the header, mgmt sends no bytes until the import completes. A retry downloads each directory model again, and each gguf file that the import did not store completely. - Keep the
AbortIncompleteMultipartUploadrule onvault-resources/(§5.2, "The vault prefix"). A stopped import leaves an open upload.
Each ingest that downloads a model holds one permit. Set the number with
hf_import_max_concurrent (default 2). Each import streams one file at a time,
and an S3 upload buffers about 190 MiB of parts. The import stores each file
under vault_blob_limit_bytes.
Each request to HuggingFace holds a request permit. Set the number with
hf_max_concurrent_requests (default 8). The number must be greater than
hf_import_max_concurrent. A request waits for a permit when every permit is
held. The process then logs hf request waited for a permit. The process logs
hf request rate-limited by hub when HuggingFace answers 429. The ingest
then answers 429.
An import logs hf import started, hf import stored record and
hf import reused record, with the owner and the HuggingFace reference. It
never logs the token. A request that asks for the stream logs
ingest answers as an ndjson stream. If the caller disconnects before the last
line, the request logs ingest stream closed by the caller; the ingest stopped.
Devices and accounts are administered with the clients and users commands
(cli.md), or over HTTP by an account that holds the grant
(Device administration,
Users: Account administration).
[auto_approve] approves a client at registration when its contact_email
matches a rule, saving the manual clients approve. It is not a security
control: the address is self-reported, so anyone can register under it. See
authentication.md §3.1.
A machine can ask for an account at POST /api/v1/users/register with no key
(POST /api/v1/users/register). The account waits
pending in users list, since nothing here sends mail, until users approve
admits it or users delete frees its address and keypair.
There is deliberately no auto-approve for users: an address is self-reported
here as much as on a client, and an operator account is the half of the
vocabulary that authors work. Set require_user_preauth_key = true to refuse
the keyless request outright, so every account is minted by an operator. The
one account created without an operator step is a Clerk sign-in's, where Clerk
has verified the address (§6.3).
With a [clerk] section, the console app's sign-in page (/app/) offers
Clerk beside the key sign-in, so people sign in with whatever the Clerk
instance offers, such as a Google account. Devices, scripts and every other
caller of the JSON API are unaffected: they still sign every request
(Users: Calling the API yourself).
On the Clerk side:
-
Create a JWT template with these claims. Its name and its
audare both thejwt_templateset below,pipette-mgmtby default. The server refuses a token from another template, and one without a verified address:{ "aud": "pipette-mgmt", "email": "{{user.primary_email_address}}", "email_verified": "{{user.email_verified}}" } -
Serve the console from an origin the instance serves: its domain, a subdomain of it, or a satellite domain. A reader with no Clerk session is sent to the instance's sign-in page, which has to be allowed to send them back to the page they left under
/app/.
On this side:
[clerk]
publishable_key = "pk_live_…"
public_key = """
-----BEGIN PUBLIC KEY-----
…
-----END PUBLIC KEY-----
"""
authorized_parties = ["https://console.example.test"]
jwt_template = "pipette-mgmt"
allowed_domains = ["example.test"]
[[clerk.provision]]
domain = "example.test"
org = "acme"publishable_keyandpublic_key(the PEM public key) are both on the Clerk dashboard's API keys page. Tokens are verified against the key offline, so the server holds no Clerk secret; update it if Clerk rotates the key.authorized_partieslists the origins the console is served from, as a browser sends them inOrigin. The sign-in page offers Clerk only at these hosts; at any other name the server answers to, it offers keys alone.jwt_templatenames the template from step 1,pipette-mgmtwhen unset. The sign-in page asks Clerk for it, and a token'saudmust be that name, so a token another app on a shared instance asks for signs nobody in here. A name Clerk has no template for fails every Clerk sign-in, so create the template first.allowed_domains, when set, limits Clerk sign-in to addresses at those domains, matched exactly:example.testadmits neithersub.example.testnorevilexample.test. Any other address is refused403before its account is looked up, so an account an operator made for it still signs in only with its key. Set it whenever the Clerk instance is shared with apps that admit outside users, since Clerk then verifies addresses this deployment should not trust. A provision rule must name an allowed domain, or the server does not start. Without the key, any verified address may sign in.- A
[[clerk.provision]]rule gives an address atdomainthat no account holds an active account on its first sign-in: a member oforg, holding theorg_membergroup, asusers create --org-memberandusers set-orgwould. The organization must exist and not be disabled. Without a rule, an address signs in only to an account an operator created for it. A keyless account request at the domain is refused, so nobody can claim an address ahead of its owner. A pending request from before the rule is such a claim:users deleteit rather than approving it, and its owner's next sign-in creates the account.
A rule trusts Clerk with the domain: whoever Clerk verifies an address at it for
gets an account. Offboard with users disable, not delete. A disabled
account's sign-ins and keys are all refused, while a deleted one at a
provisioned domain is created afresh on the next sign-in.
When a Clerk sign-in fails, the log shows why. A checking a Clerk token's claims line at info carries what the template rendered (aud, azp, iss,
email_verified, the address's domain and the rest), and a refused a Clerk sign-in line at warn names the check that failed. Only the refusal appears
for a token whose signature does not verify, which usually means public_key
is not this instance's.