Benchmark runners for evaluating large language model (LLM) inference engines on edge devices, from host-native CLIs to native iOS and Android apps. The CLIs are Rust; the iOS app is native Swift and the Android app defaults to Kotlin over llama.cpp. Remote runs can submit results to the pipette management server; local runs stay local.
The clients are the benchmark runners you build and run.
Command-line benchmark runners that run on the host machine. They share one
operating workflow: init → register → install a runtime → run → sync, plus an
optional planner claim loop (pipette worker) that pulls jobs from the
management server.
| CLI | Role | Build |
|---|---|---|
pipette |
runs benchmarks on the host; the engines it drives are listed in docs/README.md | cargo build --release -p pipette-cli |
pipette-ctl |
drives pipette across remote devices, or files the same matrix with the management server for workers to claim |
cargo build --release -p pipette-ctl |
Mobile clients run benchmarks on-device with engines compiled into the app. The linked engines differ per platform; see docs/README.md. Each app has its own build pipeline:
| App | Location | Build instructions |
|---|---|---|
| iOS | ios/Pipette/ |
docs/pipette-ios/build.md |
| Android | android/Pipette/ |
docs/pipette-android/build.md |
iPhones in a device lab run jobs through pipette-ios-proxy; see
On the farm.
crates/ is not a client. It holds the shared Rust workspace for host CLIs,
runtime libraries, artifact stores, management client, execution helpers, and
the Android rustEngine core. Plan types come from pipette-mgmt
crates/pipette-plan-types by git (see CONTRIBUTING.md).
cargo build --release -p pipette-cli # produces target/release/pipette
export PATH="$PWD/target/release:$PATH" # or copy it somewhere on PATHThen follow the usage guide (init → register → install a runtime → run → sync). Mobile app builds are linked in the table above.
Full index and documentation convention: docs/README.md.
This is an Apache-2.0 licensed open source project. See CONTRIBUTING.md, CODE_OF_CONDUCT.md, and SECURITY.md.
Copyright 2026 Liquid AI, Inc.
Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
The full license text is in LICENSE, with attribution in NOTICE and THIRD_PARTY_NOTICES.md.