de is a command-line tool for creating, querying, and inspecting RDF
data in HDT (Header, Dictionary, Triples) format.
It is intended for workflows where RDF data needs compact storage and SPARQL querying over both RDF and HDT inputs.
Run directly from a local clone (no install required):
git clone https://github.com/DeciSym/de.git
cd de
cargo run -- --helpInstall the CLI from crates.io:
cargo install deInstall with server command enabled:
cargo install --features server deInstall the CLI from a local clone:
cargo install --path .Docker image:
docker run --rm decisym/de:latest --helpThe example below corresponds to the SPARQL 1.1 Query Recommendation:
- §2.1 Triple Patterns: https://www.w3.org/TR/sparql11-query/#basicpatterns
From the repository root, create the example input data (simple.nt):
<http://example.org/book/book1> <http://purl.org/dc/elements/1.1/title> "SPARQL Tutorial" .
Create the query (simple.rq):
SELECT ?title
WHERE
{
<http://example.org/book/book1> <http://purl.org/dc/elements/1.1/title> ?title .
}Run directly against RDF:
cargo run -- query --data simple.nt --sparql simple.rq --output csvOutput:
title
SPARQL TutorialConvert to HDT and run the same query:
cargo run -- create --output-name simple.hdt --data simple.nt
cargo run -- query --data simple.hdt --sparql simple.rq --output csvOutput:
title
SPARQL TutorialUse CLI help (and installed man pages, if available in your environment) as the canonical command reference:
cargo run -- --help
cargo run -- <command> --helpIf you installed the CLI:
de --help
de <command> --helpExamples:
cargo run -- create --help
cargo run -- query --help
cargo run -- view --helpRun core checks:
cargo fmt --check
cargo clippy --all-features --all-targets -- -D warnings
cargo test --all-featuresRun W3C RDF/SPARQL integration tests:
cargo test --all-features --test w3c-sparqlThe trainmarks criterion suite covers create and query over the synthetic
e-commerce graph from trainmarks,
checked out as a submodule at benches/trainmarks. It exists to catch
performance regressions at a realistic dataset size, not to compare de
against other engines.
It measures four things: building an HDT from N-Triples, building one from
Turtle (the only path that runs the RDF parser), the five queries against a
prebuilt HDT, and one query given a Turtle file directly, which de converts
to a temporary package before evaluating.
make bench first checks out the submodule and generates the fixtures the
suite reads:
make benchThe dataset scale is BENCH_SCALE — medium (~100K triples), large (~1M,
the default) or xlarge (~10M):
BENCH_SCALE=medium make benchmake bench passes the scale through to the suite as DE_BENCH_SCALE, which
is also what to set when driving cargo bench directly. The fixtures and the
benchmark must agree on it, since the scale is part of every benchmark id
(trainmarks_query/large/q3_join_3_entities) and criterion compares each run
against the stored baseline for that id:
make bench-init # once, to lay down the fixtures
DE_BENCH_SCALE=large cargo bench --bench trainmarks
DE_BENCH_SCALE=large cargo bench --bench trainmarks -- q3_join_3_entitiesWithout the fixtures the suite prints how to get them and measures nothing, so
cargo bench still works on a fresh clone.
The queries are trainmarks' own q1–q5, shared verbatim with the other
engines in that report. q6_delete_insert is omitted: it is a SPARQL Update,
and HDT packages are immutable.
The watdiv suite runs the WatDiv workload Comunica uses for its own HDT
engine: a published dataset.hdt (~1M triples) and twenty query templates with
five instances each. Each benchmark answers a template's five instances the way
de query would, so it includes opening the package and parsing the queries.
That is noise for the complex templates, where join order decides the time, but
most of the cost of the lightest ones. make bench-watdiv downloads the
fixtures, pinned to one commit of comunica/comunica-performance-assets and
checked by SHA-256, and runs the suite:
make bench-watdiv # scale 10, ~1M triples
WATDIV_SCALE=100 make bench-watdiv # ~10M triplesCI runs both suites, trainmarks at large and WatDiv at scale 10, on pushes to
main and on pull requests that change code. It runs each twice on one
runner, on the change and on its base commit, and compares the two in the run's
summary. It flags changes over 15%, and fails when a benchmark is more than 25%
slower than on the base. Each run's results are also kept as the
bench-trainmarks and bench-watdiv artifacts, but runners differ in CPU from
run to run, so compare those only with others from the same CPU model. To
compare two runs on your own machine, save each as a criterion baseline:
DE_BENCH_SCALE=large cargo bench --bench trainmarks -- --save-baseline main # on main
DE_BENCH_SCALE=large cargo bench --bench trainmarks -- --save-baseline mine # on your branch
scripts/bench-compare.py compare target/criterion:main target/criterion:minemake bench-memory measures the heap memory a release build of de uses on
the same fixtures, under heaptrack
(apt install heaptrack). Each workload runs as a de process of its own,
because peak heap is a high-water mark for the whole process:
createfrom the trainmarks N-Triples, and from the Turtle;- each trainmarks query, and
q1_countstraight from the Turtle; - each WatDiv template's five instances.
scripts/bench-memory.py records what heaptrack reports for each workload:
peak heap, calls to allocation functions, temporary allocations, memory leaked
and peak RSS (which includes heaptrack's own overhead).
make bench-memory # writes target/bench-memory/
scripts/bench-memory.py compare target/bench-memory/trainmarks /tmp/mem-main/trainmarks
heaptrack --analyze target/bench-memory/trainmarks/create_hdt_from_nt.gz # where the memory goesCI measures memory alongside the timings, on the change and on its base, and compares the two in the run's summary. It flags a change over 5% in peak heap or in allocation calls, and fails when a workload's peak heap grows more than 15%. A change counts only if it is also over 256 KB of heap or 1,000 allocations, because a WatDiv query peaks at about 0.5 MB, so 5% of that is just a small buffer. Two runs of the same build on one machine agree to within 0.1% on allocations, and to within 2% on peak heap, where only the parallel HDT builds vary. So these thresholds are much tighter than the timing ones.
Allocation counts are the same on any machine. Peak heap isn't. On a GitHub
runner it came out up to 11% lower than on a developer machine, so compare peak
heap only between runs on the same machine, as CI does. Each run keeps the
change's recordings as the memory-trainmarks and
memory-watdiv artifacts. heaptrack_print -f NEW -d OLD shows what changed
between two of them.