Skip to content
DeciSymPublic

About

CLI tool for creating, querying and inspecting RDF data in HDT format. Enables efficient semantic workflows using SPARQL and supports a variety of RDF and result serialization formats.

Topics

Resources

Stars

6 stars

Watchers

3 watching

Forks

Repository files navigation

Lint Build Security Audit License

DeciSym Engine (de)

de is a command-line tool for creating, querying, and inspecting RDF data in HDT (Header, Dictionary, Triples) format.

It is intended for workflows where RDF data needs compact storage and SPARQL querying over both RDF and HDT inputs.

Installation

Run directly from a local clone (no install required):

git clone https://github.com/DeciSym/de.git
cd de
cargo run -- --help

Install the CLI from crates.io:

cargo install de

Install with server command enabled:

cargo install --features server de

Install the CLI from a local clone:

cargo install --path .

Docker image:

docker run --rm decisym/de:latest --help

Example

The example below corresponds to the SPARQL 1.1 Query Recommendation:

From the repository root, create the example input data (simple.nt):

<http://example.org/book/book1> <http://purl.org/dc/elements/1.1/title> "SPARQL Tutorial" .

Create the query (simple.rq):

SELECT ?title
WHERE
{
  <http://example.org/book/book1> <http://purl.org/dc/elements/1.1/title> ?title .
}

Run directly against RDF:

cargo run -- query --data simple.nt --sparql simple.rq --output csv

Output:

title
SPARQL Tutorial

Convert to HDT and run the same query:

cargo run -- create --output-name simple.hdt --data simple.nt
cargo run -- query --data simple.hdt --sparql simple.rq --output csv

Output:

title
SPARQL Tutorial

Command Reference

Use CLI help (and installed man pages, if available in your environment) as the canonical command reference:

cargo run -- --help
cargo run -- <command> --help

If you installed the CLI:

de --help
de <command> --help

Examples:

cargo run -- create --help
cargo run -- query --help
cargo run -- view --help

Development

Run core checks:

cargo fmt --check
cargo clippy --all-features --all-targets -- -D warnings
cargo test --all-features

Run W3C RDF/SPARQL integration tests:

cargo test --all-features --test w3c-sparql

Benchmarks

The trainmarks criterion suite covers create and query over the synthetic e-commerce graph from trainmarks, checked out as a submodule at benches/trainmarks. It exists to catch performance regressions at a realistic dataset size, not to compare de against other engines.

It measures four things: building an HDT from N-Triples, building one from Turtle (the only path that runs the RDF parser), the five queries against a prebuilt HDT, and one query given a Turtle file directly, which de converts to a temporary package before evaluating.

make bench first checks out the submodule and generates the fixtures the suite reads:

make bench

The dataset scale is BENCH_SCALE — medium (~100K triples), large (~1M, the default) or xlarge (~10M):

BENCH_SCALE=medium make bench

make bench passes the scale through to the suite as DE_BENCH_SCALE, which is also what to set when driving cargo bench directly. The fixtures and the benchmark must agree on it, since the scale is part of every benchmark id (trainmarks_query/large/q3_join_3_entities) and criterion compares each run against the stored baseline for that id:

make bench-init                          # once, to lay down the fixtures
DE_BENCH_SCALE=large cargo bench --bench trainmarks
DE_BENCH_SCALE=large cargo bench --bench trainmarks -- q3_join_3_entities

Without the fixtures the suite prints how to get them and measures nothing, so cargo bench still works on a fresh clone.

The queries are trainmarks' own q1–q5, shared verbatim with the other engines in that report. q6_delete_insert is omitted: it is a SPARQL Update, and HDT packages are immutable.

The watdiv suite runs the WatDiv workload Comunica uses for its own HDT engine: a published dataset.hdt (~1M triples) and twenty query templates with five instances each. Each benchmark answers a template's five instances the way de query would, so it includes opening the package and parsing the queries. That is noise for the complex templates, where join order decides the time, but most of the cost of the lightest ones. make bench-watdiv downloads the fixtures, pinned to one commit of comunica/comunica-performance-assets and checked by SHA-256, and runs the suite:

make bench-watdiv                       # scale 10, ~1M triples
WATDIV_SCALE=100 make bench-watdiv      # ~10M triples

CI runs both suites, trainmarks at large and WatDiv at scale 10, on pushes to main and on pull requests that change code. It runs each twice on one runner, on the change and on its base commit, and compares the two in the run's summary. It flags changes over 15%, and fails when a benchmark is more than 25% slower than on the base. Each run's results are also kept as the bench-trainmarks and bench-watdiv artifacts, but runners differ in CPU from run to run, so compare those only with others from the same CPU model. To compare two runs on your own machine, save each as a criterion baseline:

DE_BENCH_SCALE=large cargo bench --bench trainmarks -- --save-baseline main   # on main
DE_BENCH_SCALE=large cargo bench --bench trainmarks -- --save-baseline mine   # on your branch
scripts/bench-compare.py compare target/criterion:main target/criterion:mine

Memory

make bench-memory measures the heap memory a release build of de uses on the same fixtures, under heaptrack (apt install heaptrack). Each workload runs as a de process of its own, because peak heap is a high-water mark for the whole process:

  • create from the trainmarks N-Triples, and from the Turtle;
  • each trainmarks query, and q1_count straight from the Turtle;
  • each WatDiv template's five instances.

scripts/bench-memory.py records what heaptrack reports for each workload: peak heap, calls to allocation functions, temporary allocations, memory leaked and peak RSS (which includes heaptrack's own overhead).

make bench-memory                                  # writes target/bench-memory/
scripts/bench-memory.py compare target/bench-memory/trainmarks /tmp/mem-main/trainmarks
heaptrack --analyze target/bench-memory/trainmarks/create_hdt_from_nt.gz   # where the memory goes

CI measures memory alongside the timings, on the change and on its base, and compares the two in the run's summary. It flags a change over 5% in peak heap or in allocation calls, and fails when a workload's peak heap grows more than 15%. A change counts only if it is also over 256 KB of heap or 1,000 allocations, because a WatDiv query peaks at about 0.5 MB, so 5% of that is just a small buffer. Two runs of the same build on one machine agree to within 0.1% on allocations, and to within 2% on peak heap, where only the parallel HDT builds vary. So these thresholds are much tighter than the timing ones.

Allocation counts are the same on any machine. Peak heap isn't. On a GitHub runner it came out up to 11% lower than on a developer machine, so compare peak heap only between runs on the same machine, as CI does. Each run keeps the change's recordings as the memory-trainmarks and memory-watdiv artifacts. heaptrack_print -f NEW -d OLD shows what changed between two of them.

About

CLI tool for creating, querying and inspecting RDF data in HDT format. Enables efficient semantic workflows using SPARQL and supports a variety of RDF and result serialization formats.

Topics

Resources

Stars

6 stars

Watchers

3 watching

Forks

Releases

Contributors

Languages