feat(qs): automate OSS string database builds by vee1e · Pull Request #1327 · mandiant/flare-floss

vee1e · 2026-07-04T15:25:16Z

Summary

Adds an automated pipeline for rebuilding the FLOSS Quantumstrand OSS string databases. The pipeline is implemented by a new Python script, floss/qs/db/data/oss/build_oss_db.py, and a new GitHub Actions workflow, .github/workflows/build-oss-db.yml, that runs the script on a bi-weekly cron (the 1st and 15th of each month at 00:00 UTC) and on manual dispatch.

What the script does

The script automates the existing vcpkg plus jh technique described in floss/qs/db/data/oss/readme.md:

Install static libraries via vcpkg for a chosen triplet.
Extract string features and function names from each .lib using jh.
Convert the jh output to JSONL and compress with gzip.

Per library, it produces one .jsonl.gz file in the output directory, matching the schema consumed by floss.qs.db.oss.OpenSourceStringDatabase.

Two correctness fixes are baked into the converter:

Within-library dedup. One metadata object per unique string in a library, matching the loader's metadata_by_string semantics.
Cross-library dedup. A string that appears in two or more libraries is not a unique identifier for any of them, so it is removed from every database. Without this, the per-string lookup in floss/qs/main.py:query_library_string_database would tag the same string as #<lib1>, #<lib2>, and so on, producing false-positive library attributions at query time.

The script also merges into any .jsonl.gz already present in the output directory rather than replacing them. New entries win on string collisions, which is the right behavior when a library version changes, and libraries that are not in the current --libraries list are still re-checked against the cross-library dedup and rewritten if the shared string set changes. This makes it safe to run the script locally with --libraries <one-lib> without losing the other databases.

What the workflow does

.github/workflows/build-oss-db.yml:

Runs on a bi-weekly cron (1st and 15th of each month at 00:00 UTC) and on workflow_dispatch.
Checks out flare-floss and lancelot, sets up Python 3.12 and Rust stable.
Caches Cargo and vcpkg.
Builds jh from lancelot (cargo build --release -p lancelot-bin).
Runs the build script with --continue-on-error so a single failed library does not block the others.
Opens a PR against quantumstrand with the regenerated .jsonl.gz files if anything changed.

The auto-update PR step is what makes this end-to-end. The script plus workflow is closed-loop: any change in upstream vcpkg package versions surfaces as a database diff, which becomes a small PR with metrics.

Todo

Errors in pipeline

ncurses, s2n and liboqs are only available on windows, i'm considering removing this one because its doubtful it'll appear in windows samples
clBLAS is hitting a rust assertion error
LFS isn't being detected at all, they're reading the pointers as text lmeow

Code review

dependency handling in feat(qs): automate OSS string database builds #1327 (comment)
caching in feat(qs): automate OSS string database builds #1327 (comment)
documenting and discussing tradeoffs of chosen triplet/compilers in feat(qs): automate OSS string database builds #1327 (comment)
move the script to scripts/ and add copyright

gemini-code-assist

Code Review

This pull request introduces build_oss_db.py to automate building OSS string databases from vcpkg static libraries using the jh tool, along with a default configuration file libraries.json and updated documentation in readme.md. The review feedback identifies critical bugs where the script attempts to parse the CSV output of the jh tool as JSONL in both Converter.parse and count_jsonl_rows. Additionally, the reviewer recommends skipping database writes for libraries that fail to build to prevent data loss, and checking if self.info_dir exists before calling iterdir() to avoid a potential crash.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Adds floss/qs/db/data/oss/build_oss_db.py that automates the vcpkg + jh pipeline for building OSS string databases: 1. install static libraries via vcpkg 2. extract string features (and function names) via jh 3. convert to JSONL and compress with gzip The converter deduplicates within each library (one metadata object per unique string) and emits each library as its own .jsonl.gz file, matching the schema expected by floss.qs.db.oss.OpenSourceStringDatabase. The script accepts a JSON config via --config or env vars and supports selecting individual libraries, controlling output directory, and continuing past per-library build failures.

Adds .github/workflows/build-oss-db.yml that runs the new build script on a bi-weekly cron (the 1st and 15th of each month at 00:00 UTC) and on manual dispatch. After each run, the workflow opens a PR against quantumstrand with the regenerated .jsonl.gz files if any of them changed. The workflow caches Cargo and vcpkg to keep build times reasonable, builds jh from the lancelot source, and runs the build script with --continue-on-error so a single failed library does not block the others. lancelot is pinned to williballenthin/lancelot HEAD.

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

williballenthin

i think every 2 weeks is a reasonable starting point. if this is too noisy, we can reduce the frequency.

ultimately, i like that this demonstrates how to rebuild the databases, rather than having the databases be some magic artifact that can't be reproduced. i'm not convinced they need to be rebuilt so often, but i don't think it hurts.

williballenthin · 2026-07-04T18:31:46Z

+    "boost-test",
+    "wolfssl",
+    "boost-wave",
+    "curl",


isn't zlib a dep of curl, so does that mean the zlib strings get filter as duplicate, rather than being assigned to zlib?

how do you handle this dependency issue?

vcpkg builds curl and zlib as separate static .lib files so jh only sees zlib's symbols when processing zlib.lib directly. When processing libcurl.lib, it only extracts strings from curl's own compiled code. it doesn't re-extract zlib's symbols just because curl links against it.

you can also see a very minimal string overlap in my local fork: vee1e#9

367: ...zlib.jsonl.gz (149 entries, 8 removed as shared with other libraries)

Missing these would be bad

agree that when the dependency is via linking, then things may work out. but in some cases the dependency may be vendored, ie, source code copied into the project. in this case, i think the strings will be detected as duplicates.

to be clear, i believe this is a hard problem to handle generically. though maybe it can also be solved by a little research and hacks/configs/shorcuts.

instead of removing duplicates altogether should we be upfront to the user and tell them exactly how commonly a string was found? maybe if its truly unique we can indicate it as a "pure" indicator tag, and with every subsequent instance of the string being found we can somehow show the library tag (#zlib etc) as "less" of a true oracle for its source.

Keeping duplicates around may be the safer bet for now until we see more of this in practice or do more research into the overlaps.

williballenthin · 2026-07-04T18:33:14Z

+          path: |
+            C:\vcpkg\installed
+            C:\vcpkg\downloads
+          key: ${{ runner.os }}-vcpkg-oss-db-${{ hashFiles('flare-floss/floss/qs/db/data/oss/build_oss_db.py') }}


seems like this keys the hash on the generation script, not the contents of the libraries. am i reading that right?

i think what we'd want to use is a vcpkg lock file, if that exists

seems like this keys the hash on the generation script, not the contents of the libraries. am i reading that right?

yup, not perfect ofc, i was thinking of using the libraries.json config file as the hash so it updates whenever we add/remove a lib to build

i think what we'd want to use is a vcpkg lock file, if that exists

the requirement for that is fulfilled by the libraries.json file already imo

I think we can drop C:\vcpkg\installed from the cache because vcpkg names files in downloads/ by their content hash (e.g. openssl-3.4.1.tar.gz with a sha512 in the filename). When vcpkg installs a package it checks downloads/ first, then if the tarball is already there and the hash matches, it skips the network fetch and uses it directly.

ah ok so it's not a cache of the built artifacts, but of the source downloads.

i still think this cache key isn't right, and probably confusing as is (maybe just a comment is needed).

i still think this cache key isn't right, and probably confusing as is (maybe just a comment is needed).

yeah we can add comments on why the decision was made. maybe we can forgo the cache entirely?

i think that's fine. if we end up spending obviously a lot of computer resources maybe we can optimize some, but let's not do that prematurely.

williballenthin · 2026-07-04T18:34:31Z

@@ -0,0 +1,73 @@
+{
+  "triplet": "x64-windows-static",
+  "compiler": "msvc143",


how is this chosen and when should this be updated?

what effect, if any, does it have on the compilation?

how is this chosen and when should this be updated?

taken directly from the existing qs documentation

flare-floss/floss/qs/db/data/oss/readme.md

Lines 48 to 52 in 0fbe890

Use the triplet `x64-windows-static` to build static archives (.lib files that are AR archives containing COFF object files):

```console

PS > C:\vcpkg\vcpkg.exe install --triplet x64-windows-static zlib

```

what effect, if any, does it have on the compilation?

i tried the builds locally on macosx-arm64. no significant differences in string counts was observed

What builds did you try?

i probably picked 143 at the time because it was probably the latest version or something. i assume there's not a major difference between versions, but we should document our expectations and assumptions. that's why i raise it here.

What builds did you try?

Using raw strings on both I got this:

Library macOS MinGW In Both mac Only Win Only

binn 212 401 3 209 398

brotli 2,768 3,281 1,087 1,681 2,194

bzip2 571 632 63 508 569

curl 5,996 6,869 1,972 4,024 4,897

flatcc 751 1,085 87 664 998

fmt 719 1,246 59 660 1,187

freetype 4,938 5,985 954 3,984 5,031

libarchive 5,328 9,169 1,446 3,882 7,723

libflac 2,349 3,775 382 1,967 3,393

libjpeg-turbo 4,000 3,786 320 3,680 3,466

liblzma 1,182 1,653 159 1,023 1,494

libogg 226 269 81 145 188

libpng 1,909 2,111 436 1,473 1,675

libtheora 1,171 1,690 39 1,132 1,651

libvorbis 3,273 4,796 620 2,653 4,176

libwebp 3,185 4,803 117 3,068 4,686

libxml2 8,506 9,200 3,874 4,632 5,326

lz4 540 1,290 28 512 1,262

lzo 604 811 13 591 798

mbedtls 6,931 7,592 2,509 4,422 5,083

openal-soft 7,862 17,976 2,093 5,769 15,883

openssl 53,464 43,900 13,148 40,316 30,752

opus 2,729 3,224 581 2,148 2,643

opusfile 284 565 14 270 551

parson 245 343 5 240 338

physfs 1,202 1,588 76 1,126 1,512

sqlite3 8,335 7,518 1,221 7,114 6,297

tre 330 595 34 296 561

zlib 674 828 52 622 776

zstd 2,157 5,250 133 2,024 5,117

=== binn macOS-only (arm64 machine code) === '\t@9j' ← random bytes, not text ' #[z9' ← ditto '@9+\t@9,' ← ditto 'K)@)' ← ditto 'R)#@' ← ditto '_AddValue' ← symbol name (mangled C function) '_AdvanceDataPos' ← symbol name '__.SYMDEF SORTED'← macOS BSD ar archive metadata '#1/12 1782384003 501 20 644 57180 `' ← macOS BSD ar archive metadata === binn mingw-only (x86_64 machine code) === '\tw}A' ← random bytes '([^_]A\A]A^A_' ← fragments of x86_64 instructions '8[^_]A\A]A^A_' ← ditto 'AWAVAUATUWVSH' ← x86_64 function prologue bytes 'AVWVSH' ← ditto '/ 1783197601 0 0 0 0 2520 `' ← GNU ar archive metadata

So there is a difference in strings but the total count remains similar. I didn't look at which strings overlapped before, hence the slightly wrong conclusion that there was no difference. But most of the lack of overlap can be attributed to:

Random printable byte runs in compiled code that look like strings to strings -n 4. Different ISAs produce different random byte runs, so the noise never overlaps.

Section names differ. Mach-O has __TEXT, __cstring etc. while COFF has .rdata, .pdata, .xdata

Symbol name prefixes are different too. Mach-O adds a leading _ (_AddValue), COFF doesn't (AddValue)

It could be worth building another db set for different platforms if we want to cover all possible executables, but it'll increase build times and cost. I think its a fair point to debate imo

The comment here was on the msvc version :) interesting additional information though. With our PE focus we should stick to Windows for now.

williballenthin · 2026-07-04T18:35:39Z

+
+The `build_oss_db.py` script automates the steps above and additionally
+removes strings that appear in two or more libraries from every database.
+A shared string is not a unique identifier for any specific library, so


as noted above: unless one library links or vendors another library.

williballenthin · 2026-07-04T18:36:52Z

 $  gzip -c zlib.jsonl > zlib.jsonl.gz
 ```
+
+The `build_oss_db.py` script automates the steps above and additionally


i like that the logic lives in this script, and that the GH workflow is minimal. this reduces our dependence on GH, which i believe is prudent.

vee1e · 2026-07-04T19:34:52Z

i think every 2 weeks is a reasonable starting point. if this is too noisy, we can reduce the frequency.

me and @mr-tz had a few back and forths with this. I ran a few tests on vcpkg versions and found out that ~biweekly is the best to capture around 5-8 update events because libraries update on different schedules.

vcpkg update times (in days) that we found:

Mean of per-library averages (78 libs): 305 days (~10 months)
Median of per-library averages: 181 days (~6 months)
Pooled across all individual gaps (814 gaps from 87 libs): 157 days (~5.2 months) — this weights by how often each library actually releases

but I still do agree, i think we can get a better estimate after letting this cook for a while, so the frequency is volatile for now. we should also think about how many update events we should/need to capture — weekly would give us 3-4, monthly 10-15 from my tests.

vee1e · 2026-07-04T19:42:46Z

Ran the workflow on my own fork — 118ish minutes for the whole thing to complete

mr-tz

Nice work. Further discussion in the threads Willi opened is needed.

mr-tz · 2026-07-04T19:55:35Z

+
+# Fallback library list when neither --libraries nor libraries.json is
+# provided. The actual production list lives in libraries.json.
+DEFAULT_LIBRARIES: List[str] = [


Are these necessary or helpful here?

yeah i don't think so. removing it.

williballenthin · 2026-07-04T20:33:07Z

vcpkg update times (in days) that we found

imho a more interesting metric is how frequently does an important string get added to the db. project releases don't necessarily imply meaningful additions to the databases.

likewise, there's probably some study that could be done about strings getting removed.

and then, a scenario where strings are added and then removed without that middle release getting captured (such as a zlib version string or something). would a more comprehensive approach therefore be to build every release? what are the pros/cons there?

we don't necessarily need to do all this research, just raising some ideas.

vee1e · 2026-07-04T21:19:56Z

imho a more interesting metric is how frequently does an important string get added to the db. project releases don't necessarily imply meaningful additions to the databases.

likewise, there's probably some study that could be done about strings getting removed.

yeah i did some preliminary research on this for a few libs. results are here:

Library	Transition	+Strings	-Strings	+Funcs	-Funcs
zlib	1.3.1 → 1.3.2	5	3	10	1
expat	2.4.8 → 2.6.2	2	1	2	0
expat	2.6.2 → 2.7.3	8	3	13	2
expat	2.7.3 → 2.8.2	7	3	7	0
lz4	1.9.3 → 1.9.4	3	1	36	5
lz4	1.9.4 → 1.10.0	4	2	20	80
brotli	1.0.9 → 1.1.0	42	43	29	4
brotli	1.1.0 → 1.2.0	8	12	13	0
libpng	1.6.45 → 1.6.53	30	32	19	15
libpng	1.6.53 → 1.6.58	4	3	0	0
sqlite3	3.34.1 → 3.42.0	248	158	229	106
sqlite3	3.42.0 → 3.48.0	98	39	107	39
sqlite3	3.48.0 → 3.53.2	72	21	51	16
mbedtls	2.24.0 → 2.28.5	86	67	1	1
mbedtls	2.28.5 → 3.6.1	571	110	151	52
mbedtls	3.6.1 → 3.6.5	36	22	8	1
libxml2	2.11.8 → 2.14.6	1062	853	269	198
libxml2	2.14.6 → 2.15.3	224	124	36	197
curl	8.11.0 → 8.21.0	910	581	655	345
openssl	1.1.1j → 3.0.8	3845	825	5218	544
openssl	3.0.8 → 3.6.3	13023	987	4404	183

the almost equal share of + and - strings in each case indicate some stuff is probably just being updated with time (like Copyright 2026 → Copyright 2027 instead of new features being added. but its not 100% clear and needs more research on what strnigs are being changed instead of just how many.

and then, a scenario where strings are added and then removed without that middle release getting captured (such as a zlib version string or something). would a more comprehensive approach therefore be to build every release? what are the pros/cons there?

yeah hence my point that we shouldn't wait too long to build because of the missing middle, and we shouldn't wait too little because of wasted build time for not a lot of new information being gained.

gemini-code-assist Bot reviewed Jul 4, 2026

View reviewed changes

Comment thread scripts/build_oss_db.py

Comment thread scripts/build_oss_db.py

Comment thread scripts/build_oss_db.py

Comment thread floss/qs/db/data/oss/build_oss_db.py Outdated

Comment thread floss/qs/db/data/oss/build_oss_db.py Outdated

vee1e force-pushed the feat/auto-db-build-mandiant branch from 592ba36 to f63d643 Compare July 4, 2026 15:29

vee1e force-pushed the feat/auto-db-build-mandiant branch from f63d643 to e797372 Compare July 4, 2026 15:29

This was linked to issues Jul 4, 2026

qs: automate the construction of databases #796

Open

Create a local database build script and test it locally #1310

Open

Apply suggestions from code review

a6bf8b6

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

williballenthin self-assigned this Jul 4, 2026

williballenthin reviewed Jul 4, 2026

View reviewed changes

mr-tz reviewed Jul 4, 2026

View reviewed changes

vee1e added 4 commits July 5, 2026 02:58

fix(script): remove unnecessary defaults

e1cb1d0

fix(script): remove linux only libraries to keep the focus on PE

fb06ac3

chore(script): move the build db script to the global scripts dir

4f6044b

docs(script): add copyright

d9f704b

	Use the triplet `x64-windows-static` to build static archives (.lib files that are AR archives containing COFF object files):

	```console
	PS > C:\vcpkg\vcpkg.exe install --triplet x64-windows-static zlib
	```

Library	macOS	MinGW	In Both	mac Only	Win Only
binn	212	401	3	209	398
brotli	2,768	3,281	1,087	1,681	2,194
bzip2	571	632	63	508	569
curl	5,996	6,869	1,972	4,024	4,897
flatcc	751	1,085	87	664	998
fmt	719	1,246	59	660	1,187
freetype	4,938	5,985	954	3,984	5,031
libarchive	5,328	9,169	1,446	3,882	7,723
libflac	2,349	3,775	382	1,967	3,393
libjpeg-turbo	4,000	3,786	320	3,680	3,466
liblzma	1,182	1,653	159	1,023	1,494
libogg	226	269	81	145	188
libpng	1,909	2,111	436	1,473	1,675
libtheora	1,171	1,690	39	1,132	1,651
libvorbis	3,273	4,796	620	2,653	4,176
libwebp	3,185	4,803	117	3,068	4,686
libxml2	8,506	9,200	3,874	4,632	5,326
lz4	540	1,290	28	512	1,262
lzo	604	811	13	591	798
mbedtls	6,931	7,592	2,509	4,422	5,083
openal-soft	7,862	17,976	2,093	5,769	15,883
openssl	53,464	43,900	13,148	40,316	30,752
opus	2,729	3,224	581	2,148	2,643
opusfile	284	565	14	270	551
parson	245	343	5	240	338
physfs	1,202	1,588	76	1,126	1,512
sqlite3	8,335	7,518	1,221	7,114	6,297
tre	330	595	34	296	561
zlib	674	828	52	622	776
zstd	2,157	5,250	133	2,024	5,117

Uh oh!

Conversation

vee1e commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What the script does

What the workflow does

Todo

Errors in pipeline

Code review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

williballenthin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vee1e Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vee1e Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vee1e commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vee1e commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mr-tz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vee1e commented Jul 4, 2026 •

edited

Loading

vee1e Jul 4, 2026 •

edited

Loading

vee1e Jul 4, 2026 •

edited

Loading

vee1e commented Jul 4, 2026 •

edited

Loading

vee1e commented Jul 4, 2026 •

edited

Loading

williballenthin commented Jul 4, 2026 •

edited

Loading

vee1e commented Jul 4, 2026 •

edited

Loading