Skip to content

feat(qs): automate OSS string database builds #1327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vee1e wants to merge 12 commits into
base: quantumstrand
Choose a base branch
from
Open

feat(qs): automate OSS string database builds #1327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
edd0e5e
feat(oss-db): add automated build script
vee1e Jul 4, 2026
e797372
ci(oss-db): add bi-weekly rebuild workflow
vee1e Jul 4, 2026
a6bf8b6
Apply suggestions from code review
vee1e Jul 4, 2026
e1cb1d0
fix(script): remove unnecessary defaults
vee1e Jul 4, 2026
fb06ac3
fix(script): remove linux only libraries to keep the focus on PE
vee1e Jul 5, 2026
4f6044b
chore(script): move the build db script to the global scripts dir
vee1e Jul 5, 2026
d9f704b
docs(script): add copyright
vee1e Jul 6, 2026
8ea3fe1
fix(script): remove problematic and unneeded lib `clblas`
vee1e Jul 6, 2026
0b6f915
fix(workflow): enable LFS checkout for build-oss-db
vee1e Jul 6, 2026
cf97210
chore(workflow): drop Cargo and vcpkg caching from build-oss-db
vee1e Jul 6, 2026
f0872a3
feat(oss-db): stop deduping shared strings; stamp cross-library count
vee1e Jul 6, 2026
594ce4a
test: rename test_oss_db.py to test_qs_oss_db.py
vee1e Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions .github/workflows/build-oss-db.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
name: Build OSS String Databases

on:
# Rebuild the databases bi-weekly (1st and 15th of each month at 00:00 UTC).
schedule:
- cron: '0 0 1,15 * *'

# Allow manual runs from the Actions tab.
workflow_dispatch:

# Cancel any in-progress run when a newer one starts.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: write
pull-requests: write

jobs:
build-databases:
name: Build OSS string databases
runs-on: windows-latest

env:
# Workaround for zydis 3.1.3's CMakeLists.txt requiring an older
# cmake_minimum_required syntax. Remove once Lancelot updates zydis.
CMAKE_POLICY_VERSION_MINIMUM: '3.5'

steps:
- name: Checkout flare-floss
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
path: flare-floss
lfs: true

- name: Checkout lancelot
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
repository: williballenthin/lancelot
path: lancelot
lfs: true

- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'

- name: Set up Rust
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable

- name: Build jh
run: |
cargo build --release -p lancelot-bin
working-directory: lancelot

- name: Build OSS databases
run: |
python scripts/build_oss_db.py `
--config floss\qs\db\data\oss\libraries.json `
--jh-path ..\lancelot\target\release\jh.exe `
--output-dir floss\qs\db\data\oss `
--continue-on-error
working-directory: flare-floss

- name: Show metrics summary
if: success() || failure()
run: |
Get-Content floss\qs\db\data\oss\build_metrics.json
working-directory: flare-floss

- name: Create Pull Request if databases changed
uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6.1.0
with:
path: flare-floss
base: quantumstrand
commit-message: 'chore(oss-db): update string databases'
title: 'Update OSS string databases'
body: |
Automated bi-weekly rebuild of the OSS string databases.

- Triplet: `x64-windows-static`
- Compiler: `msvc143`
- Library list: see `floss/qs/db/data/oss/libraries.json`

Per-library entry counts and timing are in `floss/qs/db/data/oss/build_metrics.json`.
branch: update-oss-string-databases
delete-branch: true
69 changes: 69 additions & 0 deletions floss/qs/db/data/oss/libraries.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
{
"triplet": "x64-windows-static",
"compiler": "msvc143",

@williballenthin williballenthin Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this chosen and when should this be updated?

@williballenthin williballenthin Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what effect, if any, does it have on the compilation?

@vee1e vee1e Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this chosen and when should this be updated?

taken directly from the existing qs documentation

flare-floss/floss/qs/db/data/oss/readme.md

Lines 48 to 52 in 0fbe890

Use the triplet `x64-windows-static` to build static archives (.lib files that are AR archives containing COFF object files):
```console
PS > C:\vcpkg\vcpkg.exe install --triplet x64-windows-static zlib
```

what effect, if any, does it have on the compilation?

i tried the builds locally on macosx-arm64. no significant differences in string counts was observed

@mr-tz mr-tz Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What builds did you try?

@williballenthin williballenthin Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i probably picked 143 at the time because it was probably the latest version or something. i assume there's not a major difference between versions, but we should document our expectations and assumptions. that's why i raise it here.

@vee1e vee1e Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What builds did you try?

Using raw strings on both I got this:

Library macOS MinGW In Both mac Only Win Only
binn 212 401 3 209 398
brotli 2,768 3,281 1,087 1,681 2,194
bzip2 571 632 63 508 569
curl 5,996 6,869 1,972 4,024 4,897
flatcc 751 1,085 87 664 998
fmt 719 1,246 59 660 1,187
freetype 4,938 5,985 954 3,984 5,031
libarchive 5,328 9,169 1,446 3,882 7,723
libflac 2,349 3,775 382 1,967 3,393
libjpeg-turbo 4,000 3,786 320 3,680 3,466
liblzma 1,182 1,653 159 1,023 1,494
libogg 226 269 81 145 188
libpng 1,909 2,111 436 1,473 1,675
libtheora 1,171 1,690 39 1,132 1,651
libvorbis 3,273 4,796 620 2,653 4,176
libwebp 3,185 4,803 117 3,068 4,686
libxml2 8,506 9,200 3,874 4,632 5,326
lz4 540 1,290 28 512 1,262
lzo 604 811 13 591 798
mbedtls 6,931 7,592 2,509 4,422 5,083
openal-soft 7,862 17,976 2,093 5,769 15,883
openssl 53,464 43,900 13,148 40,316 30,752
opus 2,729 3,224 581 2,148 2,643
opusfile 284 565 14 270 551
parson 245 343 5 240 338
physfs 1,202 1,588 76 1,126 1,512
sqlite3 8,335 7,518 1,221 7,114 6,297
tre 330 595 34 296 561
zlib 674 828 52 622 776
zstd 2,157 5,250 133 2,024 5,117 
=== binn macOS-only (arm64 machine code) ===
  '\t@9j'           ← random bytes, not text
  ' #[z9'           ← ditto
  '@9+\t@9,'        ← ditto
  'K)@)'            ← ditto
  'R)#@'            ← ditto
  '_AddValue'       ← symbol name (mangled C function)
  '_AdvanceDataPos' ← symbol name
  '__.SYMDEF SORTED'← macOS BSD ar archive metadata
  '#1/12  1782384003  501  20  644  57180  `'
                     ← macOS BSD ar archive metadata

=== binn mingw-only (x86_64 machine code) ===
  '\tw}A'           ← random bytes
  '([^_]A\A]A^A_'   ← fragments of x86_64 instructions
  '8[^_]A\A]A^A_'   ← ditto
  'AWAVAUATUWVSH'   ← x86_64 function prologue bytes
  'AVWVSH'          ← ditto
  '/  1783197601  0  0  0  0  2520  `'  ← GNU ar archive metadata

So there is a difference in strings but the total count remains similar. I didn't look at which strings overlapped before, hence the slightly wrong conclusion that there was no difference. But most of the lack of overlap can be attributed to:

  1. Random printable byte runs in compiled code that look like strings to strings -n 4. Different ISAs produce different random byte runs, so the noise never overlaps.
  2. Section names differ. Mach-O has __TEXT, __cstring etc. while COFF has .rdata, .pdata, .xdata
  3. Symbol name prefixes are different too. Mach-O adds a leading _ (_AddValue), COFF doesn't (AddValue)

It could be worth building another db set for different platforms if we want to cover all possible executables, but it'll increase build times and cost. I think its a fair point to debate imo

@mr-tz mr-tz Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment here was on the msvc version :) interesting additional information though. With our PE focus we should stick to Windows for now.

"profile": "release",
"libraries": [
"openssl",
"capnproto",
"raylib",
"libgit2",
"libxml2",
"libimobiledevice",
"libarchive",
"libmysql",
"libhv",
"cello",
"czmq",
"boost-test",
"wolfssl",
"boost-wave",
"curl",

@williballenthin williballenthin Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't zlib a dep of curl, so does that mean the zlib strings get filter as duplicate, rather than being assigned to zlib?

how do you handle this dependency issue?

@vee1e vee1e Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vcpkg builds curl and zlib as separate static .lib files so jh only sees zlib's symbols when processing zlib.lib directly. When processing libcurl.lib, it only extracts strings from curl's own compiled code. it doesn't re-extract zlib's symbols just because curl links against it.

you can also see a very minimal string overlap in my local fork: vee1e#9

367: ...zlib.jsonl.gz (149 entries, 8 removed as shared with other libraries)

@mr-tz mr-tz Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing these would be bad

@williballenthin williballenthin Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree that when the dependency is via linking, then things may work out. but in some cases the dependency may be vendored, ie, source code copied into the project. in this case, i think the strings will be detected as duplicates.

to be clear, i believe this is a hard problem to handle generically. though maybe it can also be solved by a little research and hacks/configs/shorcuts.

@vee1e vee1e Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of removing duplicates altogether should we be upfront to the user and tell them exactly how commonly a string was found? maybe if its truly unique we can indicate it as a "pure" indicator tag, and with every subsequent instance of the string being found we can somehow show the library tag (#zlib etc) as "less" of a true oracle for its source.

@mr-tz mr-tz Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping duplicates around may be the safer bet for now until we see more of this in practice or do more research into the overlaps.

"poco",
"nng",
"libsrt",
"sail",
"mbedtls",
"boost-log",
"libsndfile",
"libevent",
"libcoap",
"sqlite3",
"sdl2",
"libbson",
"mongoose",
"duktape",
"libpcap",
"expat",
"libuv",
"llhttp",
"jansson",
"tinyfiledialogs",
"hiredis",
"zyre",
"boost-json",
"boost-serialization",
"tre",
"zlib",
"boost-graph",
"apr",
"libev",
"flatcc",
"boost-filesystem",
"boost-iostreams",
"oniguruma",
"libyaml",
"cunit",
"avro-c",
"lz4",
"parson",
"json-c",
"libzip",
"boost-container",
"lzo",
"liblzma",
"boost-chrono",
"lmdb",
"binn",
"boost-thread"
]
}
16 changes: 16 additions & 0 deletions floss/qs/db/data/oss/readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,19 @@ These files are then gzip'd:
```console
$ gzip -c zlib.jsonl > zlib.jsonl.gz
```

The `build_oss_db.py` script automates the steps above and additionally

@williballenthin williballenthin Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i like that the logic lives in this script, and that the GH workflow is minimal. this reduces our dependence on GH, which i believe is prudent.

removes strings that appear in two or more libraries from every database.
A shared string is not a unique identifier for any specific library, so

@williballenthin williballenthin Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as noted above: unless one library links or vendors another library.

attributing it would produce false-positive library matches at query time.

The script also merges into any `.jsonl.gz` already present in the output
directory rather than replacing them. For each library being rebuilt, the
newly-extracted entries are combined with the existing entries (new entries
win on string collisions, which is the right behavior when the library
version changes). Libraries that are not in the current `--libraries` list
are also re-checked against the cross-library dedup and rewritten if the
shared-string set changes. This makes it safe to incrementally add a new
library (e.g. `python build_oss_db.py --libraries newlib`) without losing
the other databases, and safe to bump one library's version while leaving
its previously-committed entries around.
5 changes: 5 additions & 0 deletions floss/qs/db/oss.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ class OpenSourceString(msgspec.Struct):
file_path: str | None = None
function_name: str | None = None
line_number: int | None = None
# Number of distinct libraries the string was observed in across the full
# OSS database set. count=1 means the string is unique to this library;
# higher counts indicate strings shared across multiple libraries (e.g.
# vendored dependencies or common runtime strings).
count: int = 1


@dataclass
Expand Down
Loading
Loading