Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions .github/workflows/build-oss-db.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
name: Build OSS String Databases

on:
# Rebuild the databases bi-weekly (1st and 15th of each month at 00:00 UTC).
schedule:
- cron: '0 0 1,15 * *'

# Allow manual runs from the Actions tab.
workflow_dispatch:

# Cancel any in-progress run when a newer one starts.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: write
pull-requests: write

jobs:
build-databases:
name: Build OSS string databases
runs-on: windows-latest

env:
# Workaround for zydis 3.1.3's CMakeLists.txt requiring an older
# cmake_minimum_required syntax. Remove once Lancelot updates zydis.
CMAKE_POLICY_VERSION_MINIMUM: '3.5'

steps:
- name: Checkout flare-floss
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
path: flare-floss

- name: Checkout lancelot
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
with:
repository: williballenthin/lancelot
path: lancelot

- name: Set up Python
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.12'

- name: Set up Rust
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable

- name: Cache Cargo
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
lancelot/target/
key: ${{ runner.os }}-cargo-jh-${{ hashFiles('lancelot/**/Cargo.lock') }}
restore-keys: |
${{ runner.os }}-cargo-jh-

- name: Cache vcpkg
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: |
C:\vcpkg\installed
C:\vcpkg\downloads
key: ${{ runner.os }}-vcpkg-oss-db-${{ hashFiles('flare-floss/floss/qs/db/data/oss/build_oss_db.py') }}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like this keys the hash on the generation script, not the contents of the libraries. am i reading that right?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think what we'd want to use is a vcpkg lock file, if that exists

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like this keys the hash on the generation script, not the contents of the libraries. am i reading that right?

yup, not perfect ofc, i was thinking of using the libraries.json config file as the hash so it updates whenever we add/remove a lib to build

i think what we'd want to use is a vcpkg lock file, if that exists

the requirement for that is fulfilled by the libraries.json file already imo

I think we can drop C:\vcpkg\installed from the cache because vcpkg names files in downloads/ by their content hash (e.g. openssl-3.4.1.tar.gz with a sha512 in the filename). When vcpkg installs a package it checks downloads/ first, then if the tarball is already there and the hash matches, it skips the network fetch and uses it directly.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok so it's not a cache of the built artifacts, but of the source downloads.

i still think this cache key isn't right, and probably confusing as is (maybe just a comment is needed).

@vee1e vee1e Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i still think this cache key isn't right, and probably confusing as is (maybe just a comment is needed).

yeah we can add comments on why the decision was made. maybe we can forgo the cache entirely?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think that's fine. if we end up spending obviously a lot of computer resources maybe we can optimize some, but let's not do that prematurely.

restore-keys: |
${{ runner.os }}-vcpkg-oss-db-

- name: Build jh
run: |
cargo build --release -p lancelot-bin
working-directory: lancelot

- name: Build OSS databases
run: |
python floss/qs/db/data/oss/build_oss_db.py `
--config floss\qs\db\data\oss\libraries.json `
--jh-path ..\lancelot\target\release\jh.exe `
--output-dir floss\qs\db\data\oss `
--continue-on-error
working-directory: flare-floss

- name: Show metrics summary
if: success() || failure()
run: |
Get-Content floss\qs\db\data\oss\build_metrics.json
working-directory: flare-floss

- name: Create Pull Request if databases changed
uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6.1.0
with:
path: flare-floss
base: quantumstrand
commit-message: 'chore(oss-db): update string databases'
title: 'Update OSS string databases'
body: |
Automated bi-weekly rebuild of the OSS string databases.

- Triplet: `x64-windows-static`
- Compiler: `msvc143`
- Library list: see `floss/qs/db/data/oss/libraries.json`

Per-library entry counts and timing are in `floss/qs/db/data/oss/build_metrics.json`.
branch: update-oss-string-databases
delete-branch: true
Loading
Loading