-
Notifications
You must be signed in to change notification settings - Fork 534
feat(qs): automate OSS string database builds #1327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: quantumstrand
Are you sure you want to change the base?
Changes from 3 commits
edd0e5e
e797372
a6bf8b6
e1cb1d0
fb06ac3
4f6044b
d9f704b
8ea3fe1
0b6f915
cf97210
f0872a3
594ce4a
1c1dbfa
d272c2c
daaf96d
582999e
da23b8c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| name: Build OSS String Databases | ||
|
|
||
| on: | ||
| # Rebuild the databases bi-weekly (1st and 15th of each month at 00:00 UTC). | ||
| schedule: | ||
| - cron: '0 0 1,15 * *' | ||
|
|
||
| # Allow manual runs from the Actions tab. | ||
| workflow_dispatch: | ||
|
|
||
| # Cancel any in-progress run when a newer one starts. | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| permissions: | ||
| contents: write | ||
| pull-requests: write | ||
|
|
||
| jobs: | ||
| build-databases: | ||
| name: Build OSS string databases | ||
| runs-on: windows-latest | ||
|
|
||
| env: | ||
| # Workaround for zydis 3.1.3's CMakeLists.txt requiring an older | ||
| # cmake_minimum_required syntax. Remove once Lancelot updates zydis. | ||
| CMAKE_POLICY_VERSION_MINIMUM: '3.5' | ||
|
|
||
| steps: | ||
| - name: Checkout flare-floss | ||
| uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 | ||
| with: | ||
| path: flare-floss | ||
|
|
||
| - name: Checkout lancelot | ||
| uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 | ||
| with: | ||
| repository: williballenthin/lancelot | ||
| path: lancelot | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| - name: Set up Rust | ||
| uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable | ||
|
|
||
| - name: Cache Cargo | ||
| uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 | ||
| with: | ||
| path: | | ||
| ~/.cargo/bin/ | ||
| ~/.cargo/registry/index/ | ||
| ~/.cargo/registry/cache/ | ||
| ~/.cargo/git/db/ | ||
| lancelot/target/ | ||
| key: ${{ runner.os }}-cargo-jh-${{ hashFiles('lancelot/**/Cargo.lock') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-cargo-jh- | ||
|
|
||
| - name: Cache vcpkg | ||
| uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 | ||
| with: | ||
| path: | | ||
| C:\vcpkg\installed | ||
| C:\vcpkg\downloads | ||
| key: ${{ runner.os }}-vcpkg-oss-db-${{ hashFiles('flare-floss/floss/qs/db/data/oss/build_oss_db.py') }} | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. seems like this keys the hash on the generation script, not the contents of the libraries. am i reading that right?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i think what we'd want to use is a vcpkg lock file, if that exists
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
yup, not perfect ofc, i was thinking of using the libraries.json config file as the hash so it updates whenever we add/remove a lib to build
the requirement for that is fulfilled by the libraries.json file already imo I think we can drop
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ah ok so it's not a cache of the built artifacts, but of the source downloads. i still think this cache key isn't right, and probably confusing as is (maybe just a comment is needed).
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
yeah we can add comments on why the decision was made. maybe we can forgo the cache entirely?
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i think that's fine. if we end up spending obviously a lot of computer resources maybe we can optimize some, but let's not do that prematurely. |
||
| restore-keys: | | ||
| ${{ runner.os }}-vcpkg-oss-db- | ||
|
|
||
| - name: Build jh | ||
| run: | | ||
| cargo build --release -p lancelot-bin | ||
| working-directory: lancelot | ||
|
|
||
| - name: Build OSS databases | ||
| run: | | ||
| python floss/qs/db/data/oss/build_oss_db.py ` | ||
| --config floss\qs\db\data\oss\libraries.json ` | ||
| --jh-path ..\lancelot\target\release\jh.exe ` | ||
| --output-dir floss\qs\db\data\oss ` | ||
| --continue-on-error | ||
| working-directory: flare-floss | ||
|
|
||
| - name: Show metrics summary | ||
| if: success() || failure() | ||
| run: | | ||
| Get-Content floss\qs\db\data\oss\build_metrics.json | ||
| working-directory: flare-floss | ||
|
|
||
| - name: Create Pull Request if databases changed | ||
| uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6.1.0 | ||
| with: | ||
| path: flare-floss | ||
| base: quantumstrand | ||
| commit-message: 'chore(oss-db): update string databases' | ||
| title: 'Update OSS string databases' | ||
| body: | | ||
| Automated bi-weekly rebuild of the OSS string databases. | ||
|
|
||
| - Triplet: `x64-windows-static` | ||
| - Compiler: `msvc143` | ||
| - Library list: see `floss/qs/db/data/oss/libraries.json` | ||
|
|
||
| Per-library entry counts and timing are in `floss/qs/db/data/oss/build_metrics.json`. | ||
| branch: update-oss-string-databases | ||
| delete-branch: true | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there an issue for that in Lancelot? if not can you please add one?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and reference here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah there is one, i'll make an issue regarding it