Repository navigation

Keep the judge's verdicts, a protection's past, and the plays of removed media #207

Workflow file for this run

.github/workflows/llm-security-scan.yml at a511607

	# LLM-based security scan (GPT-4o via the operator's OPENAI_API_KEY secret).
	#
	# Reworked from the uploaded template, which had two remote-code-execution
	# vectors and silently discarded its own findings:
	# - it executed .github/scripts/llm_security_scanner.py FROM THE PR CHECKOUT,
	# so any pull request editing that file ran arbitrary code on the runner
	# (with the OpenAI key in env for same-repo PRs). PR runs now execute the
	# scanner from the PR's BASE commit — the reviewed, trusted copy.
	# - it interpolated changed-file names into a `run:` block via ${{ }}, the
	# classic script-injection pattern; filenames now travel NUL-delimited
	# through a file and are never parsed by a shell.
	# - tj-actions/changed-files@v40 (CVE-2025-30066: tags repointed to a
	# secret-dumping commit in March 2025) is replaced by two lines of git.
	# - every push to main ran a paid full-repo scan (~63 min of GPT-4o); pushes
	# are covered by CodeQL for free, so the full scan now runs weekly and on
	# manual dispatch only.
	#
	# Fork PRs get no secrets by GitHub policy: the scan is skipped with a notice
	# instead of failing red or — worse — passing green having scanned nothing.
	name: LLM Security Scan

	on:
	pull_request:
	branches: [main]
	schedule:
	- cron: "0 8 * * 1" # Mondays, 08:00 UTC — full scan
	workflow_dispatch:

	concurrency:
	group: llm-scan-${{ github.ref }}
	cancel-in-progress: true

	permissions:
	contents: read
	issues: write

	jobs:
	security-scan:
	runs-on: ubuntu-latest
	env:
	HAS_KEY: ${{ secrets.OPENAI_API_KEY != '' }}
	# PR diffs are small — spend the flagship there (sol is ~25ct per PR).
	# The weekly sweep walks ~200 files, so it runs on the current-gen cheap
	# tier (~40ct per sweep). For a one-off specialist deep-dive, dispatch
	# manually with gpt-daybreak-red-latest (OpenAI's security line, ~$25).
	PR_MODEL: gpt-5.6-sol
	FULL_MODEL: gpt-5.6-luna
	# The weekly sweep files issues only from this severity up; lower it to
	# 'high' to widen. Every finding lands in the run summary regardless.
	ISSUE_MIN_SEVERITY: critical
	steps:
	- name: Checkout repository
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	fetch-depth: 0

	- name: Set up Python
	uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7
	with:
	python-version: "3.12"

	- name: Install dependencies
	if: env.HAS_KEY == 'true'
	run: pip install --require-hashes -r lock/requirements-llm-scan.txt

	- name: No API key — skip scan
	if: env.HAS_KEY != 'true'
	run: \|
	echo "::notice::OPENAI_API_KEY is not available (fork PRs never receive secrets) — LLM scan skipped."

	# PR path: scan only the files this PR changes, with the scanner taken
	# from the base commit. Filenames are attacker-controlled input on a
	# public repo: they go NUL-delimited into a file the scanner reads
	# itself, so no shell ever word-splits or interprets them.
	- name: LLM Security Scan (PR — changed files)
	if: github.event_name == 'pull_request' && env.HAS_KEY == 'true'
	env:
	OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
	BASE_SHA: ${{ github.event.pull_request.base.sha }}
	run: \|
	mkdir -p scan-results
	if ! git cat-file -e "$BASE_SHA:.github/scripts/llm_security_scanner.py" 2>/dev/null; then
	echo "::notice::scanner does not exist on the base commit yet — skipping (it must land on main before it scans PRs)."
	exit 0
	fi
	git show "$BASE_SHA:.github/scripts/llm_security_scanner.py" > /tmp/trusted_scanner.py
	git diff -z --name-only --diff-filter=ACMR "$BASE_SHA"...HEAD -- \
	'.py' '.js' '.ts' '.java' '.c' '.cpp' '.go' '.php' '*.rb' \
	> /tmp/changed-files.z
	if [ ! -s /tmp/changed-files.z ]; then
	echo "No scannable files changed."
	exit 0
	fi
	python /tmp/trusted_scanner.py \
	--file-list /tmp/changed-files.z --max-files 40 \
	--model "$PR_MODEL" \
	--output-format markdown --output-file scan-results/results.md \
	--json-output-file scan-results/results.json

	# Full sweep: weekly and on manual dispatch. Deliberately NOT on push —
	# CodeQL already runs there for free, and a full GPT-4o pass over the
	# repo costs real money and the better part of an hour.
	#
	# Scope = the server. Owner-run CLI tools at the repo root and under
	# scripts/ (a shell on the host already outranks anything they could
	# add), the test suite (fixtures and monkeypatches are not shipped
	# code; CodeQL still reads them) and vendored third-party code (pinned,
	# OSV-scanned, marked is wrapped in DOMPurify) are left out: every
	# finding the model produced for them was by-design and got re-filed
	# weekly under a new title. Note the model never sees frontend/index.html
	# (no .html in its extension list); the inline JS is CodeQL's job.
	- name: LLM Security Scan (Full)
	if: github.event_name != 'pull_request' && env.HAS_KEY == 'true'
	env:
	OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
	run: \|
	mkdir -p scan-results
	python .github/scripts/llm_security_scanner.py \
	--directory . \
	--exclude-dirs .git node_modules venv __pycache__ .env scan-results .github \
	--exclude-paths frontend/vendor scripts tests \
	benchmark.py build_models.py import_spotify.py run_pipeline_spotify.py \
	update_db.py curatarr_tray.pyw \
	--model "$FULL_MODEL" \
	--output-format markdown --output-file scan-results/results.md \
	--json-output-file scan-results/results.json

	# Issues only from the trusted full sweep, read from the JSON report
	# (the template parsed its own markdown back apart with a state machine
	# that misattributed nearly every finding's file). Worst first, so the
	# per-run cap surfaces a critical in src/ rather than whatever file
	# sorts first; deduplicated against every security issue ever filed
	# (the model rewords a finding each week, a closed issue stays closed);
	# and filtered through .github/llm-scan-accepted.json — the owner's
	# by-design decisions, kept in the repo where the next run can read
	# them. 2026-09-07: the weekly sweep produced 400 medium+ findings and
	# filed the ten alphabetically first, all of them re-reports of issues
	# closed as by-design four days earlier.
	- name: Create GitHub Issues for Vulnerabilities
	if: github.event_name != 'pull_request' && env.HAS_KEY == 'true'
	uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
	with:
	github-token: ${{ secrets.GITHUB_TOKEN }}
	script: \|
	const fs = require('fs');
	const path = 'scan-results/results.json';
	if (!fs.existsSync(path)) {
	core.info('No scan report — nothing to file.');
	return;
	}
	const report = JSON.parse(fs.readFileSync(path, 'utf8'));

	// The owner's by-design decisions, as file + type patterns with a
	// reason. A matching finding is logged and skipped; it still sits
	// in the uploaded report.
	const acceptedPath = '.github/llm-scan-accepted.json';
	const accepted = fs.existsSync(acceptedPath)
	? (JSON.parse(fs.readFileSync(acceptedPath, 'utf8')).accepted \|\| [])
	: [];
	const acceptedBy = (file, type) => accepted.find(a =>
	new RegExp(a.file, 'i').test(file \|\| '') && new RegExp(a.type, 'i').test(type \|\| ''));

	const RANK = { critical: 0, high: 1, medium: 2 };
	const vulns = [];
	let skipped = 0;
	for (const result of report.results \|\| []) {
	if (result.status !== 'completed') continue;
	for (const v of result.vulnerabilities \|\| []) {
	const sev = String(v.severity \|\| '').trim().toLowerCase();
	if (!(sev in RANK)) continue;
	const rule = acceptedBy(result.file, v.vulnerability_type);
	if (rule) {
	skipped++;
	core.info(`Accepted (${rule.reason}): ${result.file} — ${v.vulnerability_type}`);
	continue;
	}
	vulns.push({ ...v, file: result.file, sev });
	}
	}
	vulns.sort((a, b) => (RANK[a.sev] - RANK[b.sev]) \|\| String(a.file).localeCompare(String(b.file)));
	core.info(`${vulns.length} medium+ findings to consider, ${skipped} accepted by the owner's list.`);

	// The digest lives on the run page: counts and the worst 25, so
	// nobody has to open the artifact to know what the week looked like.
	const count = s => vulns.filter(v => v.sev === s).length;
	await core.summary
	.addHeading('LLM security sweep', 2)
	.addRaw(`${vulns.length} findings (critical ${count('critical')}, high ${count('high')}, medium ${count('medium')}); ${skipped} accepted via .github/llm-scan-accepted.json. Full report in the artifact.`, true)
	.addTable([
	[{ data: 'Severity', header: true }, { data: 'File', header: true }, { data: 'Finding', header: true }],
	...vulns.slice(0, 25).map(v => [v.sev, String(v.file), String(v.vulnerability_type \|\| '')]),
	])
	.write();

	// Issues only from the configured severity up. The model reads one
	// file at a time and cannot see the control that lives in the next
	// file, so its highs are mostly cross-file false positives; they
	// stay in the digest. ISSUE_MIN_SEVERITY widens or narrows this.
	const minSev = String(process.env.ISSUE_MIN_SEVERITY \|\| 'critical').toLowerCase();
	const toFile = vulns.filter(v => RANK[v.sev] <= (RANK[minSev] ?? 0));
	core.info(`${toFile.length} at or above ${minSev} are candidates for issues.`);
	if (!toFile.length) return;

	const norm = s => String(s \|\| '').toLowerCase().replace(/[^a-z0-9]+/g, ' ').trim();
	const existing = await github.paginate(github.rest.issues.listForRepo, {
	owner: context.repo.owner, repo: context.repo.repo,
	state: 'all', labels: 'security', per_page: 100,
	});
	const known = new Set(existing.map(i => norm(i.title)));

	const CAP = 10;
	let created = 0;
	for (const v of toFile) {
	const sevLabel = v.sev.charAt(0).toUpperCase() + v.sev.slice(1);
	const title = `[${sevLabel}] ${v.vulnerability_type \|\| 'Security finding'} in ${v.file}`;
	if (known.has(norm(title))) {
	core.info(`Already filed (open or closed): ${title}`);
	continue;
	}
	if (created >= CAP) {
	core.warning(`Issue cap (${CAP}) reached — remaining findings stay in the artifact.`);
	break;
	}
	const lines = Array.isArray(v.line_numbers) ? v.line_numbers.join(', ') : String(v.line_numbers ?? '');
	const body = [
	`## Security Finding: ${v.vulnerability_type \|\| 'Unknown'}`,
	'',
	`File: \`${v.file}\``,
	`Severity: ${sevLabel}`,
	`Line Numbers: ${lines}`,
	'',
	'### Description', v.description \|\| '_none provided_',
	'',
	'### Impact', v.impact \|\| '_none provided_',
	'',
	'### Recommendation', v.recommendation \|\| '_none provided_',
	'',
	v.fix_example ? `### Fix Example\n\`\`\`\n${v.fix_example}\n\`\`\`` : '',
	'',
	'---',
	'Filed automatically by the LLM security scan. Findings are model-generated — verify before acting.',
	].join('\n');
	try {
	const issue = await github.rest.issues.create({
	owner: context.repo.owner, repo: context.repo.repo,
	title, body, labels: ['security', `severity:${v.sev}`],
	});
	known.add(norm(title));
	created++;
	core.info(`Created #${issue.data.number}: ${title}`);
	} catch (e) {
	core.error(`Failed to create issue for ${v.file}: ${e.message}`);
	}
	}

	- name: Upload Scan Results
	if: env.HAS_KEY == 'true' && hashFiles('scan-results/**') != ''
	uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
	with:
	name: security-scan-results
	path: scan-results/

Navigation Menu

Repository navigation

Keep the judge's verdicts, a protection's past, and the plays of removed media #207

Workflow file

Keep the judge's verdicts, a protection's past, and the plays of removed media #207

Uh oh!

Workflow file for this run