draft: try running live tests - v1

[jasonish]

SV_BRANCH=OISF/suricata-verify#3212

[suricata-qa]

Information: QA skipped due to no C or rust code changed detected. Set to force a run.

Pipeline = code

[suricata-qa]

Information: QA skipped due to no C or rust code changed detected. Set to force a run.

Pipeline = code

[suricata-qa]

Information: QA skipped due to no C or rust code changed detected. Set to force a run.

Pipeline = code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (8455efd) to head (01f4de8).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #15762      +/-   ##
==========================================
+ Coverage   83.03%   83.07%   +0.04%     
==========================================
  Files        1003     1003              
  Lines      276582   276582              
==========================================
+ Hits       229647   229758     +111     
+ Misses      46935    46824     -111     

Flag	Coverage Δ
fuzzcorpus	61.53% <ø> (-0.04%)	⬇️
livemode	25.74% <ø> (+7.27%)	⬆️
netns	26.75% <ø> (+3.84%)	⬆️
pcap	45.36% <ø> (-0.10%)	⬇️
suricata-verify	67.00% <ø> (-0.08%)	⬇️
unittests	58.46% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[suricata-qa]

Information: QA skipped due to tag. Set to force a run.

Pipeline = skip

[jasonish]

AI-generated review posted automatically by Suricata ai-review.

Verdict: request changes

• Medium — .github/workflows/builds.yml:2135 — The job runs all live tests but builds only with --enable-nfqueue and lacks tcpreplay and the XDP build prerequisites. Consequently, all three XDP tests—including the 8440 and 8699 regressions—are silently skipped. Install the EBPF/XDP dependencies and tcpreplay, configure with --enable-ebpf --enable-ebpf-build, and make this job reject unexpected skips.

• Medium — suricata-verify/live/tests/demo-replay-xdp-bypass-vlan/test.yaml:27 — The ported ticket-8699 test asserts only capture.kernel_packets >= 100. Unfixed Suricata also captures all 100 packets because XDP bypass never activates, so this test passes against the bug. Assert positive flow_bypassed.pkts/bytes or equivalent bypassed-flow state.

• Medium — suricata-verify/live/tests/demo-afp-xdp-bypass-stats-shutdown/test.yaml:35 — The ticket-8440 port checks local_capture_pkts and local_capture_bytes, which the ticket shows were already present before the fix. The missing shutdown results are flow_bypassed.pkts and flow_bypassed.bytes; restore positive assertions for those counters.

• Low — .github/workflows/builds.yml:2135 — The live suite runs coverage-instrumented Suricata without LLVM_PROFILE_FILE, and line 2137 merges only the eight pre-existing profiles. Thus the new executions contribute no data to combined.profdata. Use a collision-safe profile pattern and include the resulting live-test profiles in the merge.

Redmine and suricata-verify assessment: Tickets 8440, 8442, 8699, and transitive dependency ticket 7674 were fetched successfully. PR #3212 does not adequately cover 8440 or 8699: both weakened ports pass against the still-unfixed main, while their source PRs #3194 and #3204 contain distinguishing assertions. Tickets 8442 and 7674 appear only through transitive discussions/framework dependencies and are not claimed targets of #3212, so their absence is not an additional finding. The remaining namespace tests provide useful AF_PACKET, NFQ, IDS, firewall/reload, drop, and bypass coverage.

Validation: reviewed the complete one-commit diff from 8455efd9a to 4e7146487, the exact linked PR heads, discussions, commit histories, private Redmine content, workflow configuration, and CI logs. A prior integration run with the same #3212 head confirmed 14 passes, one unrelated failure, and the three XDP skips; #3212’s own properly provisioned live job ran 18 tests with zero skips. git diff --check passed and the worktree remained clean. Per review instructions, no builds or tests were run. The current-head namespace job remained queued, so no result was available for that revision.

Both investigator and adjudicator used Codex CLI with model gpt-5.6-sol at xhigh reasoning effort, and Codex was the final adjudicator.

[jasonish]

AI-generated review posted automatically by Suricata ai-review.

Verdict: request changes

• Medium — .github/workflows/builds.yml:2135 — Five live tests will be silently skipped. The job lacks tcpdump, tcpreplay, XDP/eBPF dependencies, and --enable-ebpf --enable-ebpf-build. The runner treats unmet requirements as skips and still exits successfully. Install the dependencies used by PR 3212’s live workflow and fail CI on unexpected skips.

• Medium — suricata-verify/live/tests/demo-afp-xdp-bypass-stats-shutdown/test.yaml:35 — The Redmine 8440 test asserts counters already present without the fix. It must assert positive flow_bypassed.pkts and flow_bypassed.bytes after shutdown while the bypass timeout remains longer than the test.

• Medium — suricata-verify/live/tests/demo-replay-xdp-bypass-vlan/test.yaml:27 — The Redmine 8699 test checks only captured packets, which remain positive when XDP bypass fails. Assert positive flow_bypassed.pkts/bytes or equivalent proof that VLAN-tagged traffic was bypassed.

• Medium — .github/workflows/builds.yml:2135 — This invocation currently depends on unmerged draft suricata-verify PR 3212. Non-PR builds default to suricata-verify master, where live/run.py does not exist, so merging this PR first would break the push workflow. Merge PR 3212 first or guard/version the invocation.

Redmine assessment: 8440 requires final capture-bypass packet/byte statistics during shutdown, and 8699 requires successful VLAN-aware XDP bypass; neither linked regression test distinguishes the fix from current unfixed main. Tickets 8442 and 7674 were fetched but are only transitive context, not claimed targets.

Linked suricata-verify assessment: PR 3212 at ff7fb68f provides useful AF_PACKET, NFQUEUE, IDS, firewall/reload, bypass, bond, and multi-network coverage. Its properly provisioned workflow passed 20 tests with zero skips, but this Suricata workflow would skip three XDP tests and two bond tests.

Validation and limitations: reviewed the complete diff from merge base 8455efd9a to bfe4ad43b, PR discussions, linked/source test PRs, all referenced Redmine records, workflow requirements, and CI logs. git diff --check passed and the worktree remained clean. No builds or tests were run per review instructions. The current Suricata namespace job remained queued. Podman is the only new CI dependency (Apache-2.0); no production linkage dependency was added.

Both investigator and adjudicator used Codex CLI with model gpt-5.6-sol at xhigh reasoning effort, and Codex was the final adjudicator.

[jasonish]

Live-test coverage now provided by suricata-verify

The paired suricata-verify PR, OISF/suricata-verify#3212, now contains 27 live tests. Of those, 15 are SV replacements for existing qa/live steps in this repository.

Existing ubuntu-latest-namespace-ips steps

Existing GitHub step	SV test
afp-ips-netns-bridge.sh 2 workers ips-netns.yaml	ips-drop-icmp-afp
afp-ips-netns-bridge.sh 2 autofp ips-netns.yaml	ips-drop-icmp-afp-autofp
afp-ips-netns-bond-bridge.sh 2 workers ips-netns-bond.yaml	afp-ips-bond
afp-ips-netns-bond-bridge2.sh 2 workers ips-netns-bond2.yaml	afp-ips-bond-two-networks
nfq-ips-netns-route.sh autofp ips-netns.yaml	ips-drop-icmp-nfq
nfq-ips-netns-route.sh workers ips-netns.yaml	ips-drop-icmp-nfq-workers
nfq-fw-netns-route.sh autofp fw-netns.yaml	nfq-fw
afp-fw-netns-bridge.sh 2 workers fw-netns.yaml	afp-fw

These cover AF_PACKET and NFQUEUE IPS, workers/autofp runmodes, single- and two-network bonded AF_PACKET topologies, firewall rule reloads, alert/verdict accounting, allowed HTTP traffic, blocked ICMP traffic, and server-side packet verification. The SV versions use the deterministic 10.200.x.x namespace topology.

Existing ubuntu-24-04-cov-afpdpdk and ubuntu-24-04-asan-afpdpdk steps

Existing GitHub step	SV test
afp-ids.sh 2 autofp	afp-ids-tpacket2-autofp
afp-ids.sh 2 workers	afp-ids-tpacket2-workers
afp-ids.sh 3 autofp	afp-ids-tpacket3-autofp
afp-ids.sh 3 workers	afp-ids-tpacket3-workers
pcap.sh autofp	pcap-ids-autofp
pcap.sh single	pcap-ids-single
multi-tenant.sh autofp	pcap-multi-tenant-autofp

These preserve the legacy tpacket argument behavior and cover AF_PACKET/libpcap packet capture, runmodes, IPv4/IPv6 datasets, malformed dataset input, rule reload, interface and capture-mode socket commands, hostbits, and tenant registration/reload/removal. Both jobs now run these seven tests through the legacy-afpdpdk tag alongside the existing shell steps. The coverage job gives these runs collision-safe LLVM profile names and includes them in its existing profile merge.

Additional SV live coverage

The remaining 12 tests add coverage beyond direct replacements for current qa/live job steps:

• simple-http-ids, simple-http-afp, and simple-http-nfq: end-to-end HTTP IDS/IPS traffic and alerting.
• ips-local-bypass-afp and ips-local-bypass-nfq: local bypass behavior while preserving end-to-end traffic.
• afp-ips-xdp-bypass: AF_PACKET IPS XDP capture bypass.
• nfq-ips-bypass: NFQUEUE IPS bypass behavior.
• demo-afp-xdp-bypass-stats-shutdown: XDP bypass statistics during the shutdown scenario from Redmine Pr 8341 and payload len fix/v5 #8440.
• demo-replay-xdp-bypass-vlan: VLAN-aware XDP bypass with the replay scenario from Redmine Suricata library v2 #8699.
• firewall-icmp: firewall rule-type behavior and reload.
• ruletype-firewall-96-lt-response-body-no-match-afp and ruletype-firewall-96-lt-response-body-no-match-nfq: HTTP response-body firewall regression coverage in both inline backends.

The new seven-test legacy-afpdpdk group passes locally with PASS: 7, FAIL: 0, and SKIP: 0.

DPDK null-PMD startup, mempool/cache/descriptor, and synthetic bonding configuration checks remain in their existing DPDK-specific shell tests; they are not claimed as namespace live-test ports.

[jasonish]

AI-generated review posted automatically by Suricata ai-review.

Verdict: request changes

• Medium — .github/workflows/builds.yml:1893 — All three new invocations depend on unmerged suricata-verify PR Compact DNS logging - v5 #3212. PR builds use the body override, but post-merge push builds default to suricata-verify master, where live/run.py does not exist. Merge Compact DNS logging - v5 #3212 first or guard/version these steps until the runner is available on master.

• Medium — .github/workflows/builds.yml:2142 — Five tests are silently skipped. The job lacks tcpdump, tcpreplay, XDP dependencies, and --enable-ebpf --enable-ebpf-build; the runner considers skips successful. A prior equivalent run reported 14 passes, one separate failure, and five skips: both bond tests and all three XDP tests. Install the dependencies/configuration used by Compact DNS logging - v5 #3212’s live workflow and reject unexpected skips.

• Medium — suricata-verify/live/tests/demo-afp-xdp-bypass-stats-shutdown/test.yaml:35 — The Redmine Pr 8341 and payload len fix/v5 #8440 regression checks local_capture_pkts/bytes, which the ticket shows are already present on unfixed main. The missing shutdown results are flow_bypassed.pkts/bytes. Assert both are positive while retaining a bypass timeout longer than the test.

• Medium — suricata-verify/live/tests/demo-replay-xdp-bypass-vlan/test.yaml:27 — The Redmine Suricata library v2 #8699 test only asserts captured packets. That succeeds when VLAN-aware XDP bypass is broken; Compact DNS logging - v5 #3212’s live CI passes this test against unfixed Suricata main. Assert positive flow_bypassed.pkts/bytes or equivalent proof that the VLAN-tagged flow was bypassed.

Redmine assessment: #8440 requires final capture-bypass packet/byte statistics during shutdown, and #8699 requires VLAN-aware XDP bypass to activate; neither current test distinguishes the fix. #8442 was only a transitive concurrency-test suggestion, while #7674 was provenance for prerequisite commits, so they add no requirements to this PR. All four tickets were fetched successfully with private content.

Linked suricata-verify assessment: PR #3212 at 73229f5e contains 27 useful live tests, including seven credible ports for the AF_PACKET/DPDK jobs. Its fully provisioned live workflow passed all 27 with zero skips. However, coverage remains ineffective for #8440 and #8699, and this Suricata workflow would skip five tests.

Validation and limitations: reviewed the complete two-commit diff from 8455efd9a to 5b1191ab0, PR discussions and commit histories, #3212 and source PRs #3194/#3204, all referenced Redmine records, runner semantics, job environments, and CI logs. git diff --check passed and the worktree remained clean. No builds or tests were run per review instructions; current-head Suricata jobs were still pending. New dependencies are CI-only: Podman is Apache-2.0, while ethtool, iproute2, and iptables use GPL-2.0-family licenses; no production linkage dependency was added.

Both investigator and adjudicator used Codex CLI with model gpt-5.6-sol at xhigh reasoning effort, and Codex was the final adjudicator.

[jasonish]

AI-generated review posted automatically by Suricata ai-review.

Verdict: request changes

• Medium — .github/workflows/builds.yml:1893 (also :2142, :2273) — All three invocations depend on draft suricata-verify PR Compact DNS logging - v5 #3212. PR builds use the body override, but push builds default to suricata-verify/master, where live/run.py does not exist. Merging this PR first would break these post-merge jobs. Land Compact DNS logging - v5 #3212 first, or guard/pin the steps to an available runner.

• Medium — .github/workflows/builds.yml:2142 — Five tests are silently skipped: two bond tests need tcpdump, while three XDP tests need tcpreplay, XDP dependencies, and --enable-ebpf --enable-ebpf-build. The runner exits successfully when tests skip. Install the missing dependencies/configuration and fail on unexpected skips.

• Medium — suricata-verify/live/tests/demo-afp-xdp-bypass-stats-shutdown/test.yaml:35 — The Redmine Pr 8341 and payload len fix/v5 #8440 regression checks local_capture_pkts/bytes, which the ticket demonstrates are already present on unfixed main. It therefore misses the shutdown bug. Explicitly keep the bypass timeout longer than the test and assert positive flow_bypassed.pkts and flow_bypassed.bytes.

• Medium — suricata-verify/live/tests/demo-replay-xdp-bypass-vlan/test.yaml:27 — The Redmine Suricata library v2 #8699 test only checks that 100 packets reached capture, which also succeeds when VLAN-aware XDP bypass is broken. PR Compact DNS logging - v5 #3212 passes this test against unfixed main. Assert positive flow_bypassed.pkts/bytes or equivalent proof that the VLAN-tagged flow entered XDP bypass.

Redmine assessment: #8440 requires final capture-bypass packet/byte statistics during shutdown; #8699 requires VLAN-aware XDP bypass to activate. Neither regression test distinguishes its fix. #8442 is only a separate concurrency-test suggestion, and #7674 is prerequisite/provenance for the replay framework; neither adds claimed requirements here. All four tickets were fetched successfully.

Linked suricata-verify assessment: PR #3212 at 73229f5e contains 27 useful tests, including credible ports of the existing namespace and AF_PACKET/PCAP jobs. Its fully provisioned workflow reported 27 passes, zero failures, and zero skips. Coverage remains ineffective for #8440 and #8699, and this Suricata job would skip five tests.

Validation and limitations: reviewed the complete three-commit diff from 8455efd9ac to 97f7057477, PR discussions, linked/source test PRs #3194, #3204, #3209, and #3212, ticket contents, runner semantics, job environments, and CI logs. The current coverage job’s seven-test live step passed; the current-head namespace and ASAN live steps were still pending. git diff --check passed, no merge conflicts were found, and the worktree remained clean. Per instructions, no builds or tests were run. New dependencies are CI-only: Podman is Apache-2.0; ethtool, iproute2, and iptables use GPL-2.0-family licenses. No production linkage dependency was added.

Both investigator and adjudicator used Codex CLI with model gpt-5.6-sol at xhigh reasoning effort, and Codex was the final adjudicator.

[jasonish]

AI-generated review posted automatically by Suricata ai-review.

Verdict: request changes

• Medium — .github/workflows/builds.yml:1893 (also :2142, :2273, :2924) — All four invocations depend on draft suricata-verify PR Compact DNS logging - v5 #3212. PR builds use the body override, but post-merge push builds default to suricata-verify/master, where live/run.py does not exist. Merge Compact DNS logging - v5 #3212 first, or guard/pin these steps until the runner is available.

• Medium — .github/workflows/builds.yml:2142 — Five tests are never executed by any new invocation. The two bond tests lack tcpdump; all three XDP tests lack the XDP dependencies and --enable-ebpf --enable-ebpf-build, while the VLAN replay also lacks tcpreplay. Because requirements become successful skips, CI remains green. Provision at least one job for these tests and enforce expected pass/skip counts.

• Medium — suricata-verify/live/tests/demo-afp-xdp-bypass-stats-shutdown/test.yaml:35 — The Redmine Pr 8341 and payload len fix/v5 #8440 regression checks local_capture_pkts/bytes, which the ticket shows are already present on unfixed main. Explicitly retain a bypass timeout longer than the test and assert positive flow_bypassed.pkts and flow_bypassed.bytes.

• Medium — suricata-verify/live/tests/demo-replay-xdp-bypass-vlan/test.yaml:27 — The Redmine Suricata library v2 #8699 test only proves the 100 packets reached capture, which also succeeds when VLAN-aware XDP bypass is broken. Assert positive flow_bypassed.pkts/bytes or equivalent evidence that the VLAN flow entered capture bypass.

Redmine assessment: #8440 requires final bypass packet/byte statistics during shutdown; #8699 requires VLAN-aware XDP bypass to activate. Neither current regression test distinguishes its fix. #8442 is a separate concurrency issue mentioned as future work, while #7674 supplies replay-framework prerequisites rather than claimed scope. All four tickets were fetched successfully with private content.

Linked suricata-verify assessment: PR #3212 at 69827cc3 contains 28 tests with useful AF_PACKET, NFQUEUE, PCAP, firewall, bypass, bond, multi-network, and DPDK coverage. Its live job ran 27 tests successfully with one expected DPDK skip, but it also passed both regressions against unfixed Suricata main; fixes #15774 and #15763 remain open.

Validation and limitations: reviewed the complete four-commit diff from 8455efd9ac to a5fe55e282, all 28 linked tests, source/predecessor test PRs, ticket records, runner skip semantics, workflow environments, and available CI logs. A prior equivalent namespace run confirmed 22 passes and the five identified skips. git diff --check passed, GitHub reports the PR mergeable, and the worktree remained clean. No builds or tests were run per instructions. Current-head live and DPDK jobs were still queued or running. New dependencies are CI-only: Podman is Apache-2.0; ethtool, iproute2, and iptables use GPL-2.0-family licenses; GNU inetutils is GPL-3.0-or-later; procps-ng contains GPL-2.0-or-later/LGPL-2.1-or-later components. No production linkage dependency was added.

Both investigator and adjudicator used Codex CLI with model gpt-5.6-sol at xhigh reasoning effort, and Codex was the final adjudicator.

[jasonish]

AI-generated review posted automatically by Suricata ai-review.

Verdict: request changes

• Medium — .github/workflows/builds.yml:1894 (also :2143, :2275, :2927) — All four invocations depend on draft suricata-verify PR Compact DNS logging - v5 #3212. PR builds use the body override, but post-merge push builds default to suricata-verify/master, where live/run.py does not exist. Merge Compact DNS logging - v5 #3212 first, or guard/pin these steps to an available runner.

• Medium — .github/workflows/builds.yml:2143 — Five tests are never exercised by any new invocation. Both bond tests require tcpdump; all three XDP tests require an XDP/eBPF build, while the VLAN replay also requires tcpreplay. Requirements become successful skips: an equivalent namespace run reported 22 passes and six skips, including these five. Provision at least one job with the necessary dependencies and --enable-ebpf --enable-ebpf-build, and reject unexpected skips.

• Medium — suricata-verify/live/tests/demo-afp-xdp-bypass-stats-shutdown/test.yaml:35 — The Redmine Pr 8341 and payload len fix/v5 #8440 regression checks only local_capture_pkts/bytes, which already work without the fix. Against unfixed main, it passed with those counters positive while flow_bypassed.pkts and .bytes remained zero. Pin the bypass timeout beyond the test duration and assert both final bypass counters are positive.

• Medium — suricata-verify/live/tests/demo-replay-xdp-bypass-vlan/test.yaml:27 — The Redmine Suricata library v2 #8699 port only proves that packets reached capture. It passed against unfixed main with flow_bypassed.pkts and .bytes both zero, meaning VLAN-aware XDP bypass never activated. Assert positive bypass counters or equivalent bypass state.

• Medium — suricata-verify/live/tests/pcap-multi-tenant-autofp/client.sh:25 — This newly enabled test remains nondeterministic even after the 15-second startup delay. At the current Compact DNS logging - v5 #3212 head, its fully provisioned workflow failed the second reload-tenants; one detect loader failed to load YAML. Fix the concurrent reload failure and demonstrate repeated clean runs, or exclude this test from the unfiltered gate until reliable.

• Low — .github/workflows/builds.yml:1894 — None of the new Suricata workflow invocations uploads suricata-verify/live/tests/**/output/** after failure. These contain the client, server, and Suricata diagnostics needed for intermittent failures. Add a failure-only artifact upload with a unique job/matrix name.

Redmine assessment: #8440 requires final capture-bypass packet/byte statistics when shutdown precedes the bypass timeout; #8699 requires VLAN-aware XDP bypass to activate. Neither linked regression distinguishes its fix. #8442 is separate concurrency work mentioned in discussion, while #7674 is provenance for replay-framework prerequisites; neither adds claimed requirements here. All four tickets were fetched successfully with private content.

Linked suricata-verify assessment: PR #3212 at feb23bbb6 is an open draft with 28 tests and substantial AF_PACKET, NFQUEUE, PCAP, firewall, bypass, bond, multi-network, runmode, and DPDK coverage, including 15 credible ports of existing qa/live steps. Its latest fully provisioned run reported 26 passes, one multi-tenant failure, and one expected DPDK skip. The source tests in #3194/#3209 and #3204 contain the distinguishing assertions weakened in #3212. Related framework and ticket-test predecessor PRs were also reviewed.

Validation and limitations: reviewed the complete diff from merge base 8455efd9ac to 01f4de8601, PR discussions and linked histories, all 28 tests, runner skip semantics, private ticket contents, and available CI logs/artifacts. git diff --check passed for the Suricata change, GitHub reports it mergeable, and both worktrees remained clean. No builds or tests were run, per review instructions. Current-head Suricata live/DPDK jobs remain queued or in progress.

New dependencies are CI-only: Podman is Apache-2.0; ethtool, iproute2, and iptables use GPL-2.0-family licensing; GNU inetutils is GPL-3.0-or-later; procps-ng contains GPL-2.0-or-later/LGPL-2.1-or-later components. No production linkage dependency was added.

Both investigator and adjudicator used Codex CLI with model gpt-5.6-sol at xhigh reasoning effort, and Codex was the final adjudicator.

[jasonish]

Replaced by #16076