detect/bytemath: bound shift counts

[jlucovsky]

Continuation of #16046

Link to ticket: https://redmine.openinfosecfoundation.org/issues/8845

Describe changes:

• Zero the byte_math result when a right shift count reaches 64, the width of the uint64_t being shifted. The left shift case has been bounded this way since 473ca6d; the right shift case was left as it was, so DetectByteMathDoMatch() shifted by whatever count it was handed. The count is reachable from the payload through byte_extract and a variable rvalue.
• Reject at rule load a literal rvalue of 64 or more paired with << or >>. rvalue was bounded only to u32::MAX. A variable rvalue still loads, since its value is not known until the rule runs; the runtime guard is what covers that path.
• Document that << and >> yield 0 for a count of 64 or more, beside the existing note about division by zero.

Verification: clean build with --enable-warnings --enable-rust-strict
--enable-unittests; C unit tests and cargo test detect::byte_math pass; both new suricata-verify tests pass and both fail on an unfixed build.

Note the second commit fails rules that load today (byte_math with a literal
shift count of 64 or more). That suits 9.0.0-beta1. It is deliberately a
separate commit from the fix so the backport question can be answered on its
own -- main-8.0.x still has the unguarded shift at src/detect-bytemath.c:191.

Updates:

• Rebase

Provide values to any of the below to override the defaults.

SV_BRANCH=OISF/suricata-verify#3288

[codecov]

Codecov Report

❌ Patch coverage is 98.07692% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.99%. Comparing base (17bd53c) to head (986e7bd).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16067      +/-   ##
==========================================
- Coverage   83.02%   82.99%   -0.04%     
==========================================
  Files        1002     1002              
  Lines      276951   276999      +48     
==========================================
- Hits       229933   229886      -47     
- Misses      47018    47113      +95     

Flag	Coverage Δ
fuzzcorpus	61.56% <76.47%> (+<0.01%)	⬆️
livemode	18.41% <0.00%> (-0.09%)	⬇️
netns	22.87% <0.00%> (-0.02%)	⬇️
pcap	45.37% <0.00%> (-0.05%)	⬇️
suricata-verify	67.03% <88.23%> (-0.03%)	⬇️
unittests	58.50% <98.07%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[suricata-qa]

Information: QA ran without warnings.

Pipeline = 33203

[catenacyber]

Thanks for the work,

CI : ✅
Git ID set : looks fine for me
CLA : you already contributed
Doc update : cool
Redmine ticket : ok
Tests : cool
Dependencies added: none
Code : good
Commits segmentation : I would squash the doc one but you know it already
Commit messages : 🟡 a bit too verbose to my taste

[xhon-pelushi]

Followed this over from #16046 — the shift bound and the new DetectByteMathParseTest17 both look right to me, and rejecting the literal form at rule load while leaving the variable form to be clamped at match time is the right split.

One small thing that didn't carry over, non-blocking since this is already approved:

The docs line still reads

Note: if oper is << or >> and rvalue is 64 or greater, the result is 0.

but after this PR a literal rvalue of 64 or more is refused at rule load with rvalue must be less than 64 when used with "<<" or ">>". So as written the sentence describes an outcome the reader cannot actually produce the obvious way — the "result is 0" behaviour is now only reachable when rvalue is a variable resolved at match time. Something like:

Note: if oper is << or >>, an rvalue of 64 or greater is rejected at rule load. If rvalue is a variable whose value at match time is 64 or greater, the result is 0.

Also worth a word in DetectByteMathPacket03 about why the count is specifically 0x40: the test discriminates only because 0xff >> 64 masks to >> 0 on x86-64 and yields 255 against an unfixed build. Pick 72 instead and the same expression masks to >> 8, which yields 0 — the very value the test asserts — so it would pass unfixed. The existing comment explains what the bytes are but not that the choice is load-bearing, and that's the kind of thing a later cleanup removes.

[victorjulien]

Merged in #16141, thanks!