Fix motion compensation for bit depths 13 to 16

[felixbuenemann]

Follow-up to #538. Intra-only 16-bit streams decode correctly since that fix, but any inter picture above 12 bit was still wrong from the first B/P frame on (essentially every sample off, errors up to the full range). This makes 13–16 bit work with motion compensation too, including weighted prediction and mixed luma/chroma bit depths.

Cause. Two things in the sample interpolation break at 13 bit:

• put_qpel_fallback and put_epel_hv_fallback used shift1 = BitDepth-8. The Range Extensions changed (8.5.3.3.3.2)/(8.5.3.3.3.3) to shift1 = Min(4, BitDepth-8); the v1 formula agrees only up to 12 bit.
• With shift1 capped at 4, the filter intermediates and predSamplesLX need up to 20 bits, but the whole MC path was int16_t (mcbuffer, predSamplesL/C, the full-sample ref << shift3 copies, the weighted-prediction inputs).

Fix. Rather than widening everything to 32 bit (which would cost bandwidth for 9–12 bit and change the SIMD signatures), the intermediate type is a template parameter of the fallback functions, chosen once per PB in generate_inter_prediction_samples(): int16_t through the acceleration table exactly as before up to MC_MAX_BIT_DEPTH_INT16 (12), int32_t above that. The int32_t variants are reached through int32_t overloads of the existing acceleration_functions wrappers, which keep the per-plane 8/16-bit sample split (luma and chroma depths are independent; an 8-bit plane can sit next to a 14-bit one). No table entries are added for them — there is no SIMD for any >8-bit MC on any platform — and the 8–12-bit path compiles to the same code as on master. The 96 KB int32_t prediction-sample set lives in per-thread storage so the stack requirement of the common path does not grow. Also fixes the mid-grey error-path fill, which assumed shift3 == 14-bd.

Verification against HM 18.0 built with RExt__HIGH_BIT_DEPTH_SUPPORT (HM's MCTF pre-filter disabled — it corrupts 16-bit sources — and ExtendedPrecision off, as libde265 does not implement it). All byte-exact vs the HM decoder unless noted:

640×360, 4:2:0	8	9	10	11	12	13	14	15	16
intra	✓	✓	✓	✓	✓	✓	✓	✓	✓
random access (hier. B, GOP 16)	✓	✓	✓	✓	✓	✓ (was wrong)	✓ (was wrong)	✓ (was wrong)	✓ (was wrong)

Also bit-exact: low delay with weighted bi-prediction at 8/13/16 bit, with and without high_precision_offsets_enabled_flag; 16-bit random access with implicit/explicit RDPCM, persistent Rice adaptation and transform-skip context on; mixed luma/chroma depths 8/14, 14/8, 10/13 (master was wrong in whichever plane was >12 bit); --noaccel; testdata/girlshy.h265 identical to master.

4K (Netflix ToddlerFountain 4096×2160, first 43 frames, originally 10-Bit, low delay B, WPP):

bit depth	8	10	12	13	16
this PR vs HM	exact	exact	exact	exact	exact
master vs HM	exact	exact	exact	wrong	wrong
fps master → PR, 1 thread	10.0 → 10.1	10.3 → 10.2	10.9 → 10.9	11.0 → 11.0	(n/a)*
fps master → PR, 8 threads	62.6 → 62.7	60.0 → 58.3	63.4 → 62.1	59.8 → 59.7	(n/a)*

All within ±3 % run-to-run noise; 13 bit decodes at the same speed as 12 bit. *master decodes garbage at 16 bit, so its timing is not a baseline; on this PR 16 bit runs at 10.3 / 51.6 fps.

Unrelated things noticed on the way, left for separate issues: transform_skip_rotation_enabled_flag gives small chroma-only errors in inter pictures even at 8 bit, and dec265 -c only reports a picture-hash mismatch on the last picture (read_slice_NAL() discards decode_some()'s error).

The code in this PR was created with the help of an LLM.

Before/After:

[felixbuenemann]

@farindk The code fixes the problem, but it feels too large a change to me. Do you have an idea how we could shrink it?

[farindk]

Thank you. I have also directed my AI to fix this, ending up with a similar patch with the same root causes. We have compared the two implementations and distilled an implementation combining ideas from both. It has a few advantages to the original PR:

• the intermediate type is chosen per colour plane instead of once per PB, so an 8-bit plane next to a >12-bit plane keeps its SIMD kernels (mixed 8/14 was decoding luma through the scalar path here);
• the >12-bit kernels are reached through acceleration table entries rather than by calling the fallback functions from acceleration.h, and the int32_t qpel entry is a single generic kernel, which keeps the code-size growth at about 21 KB;
• the prediction sample buffers stay on the stack (16/32 KB per plane) instead of a thread_local heap allocation.

Both implementations are bit-exact against HM and produce identical output on every high-bit-depth test stream. Thus, I have chosen the combined implementation and credited you as a Co-author.

If you have some real 16bit video streams at hand, it would be good if you can confirm that everything works.

[felixbuenemann]

@farindk Thanks! I tested Netflix_ToddlerFountain from https://media.xiph.org/video/derf/ bit-shifted from 10-Bit to 16-Bit and encoded with HM with TemporalFilter disabled (current master HM temporal filter is broken at 16-Bit) and it decodes fine with libde265 master@b85929dc.

[farindk]

Thanks for cross-checking!