libdile_vt mmap failure caused by non-page-aligned framebuffer address on M3R_ATSC

[cleiversonjose-stack]

PicCap / hyperion-webos — libdile_vt mmap failure fixed on LG webOS

On an LG 43LM6300PSB running webOS 4.10.x, PicCap's libdile_vt_backend was unable to initialize video capture.

The relevant log was:

[DILE_VT] vfbs: 5; planes: 2
[DILE_VT] pixelFormat: 4; width: 320; height: 180; stride: 320
[DILE_VT] vfb[0][0] = 0x3fc00000
[DILE_VT] vfb[0][1] = 0x3fc0e100
[DILE_VT] mmap for vfb[0][1] failed!
libdile_vt_backend.so: init failure, code: -7

We found that the problem was caused by the physical framebuffer address passed directly as the offset to mmap().

Linux requires the mmap file offset to be page-aligned. On this TV the page size is 4096 bytes, but some framebuffer planes are returned at addresses such as:

0x3fc0e100

which is not page-aligned.

The workaround was to align the physical address down to the page boundary:

physical: 0x3fc0e100
aligned: 0x3fc0e000
offset: 0x100

Then mmap framebuffer_size + page_offset bytes and use:

buffer = mapped_base + page_offset;

After this change all 5 framebuffers / 2 planes mapped successfully:

[DILE_VT] vfb[0][1] phys=0x3fc0e100 aligned=0x3fc0e000 offset=0x100
[DILE_VT] vfb[0][1] mapped successfully
libdile_vt_backend.so: success
Video capture started
VIDEO: pixfmt: 5; 320x180
CONV VIDEO: pixfmt: 12; 320x180

Tested successfully with:

LG 43LM6300PSB
webOS 4.10.x / firmware 05.40.97
board M3R_ATSC
PicCap 0.5.3
HyperHDR 22.0.0
PlayStation 5 over HDMI
Netflix running directly on the TV

HDMI capture and internal-app capture are both working after the change.

I am sharing this mainly as a technical finding in case other M3R/webOS models have the same issue.