Camera board • Re: Latency builds up when decoding H264 with V4L, not with LAVC

Can you inform on what versions of EEPROM, firmware, kernel?

It has persisted across multiple kernel versions. Currently I'm running "6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux" (64-bit kernel with a 64-bit userland, installed a week ago). I've tried recompiling the kernel, I've tried out running a 32-bit kernel on a 64-bit userland.

I don't know how to determine relevant firmware or EEPROM versions. Image may be NSFW.
Clik here to view. Both of my Raspberries originate from the same batch, recently bought from BerryBase.

I am not sure if they are included in the 'UDP pipe' or not, but I have been using format mpegts between computers and also between process from libcamera to ffmpeg. If not, out-of-sync sometimes. The HW codec runs on vc4, not cortex-a72, so maybe has slightly other timing, that is my guess.

I suspect (don't know for sure) that the only timestamps transmitted are my own - they are not inside the video stream, but outside it. According to this source, the H264 stream generated on a Pi typically lacks PTS (presentation timestamps).

I have tried initialising the encoder both by giving it knowledge of the intended frame rate, and by denying it knowledge of the frame rate. Sadly, the result is the same - I can't make it better (or even worse) by doing that. I have also tried raising and lowering the bitrate: no effect. Encoder initialisation goes like:

v4l2_control v4l_ctl = {};

// Set video data rate in mbit/s, megabits per second.
v4l_ctl.id = V4L2_CID_MPEG_VIDEO_BITRATE;
v4l_ctl.value = 2e6;
if (ioctl(v4l_fd, VIDIOC_S_CTRL, &v4l_ctl) < 0)
elog("Failed to set H264 bitrate.\n");

v4l_ctl.id = V4L2_CID_MPEG_VIDEO_H264_PROFILE;
v4l_ctl.value = V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE; // allegedly more robust, faster methods
// v4l_ctl.value = V4L2_MPEG_VIDEO_H264_PROFILE_MAIN;
// v4l_ctl.value = V4L2_MPEG_VIDEO_H264_PROFILE_HIGH; // smaller size, but at high computational cost
if (ioctl(v4l_fd, VIDIOC_S_CTRL, &v4l_ctl) < 0)
elog("Failed to set H264 profile.\n");

// Do not force a level?
v4l_ctl.id = V4L2_CID_MPEG_VIDEO_H264_LEVEL;
v4l_ctl.value = V4L2_MPEG_VIDEO_H264_LEVEL_4_1;
if (ioctl(v4l_fd, VIDIOC_S_CTRL, &v4l_ctl) < 0)
elog("Failed to set H264 level.\n");

v4l_ctl.id = V4L2_CID_MPEG_VIDEO_H264_I_PERIOD;
v4l_ctl.value = 10; // how often to send i-frames
if (ioctl(v4l_fd, VIDIOC_S_CTRL, &v4l_ctl) < 0)
elog("Failed to set H264 i-frame interval.\n");

v4l_ctl.id = V4L2_CID_MPEG_VIDEO_REPEAT_SEQ_HEADER;
v4l_ctl.value = 1;
if (ioctl(v4l_fd, VIDIOC_S_CTRL, &v4l_ctl) < 0)
elog("Failed to set H264 inline headers.\n");

Also, asking the decoder for frames more agressively doesn't help either - I tried inserting an extra request for a decoded frame once per 10 frames, or once per 5 frames, to no avail. If it has them, it's not giving them. Image may be NSFW.
Clik here to view.

Statistics: Posted by diastrikos — Sun Sep 01, 2024 11:57 pm

---