I'm able to reproduce the same behavior with Raspbian. The log output via Serial differs from time to time. The device hangs forever and needs a power cycle. Exactly the same as I see with Yocto.
Raspbian Example 1, it hangs forever after initiating and starting a reboot somewhere during shutdown of the system:
Raspbian Example 2, it hangs forever after initiating and starting a reboot somewhere during startup of the system:The log outputs above are captured via Serial and are the last messages captured before the system hangs forever.
Question:
Let's assume the issue is with my custom DSI Panel, how can I narrow this down? Adding some log output to the DSI driver to see that it hangs somewhere in there?
Is it even the case, that DSI performs init/uninit operations after the Systemd watchdog is already disabled and thus a hang means a hang forever without watchdog protection?
Raspbian Example 1, it hangs forever after initiating and starting a reboot somewhere during shutdown of the system:
Code:
[ 1.752563] usbcore: registered new interface driver usbhid[ 1.752571] usbhid: USB HID core driver[ 1.752647] bcm2835_vchiq fe00b840.mailbox: there is not valid maps for state default[ 1.755838] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available[ 1.756514] NET: Registered PF_PACKET protocol family[ 1.756566] Key type dns_resolver registered[ 1.769859] registered taskstats version 1[ 1.770034] Loading compiled-in X.509 certificates[ 1.775442] Key type .fscrypt registered[ 1.775473] Key type fscrypt-provisioning registered[ 1.779204] uart-pl011 fe201000.serial: there is not valid maps for state default[ 1.779441] uart-pl011 fe201000.serial: cts_event_workaround enabled[ 1.779621] fe201000.serial: ttyAMA1 at MMIO 0xfe201000 (irq = 36, base_baud = 0) is a PL011 rev3[ 1.779739] serial serial0: tty port ttyAMA1 registered[ 1.780841] bcm2835-aux-uart fe215040.serial: there is not valid maps for state default[ 1.781094] printk: console [ttyS0] disabled[ 1.781493] fe215040.serial: ttyS0 at MMIO 0xfe215040 (irq = 37, base_baud = 62500000) is a 16550[ 1.781531] printk: console [ttyS0] enabled[ 3.363168] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer[ 3.369759] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver[ 3.377517] mmc-bcm2835 fe300000.mmcnr: mmc_debug:0 mmc_debug2:0[ 3.383625] mmc-bcm2835 fe300000.mmcnr: DMA channel allocated[ 3.414316] of_cfs_init[ 3.417153] of_cfs_init: OK[ 3.420245] clk: Disabling unused clocks[ 3.456462] mmc0: SDHCI controller on fe340000.mmc [fe340000.mmc] using ADMA[ 3.468090] Freeing unused kernel memory: 1920K[ 3.472809] Run /init as init process[ 3.502022] mmc1: new high speed SDIO card at address 0001[ 3.524360] mmc0: new DDR MMC card at address 0001[ 3.529915] mmcblk0: mmc0:0001 AJTD4R 14.6 GiB[ 3.535981] mmcblk0: p1 p2[ 3.539218] mmcblk0: mmc0:0001 AJTD4R 14.6 GiB[ 3.544078] mmcblk0boot0: mmc0:0001 AJTD4R 4.00 MiB[ 3.550554] mmcblk0boot1: mmc0:0001 AJTD4R 4.00 MiB[ 3.556850] mmcblk0rpmb: mmc0:0001 AJTD4R 4.00 MiB, chardev (243:0)[ 3.960275] EXT4-fs (mmcblk0p2): mounted filesystem a36be96c-66be-4487-a7a6-0481bca99d89 ro with ordered data mode. Quota mode: none.[ 4.259243] systemd[1]: System time before build time, advancing clock.[ 4.353992] NET: Registered PF_INET6 protocol family[ 4.359747] Segment Routing with IPv6[ 4.363492] In-situ OAM (IOAM) with IPv6[ 4.397925] systemd[1]: systemd 252.33-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)[ 4.431147] systemd[1]: Detected architecture arm64.[ 4.441263] systemd[1]: Hostname set to <cm4-argos2>.[ 4.839438] systemd[1]: Queued start job for default target graphical.target.[ 4.868849] systemd[1]: Created slice system-getty.slice - Slice /system/getty.[ 4.877810] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.[ 4.887258] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.[ 4.897169] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck.[ 4.906636] systemd[1]: Created slice user.slice - User and Session Slice.[ 4.913907] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.[ 4.924982] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.[ 4.937865] systemd[1]: Expecting device dev-disk-by\x2dpartuuid-5409ea12\x2d01.device - /dev/disk/by-partuuid/5409ea12-01...[ 4.949458] systemd[1]: Expecting device dev-dri-card0.device - /dev/dri/card0...[ 4.957150] systemd[1]: Expecting device dev-dri-renderD128.device - /dev/dri/renderD128...[ 4.965724] systemd[1]: Expecting device dev-ttyS0.device - /dev/ttyS0...[ 4.972772] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.[ 4.981935] systemd[1]: Reached target nss-user-lookup.target - User and Group Name Lookups.[ 4.990654] systemd[1]: Reached target slices.target - Slice Units.[ 4.997159] systemd[1]: Reached target swap.target - Swaps.[ 5.002949] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.[ 5.011845] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.[ 5.020938] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.[ 5.030459] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.[ 5.039301] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).[ 5.048736] systemd[1]: Listening on systemd-journald.socket - Journal Socket.[ 5.056893] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.[ 5.065487] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.[ 5.073943] systemd[1]: dev-hugepages.mount - Huge Pages File System was skipped because of an unmet condition check (ConditionPathExists=/sys/kernel/mm/hugepages).[ 5.110542] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...[ 5.122907] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...[ 5.133103] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...[ 5.141676] systemd[1]: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).[ 5.160369] systemd[1]: Starting fake-hwclock.service - Restore / save the current clock...[ 5.171531] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...[ 5.183236] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...[ 5.195450] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...[ 5.206996] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...[ 5.218178] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...[ 5.229140] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...[ 5.239419] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com[ 5.241041] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...[ 5.258675] systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...[ 5.267181] systemd[1]: systemd-fsck-root.service - File System Check on Root Device was skipped because of an unmet condition check (ConditionPathExists=!/run/initramfs/fsck-root).[ 5.287899] fuse: init (API version 7.39)[ 5.288418] systemd[1]: Starting systemd-journald.service - Journal Service...[ 5.303630] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...[ 5.315296] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...[ 5.328410] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...[ 5.341564] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.[ 5.351980] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.[ 5.360932] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.[ 5.369041] i2c_dev: i2c /dev entries driver[ 5.371377] systemd[1]: Finished fake-hwclock.service - Restore / save the current clock.[ 5.387594] systemd[1]: Finished keyboard-setup.service - Set the console keyboard layout.[ 5.394967] EXT4-fs (mmcblk0p2): re-mounted a36be96c-66be-4487-a7a6-0481bca99d89 r/w. Quota mode: none.[ 5.412707] systemd[1]: Started systemd-journald.service - Journal Service.[ 5.501731] systemd-journald[254]: Received client request to flush runtime journal.[ 5.516594] systemd-journald[254]: File /var/log/journal/4ccd6ac23fde41c18fb408c0343953b5/system.journal corrupted or uncleanly shut down, renaming and replacing.** 689359 printk messages dropped **[ 1677.160083] Disabling IRQ #42Raspbian Example 2, it hangs forever after initiating and starting a reboot somewhere during startup of the system:
Code:
[ 1.748011] sdhci-pltfm: SDHCI platform and OF driver helper[ 1.752155] ledtrig-cpu: registered to indicate activity on CPUs[ 1.752319] hid: raw HID events driver (C) Jiri Kosina[ 1.752369] usbcore: registered new interface driver usbhid[ 1.752376] usbhid: USB HID core driver[ 1.752439] bcm2835_vchiq fe00b840.mailbox: there is not valid maps for state default[ 1.755604] hw perfevents: enabled with armv8_cortex_a72 PMU driver, 7 counters available[ 1.756252] NET: Registered PF_PACKET protocol family[ 1.756310] Key type dns_resolver registered[ 1.768998] registered taskstats version 1[ 1.769173] Loading compiled-in X.509 certificates[ 1.774073] Key type .fscrypt registered[ 1.774100] Key type fscrypt-provisioning registered[ 1.777824] uart-pl011 fe201000.serial: there is not valid maps for state default[ 1.778056] uart-pl011 fe201000.serial: cts_event_workaround enabled[ 1.779136] fe201000.serial: ttyAMA1 at MMIO 0xfe201000 (irq = 36, base_baud = 0) is a PL011 rev3[ 1.779266] serial serial0: tty port ttyAMA1 registered[ 1.780347] bcm2835-aux-uart fe215040.serial: there is not valid maps for state default[ 1.780768] printk: console [ttyS0] disabled[ 1.780976] fe215040.serial: ttyS0 at MMIO 0xfe215040 (irq = 37, base_baud = 62500000) is a 16550[ 1.781012] printk: console [ttyS0] enabled[ 3.362261] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 watchdog timer[ 3.368843] bcm2835-power bcm2835-power: Broadcom BCM2835 power domains driver[ 3.376572] mmc-bcm2835 fe300000.mmcnr: mmc_debug:0 mmc_debug2:0[ 3.382675] mmc-bcm2835 fe300000.mmcnr: DMA channel allocated[ 3.414257] of_cfs_init[ 3.416859] of_cfs_init: OK[ 3.419932] clk: Disabling unused clocks[ 3.454224] mmc0: SDHCI controller on fe340000.mmc [fe340000.mmc] using ADMA[ 3.465552] Freeing unused kernel memory: 1920K[ 3.470307] Run /init as init process[ 3.500517] mmc1: new high speed SDIO card at address 0001[ 3.522124] mmc0: new DDR MMC card at address 0001[ 3.527699] mmcblk0: mmc0:0001 AJTD4R 14.6 GiB[ 3.534490] mmcblk0: p1 p2[ 3.537717] mmcblk0: mmc0:0001 AJTD4R 14.6 GiB[ 3.542618] mmcblk0boot0: mmc0:0001 AJTD4R 4.00 MiB[ 3.548986] mmcblk0boot1: mmc0:0001 AJTD4R 4.00 MiB[ 3.555139] mmcblk0rpmb: mmc0:0001 AJTD4R 4.00 MiB, chardev (243:0)[ 4.008219] EXT4-fs (mmcblk0p2): mounted filesystem a36be96c-66be-4487-a7a6-0481bca99d89 ro with ordered data mode. Quota mode: none.[ 4.306923] systemd[1]: System time before build time, advancing clock.[ 4.401343] NET: Registered PF_INET6 protocol family[ 4.407144] Segment Routing with IPv6[ 4.410905] In-situ OAM (IOAM) with IPv6[ 4.445693] systemd[1]: systemd 252.33-1~deb12u1 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)[ 4.478918] systemd[1]: Detected architecture arm64.[ 4.489037] systemd[1]: Hostname set to <cm4-argos2>.[ 4.896884] systemd[1]: Queued start job for default target graphical.target.[ 4.928807] systemd[1]: Created slice system-getty.slice - Slice /system/getty.[ 4.937959] systemd[1]: Created slice system-modprobe.slice - Slice /system/modprobe.[ 4.947096] systemd[1]: Created slice system-serial\x2dgetty.slice - Slice /system/serial-getty.[ 4.956797] systemd[1]: Created slice system-systemd\x2dfsck.slice - Slice /system/systemd-fsck.[ 4.966208] systemd[1]: Created slice user.slice - User and Session Slice.[ 4.973463] systemd[1]: Started systemd-ask-password-wall.path - Forward Password Requests to Wall Directory Watch.[ 4.984552] systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.[ 4.997438] systemd[1]: Expecting device dev-disk-by\x2dpartuuid-5409ea12\x2d01.device - /dev/disk/by-partuuid/5409ea12-01...[ 5.009028] systemd[1]: Expecting device dev-dri-card0.device - /dev/dri/card0...[ 5.016740] systemd[1]: Expecting device dev-dri-renderD128.device - /dev/dri/renderD128...[ 5.025318] systemd[1]: Expecting device dev-ttyS0.device - /dev/ttyS0...[ 5.032360] systemd[1]: Reached target integritysetup.target - Local Integrity Protected Volumes.[ 5.041545] systemd[1]: Reached target nss-user-lookup.target - User and Group Name Lookups.[ 5.050259] systemd[1]: Reached target slices.target - Slice Units.[ 5.056768] systemd[1]: Reached target swap.target - Swaps.[ 5.062569] systemd[1]: Reached target veritysetup.target - Local Verity Protected Volumes.[ 5.071487] systemd[1]: Listening on systemd-fsckd.socket - fsck to fsckd communication Socket.[ 5.080593] systemd[1]: Listening on systemd-initctl.socket - initctl Compatibility Named Pipe.[ 5.090085] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Socket.[ 5.098901] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket (/dev/log).[ 5.108340] systemd[1]: Listening on systemd-journald.socket - Journal Socket.[ 5.116523] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket.[ 5.125156] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.[ 5.133640] systemd[1]: dev-hugepages.mount - Huge Pages File System was skipped because of an unmet condition check (ConditionPathExists=/sys/kernel/mm/hugepages).[ 5.166467] systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...[ 5.177802] systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...[ 5.188294] systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...[ 5.196879] systemd[1]: auth-rpcgss-module.service - Kernel Module supporting RPCSEC_GSS was skipped because of an unmet condition check (ConditionPathExists=/etc/krb5.keytab).[ 5.215530] systemd[1]: Starting fake-hwclock.service - Restore / save the current clock...[ 5.226664] systemd[1]: Starting keyboard-setup.service - Set the console keyboard layout...[ 5.238352] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Device Nodes...[ 5.250583] systemd[1]: Starting modprobe@configfs.service - Load Kernel Module configfs...[ 5.262113] systemd[1]: Starting modprobe@dm_mod.service - Load Kernel Module dm_mod...[ 5.273271] systemd[1]: Starting modprobe@drm.service - Load Kernel Module drm...[ 5.284023] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...[ 5.294133] device-mapper: ioctl: 4.48.0-ioctl (2023-03-01) initialised: dm-devel@redhat.com[ 5.295789] systemd[1]: Starting modprobe@fuse.service - Load Kernel Module fuse...[ 5.313448] systemd[1]: Starting modprobe@loop.service - Load Kernel Module loop...[ 5.322175] systemd[1]: systemd-fsck-root.service - File System Check on Root Device was skipped because of an unmet condition check (ConditionPathExists=!/run/initramfs/fsck-root).[ 5.322483] fuse: init (API version 7.39)[ 5.345265] systemd[1]: Starting systemd-journald.service - Journal Service...[ 5.361815] systemd[1]: Starting systemd-modules-load.service - Load Kernel Modules...[ 5.373073] systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...[ 5.385412] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices...[ 5.399454] systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.[ 5.408529] systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.[ 5.418182] systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.[ 5.426317] i2c_dev: i2c /dev entries driver[ 5.434364] systemd[1]: Finished fake-hwclock.service - Restore / save the current clock.[ 5.442322] EXT4-fs (mmcblk0p2): re-mounted a36be96c-66be-4487-a7a6-0481bca99d89 r/w. Quota mode: none.[ 5.455165] systemd[1]: Started systemd-journald.service - Journal Service.[ 5.546390] systemd-journald[254]: Received client request to flush runtime journal.[ 5.562133] systemd-journald[254]: File /var/log/journal/4ccd6ac23fde41c18fb408c0343953b5/system.journal corrupted or uncleanly shut down, renaming and replacing.Question:
Let's assume the issue is with my custom DSI Panel, how can I narrow this down? Adding some log output to the DSI driver to see that it hangs somewhere in there?
Is it even the case, that DSI performs init/uninit operations after the Systemd watchdog is already disabled and thus a hang means a hang forever without watchdog protection?
Statistics: Posted by batwing — Fri Mar 14, 2025 9:27 am