X1012 - NVMe drive "disappears" after 12.7 upgrade?

• D

  DigiK

  10/17/24, 1:16 AM

  I've been using an X1012 PoE + NVMe board for a few months now, and it has been very stable until this past week. 

  I updated from PiOS 12.6 to 12.7, and now it seems that my drive "disappears" entirely. Sometimes, it only takes 30 minutes. Sometimes a day or two. If I stay logged in over ssh, the RPi is still responsive to hitting enter, but even simple things like "ls" will show "command not found". Interestingly, "exit" works, and produces I/O error messages when it looks for logout scripts to process.

  If I cycle power, it boots right up. I'm not seeing anything interesting at all in the dmesg or application logs that point to a bad drive. And fsck checks out. I've even tried replacing the drive with a new one fresh out of the box, and the same thing happens. It's like it just... disappears. 

  Has anyone else had this happen?

  0 likes 0 likes Liked Like

  ---

  4
  Posts

  42
  Views

  Reply

  ---

• HH

  Harry Huang Staff member

  10/17/24, 11:51 AM

  PiOS 12.6 to 12.7

  What version is this?

  My current version are as follows:

  pi@raspberrypi:~ $ uname -a
  Linux raspberrypi 6.6.47+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.47-1+rpt1 (2024-09-02) aarch64 GNU/Linux

  0 likes 0 likes Liked Like

  ---

• D

  DigiK

  10/17/24, 7:26 PM

  PiOS/Debian 12.7

  Linux rpi3 6.6.31+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux

  After setting up an identical system, I'm beginning to wonder if maybe it's just a bad X1012. A different X1012 on a different Pi 5 board but with the same NVMe drive has now been running without problem for 24 hours.

  On the one that goes bad, the only thing it seems to log is the services noticing that the drive has disappeared:

  -- Boot 27365b12128d45038a5e37e48669f70a --
  Oct 16 16:52:48 rpi3 kernel: F2FS not supported on PAGE_SIZE(16384) != 4096
  Oct 16 16:52:48 rpi3 kernel: nvme nvme0: missing or invalid SUBNQN field.
  Oct 16 16:52:48 rpi3 kernel: genirq: irq_chip rp1_irq_chip did not update eff. affinity mask of irq 106
  Oct 16 16:52:48 rpi3 systemd-sysctl[313]: Couldn't write '1' to 'kernel/unprivileged_userns_clone', ignoring: No such >
  -- Boot 264a9548c11d471183f44fd62f89f2ff --
  Oct 16 17:17:01 rpi3 kernel: F2FS not supported on PAGE_SIZE(16384) != 4096
  Oct 16 17:17:01 rpi3 kernel: nvme nvme0: missing or invalid SUBNQN field.
  Oct 16 17:17:01 rpi3 kernel: genirq: irq_chip rp1_irq_chip did not update eff. affinity mask of irq 106
  Oct 16 17:17:01 rpi3 systemd-sysctl[319]: Couldn't write '1' to 'kernel/unprivileged_userns_clone', ignoring: No such >
  Oct 16 17:17:01 rpi3 systemd-journald[304]: File /var/log/journal/4dd8a9732af44c2981f848e3283d0878/system.journal corr>

  0 likes 0 likes Liked Like

  ---

• HH

  Harry Huang Staff member

  10/18/24, 12:03 PM

  I use AI to analyze your logs as follows:

   

  From the log information you provided, there are several key errors and warnings to note:

  1. F2FS not supported on PAGE_SIZE(16384) != 4096:
     This message indicates that the F2FS (Flash-Friendly File System) on your system does not support the current page size setting. F2FS, a log-structured file system designed for NAND flash memory, requires a page size of 4096 bytes. However, your system's page size is set to 16384 bytes, which is incompatible. Typically, this setting is not directly controlled by the user and may be determined by your kernel configuration or hardware platform. If you need to use F2FS, you may need to consider changing your kernel configuration or hardware.

  2. nvme nvme0: missing or invalid SUBNQN field:
     This message indicates that the SUBNQN (Subsystem Name Qualifier and Name) field for the NVMe device (such as an SSD) is missing or invalid. SUBNQN is used for NVMe over Fabrics (NVMeF) functionality. If your system or storage device does not support NVMeF, this warning may not affect functionality. However, if NVMeF functionality is required, you may need to check the SSD's firmware or configuration.

  3. genirq: irq_chip rp1_irq_chip did not update eff. affinity mask of irq 106:
     This message indicates that there may be an issue with some configurations of the interrupt handling chip, specifically that the interrupt affinity setting for IRQ 106 was not correctly updated. This is typically related to interrupt handling in hardware or the kernel. If the system is running stably, this warning may not require immediate attention. However, if issues related to interrupt handling are encountered, further investigation may be needed.

  4. Couldn't write '1' to 'kernel/unprivileged_userns_clone', ignoring: No such file or directory:
     This message indicates that systemd-sysctl attempted to modify the kernel parameter kernel/unprivileged_userns_clone but could not find the corresponding file or directory. This usually means that the current kernel version or configuration does not support this parameter. If the parameter is not needed, this warning can be ignored. If it is needed, you may need to upgrade the kernel or adjust the kernel configuration.

  In summary, these log messages relate to filesystem support, storage device configuration, interrupt handling, and kernel parameter settings. If the system is running stably and there are no obvious performance issues or missing functionality, some of these warnings can be temporarily ignored. However, if you encounter specific problems, such as inability to mount the filesystem, storage device performance issues, or system instability, you may need to further troubleshoot and resolve the issues based on the above information.

  ---

  0 likes 0 likes Liked Like

  ---