https://github.com/lvmteam/lvm2/blob/master/doc/release-notes/2.02.178
https://www.linuxquestions.org/questions/slackware-14/lvm2-upgrade-caused-lvm-partition-to-not-be-recognized-4175632402/
 
这个坑浪费了我 2 天的业余时间，起因就是我在准备给 Linux 上游提交补丁时候顺带升级了下系统，然后神奇般地，新编译的内核起不来了，点都点不亮！
 
搞了一条串口转 USB 线之后，发现问题在于 / 分区挂不上（点不亮的原因是 radeon 虽然编译进内核了，但忘记一同嵌入固件），进入 dracut rescue shell 之后发现日志里体现以下内容，后来在一个正常内核下也复现了：

$ sudo lvm pvdisplay

  Error reading device /dev/ram0 at 0 length 4096.

  Error reading device /dev/sda at 0 length 512.

  Error reading device /dev/sda at 0 length 4096.

  Error reading device /dev/vg0/rootvol at 0 length 512.

  Error reading device /dev/vg0/rootvol at 0 length 4096.

  Error reading device /dev/ram1 at 0 length 4096.

  Error reading device /dev/sda1 at 0 length 4096.

  Error reading device /dev/vg0/varvol at 0 length 512.

  Error reading device /dev/vg0/varvol at 0 length 4096.

  Error reading device /dev/ram2 at 0 length 4096.

  Error reading device /dev/sda2 at 0 length 4096.

  Error reading device /dev/vg0/portagevol at 0 length 512.

  Error reading device /dev/vg0/portagevol at 0 length 4096.

  Error reading device /dev/ram3 at 0 length 4096.

  Error reading device /dev/sda3 at 0 length 4096.

  Error reading device /dev/vg0/homevol at 0 length 512.

  Error reading device /dev/vg0/homevol at 0 length 4096.

  Error reading device /dev/ram4 at 0 length 4096.

  Error reading device /dev/sda4 at 0 length 4096.

  Error reading device /dev/ram5 at 0 length 4096.

  Error reading device /dev/ram6 at 0 length 4096.

  Error reading device /dev/ram7 at 0 length 4096.

  Error reading device /dev/ram8 at 0 length 4096.

  Error reading device /dev/ram9 at 0 length 4096.

  Error reading device /dev/ram10 at 0 length 4096.

  Error reading device /dev/ram11 at 0 length 4096.

  Error reading device /dev/ram12 at 0 length 4096.

  Error reading device /dev/ram13 at 0 length 4096.

  Error reading device /dev/ram14 at 0 length 4096.

  Error reading device /dev/ram15 at 0 length 4096.

拉出 strace，节录关键部分：

io_submit(0x7ffff698c000, 1, [{aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=0, aio_buf=0x6, aio_nbytes=93825234075648, aio_offset=0}]) = -1 EINVAL (Invalid argument)

write(2, "  ", 2  )                       = 2

write(2, "Error reading device /dev/sda3 a"..., 48Error reading device /dev/sda3 at 0 length 4096.) = 48

write(2, "\n", 1

)                       = 1

 
可以看到这个 struct iocb 很有问题，像是 ABI 错乱，参数顺序问题之类的。
 
把没问题的内核的 initramfs 解包，进去调试：

mkdir initramfs-test

cd initramfs-test

# 我知道这个 img 是 gzip 压缩的

sudo zcat /boot/initramfs-xxx.img | sudo cpio -i

sudo mount --rbind /dev ./dev

sudo mount --rbind /proc ./proc

sudo mount --rbind /sys ./sys

sudo cp /usr/bin/strace ./usr/bin/

cd ..

sudo chroot ./initramfs-test /bin/sh



# 在 chroot 里

strace lvm pvdisplay

 
拉出 strace，节录关键部分：

lseek(4, 0, SEEK_SET)                   = 0

read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096

lseek(4, 4096, SEEK_SET)                = 4096

read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096

 
可见并没有异步 I/O。。简单搜索一下 lvm2 libaio 就找到文首的两个链接，降级 LVM2，重做 initramfs，问题解决。所有其他变更（我的几个整理好&整理中的补丁、有问题的 R9 285 显卡、加载失败的几个玩意、eu-strip 报 ill-formed 的内核模块）都是烟雾弹，浪费了大量时间，非常生气。
 
虽然没空折腾具体的 ABI 问题是在哪一层解决的，不过至少我们需要注意 MIPS 平台上不要贸然升级 LVM2。。。否则你就会挂不上分区，进不了系统！相关的小伙伴可以行动起来了。。。