Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kmod-zfs-2.3.1 pre-built kernel module fails to load on kernel-5.14.0-503.33.1.el9_5.x86_64 #17165

Open
stuartthebruce opened this issue Mar 21, 2025 · 6 comments
Open

kmod-zfs-2.3.1 pre-built kernel module fails to load on kernel-5.14.0-503.33.1.el9_5.x86_64 #17165

stuartthebruce opened this issue Mar 21, 2025 · 6 comments
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@stuartthebruce
Copy link

System information

Type Version/Name
Distribution Name Rocky Linux
Distribution Version 9.5
Kernel Version 5.14.0-503.33.1.el9_5
Architecture x86_64
OpenZFS Version 2.3.1

Describe the problem you're observing

kernel-5.14.0-503.33.1.el9_5.x86_64 fails to load kmod-zfs-2.3.1-1.el9.x86_64

Describe how to reproduce the problem

Update kernel and attempt to load pre-built binary ZFS kernel module.

Include any warning/errors/backtraces from the system logs

Mar 21 12:22:16 ldas-pcdev15 kernel: module spl: .gnu.linkonce.this_module section size must match the kernel's built struct module size at run time

which may be related to an earlier message when upgrading the kernel,

2025-03-21T12:14:38-0700 INFO depmod: WARNING: /lib/modules/5.14.0-503.33.1.el9_5.x86_64/weak-updates/zfs/zfs.ko needs unknown symbol taskq_create_synced

Note, the same system loads ZFS with the previous kernel-5.14.0-503.31.1.el9_5.x86_64
@stuartthebruce stuartthebruce added the Type: Defect Incorrect behavior (e.g. crash, hang) label Mar 21, 2025
@tonyhutter
Copy link
Contributor

That's weird. It works for me on Almalinux 9:

$ uname -a
Linux centos9 5.14.0-503.33.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 20 03:39:23 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux

$ dmesg | grep ZFS
[  178.545638] ZFS: Loaded module v2.3.1-1, ZFS pool version 5000, ZFS filesystem version 5

@stuartthebruce
Copy link
Author

Good to know. I will try updating a second RL9 system to see what happens.

@stuartthebruce
Copy link
Author

stuartthebruce commented Mar 25, 2025 •
edited
Loading

That's weird. It works for me on Almalinux 9:

Nothing like proof by example. I went back to my problematic RL9 system and after completely uninstalling ZFS, rebooting, and reinstalling kmod-zfs it is also now working,

[root@ldas-pcdev15 ~]# uname -r
5.14.0-503.33.1.el9_5.x86_64

[root@ldas-pcdev15 ~]# rpm -q kmod-zfs
kmod-zfs-2.3.1-1.el9.x86_64
[root@ldas-pcdev15 ~]#

Sorry for the noise.

@stuartthebruce
Copy link
Author

FYI, I saw this issue on a few other systems being upgraded and I think something is not quite right with weak-updates. When I saw this problem it was on systems with several symbolic links for zfs.ko, e.g.,

[root@kafkaifo6 ~]# find /lib/modules -name "*zfs*" -ls
  1151941   1044 -rw-r--r--   1 root     root      1066364 Mar  6  2024 /lib/modules/5.14.0-362.18.1.el9_3.0.1.x86_64/extra/zfs.ko.xz
 36533317   1048 -rw-r--r--   1 root     root      1069608 Mar 19  2024 /lib/modules/5.14.0-362.24.1.el9_3.x86_64/extra/zfs.ko.xz
 51295365   1044 -rw-r--r--   1 root     root      1069048 Apr  9  2024 /lib/modules/5.14.0-362.24.1.el9_3.0.1.x86_64/extra/zfs.ko.xz
    54329   1548 -rw-r--r--   1 root     root      1583252 Jun 25  2024 /lib/modules/5.14.0-427.22.1.el9_4.x86_64/extra/zfs.ko.xz
  1186706   1548 -rw-r--r--   1 root     root      1581088 Jul 16  2024 /lib/modules/5.14.0-427.24.1.el9_4.x86_64/extra/zfs.ko.xz
    14408   1556 -rw-r--r--   1 root     root      1591212 Jul 30  2024 /lib/modules/5.14.0-427.26.1.el9_4.x86_64/extra/zfs.ko.xz
 17551506   1560 -rw-r--r--   1 root     root      1594424 Aug  6  2024 /lib/modules/5.14.0-427.28.1.el9_4.x86_64/extra/zfs.ko.xz
 17066243   1552 -rw-r--r--   1 root     root      1585908 Aug 20  2024 /lib/modules/5.14.0-427.31.1.el9_4.x86_64/extra/zfs.ko.xz
 33686364      0 drwxr-xr-x   2 root     root           34 Mar 24 21:42 /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs
 33686366  11908 -rw-r--r--   1 root     root     12191472 Mar 10 10:46 /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko
     9245      0 drwxr-xr-x   2 root     root           34 Mar 24 21:42 /lib/modules/5.14.0-503.29.1.el9_5.x86_64/weak-updates/zfs
     9871      0 lrwxrwxrwx   1 root     root           58 Mar 24 21:42 /lib/modules/5.14.0-503.29.1.el9_5.x86_64/weak-updates/zfs/zfs.ko -> /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko
 50332444      0 lrwxrwxrwx   1 root     root           57 Mar 24 21:42 /lib/modules/5.14.0-503.29.1.el9_5.x86_64/weak-updates/zfs.ko.xz -> /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs.ko.xz
 16778313      0 drwxr-xr-x   2 root     root           34 Mar 24 21:42 /lib/modules/5.14.0-503.31.1.el9_5.x86_64/weak-updates/zfs
 16778320      0 lrwxrwxrwx   1 root     root           58 Mar 24 21:42 /lib/modules/5.14.0-503.31.1.el9_5.x86_64/weak-updates/zfs/zfs.ko -> /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko
   676280      0 lrwxrwxrwx   1 root     root           57 Mar 24 21:42 /lib/modules/5.14.0-503.31.1.el9_5.x86_64/weak-updates/zfs.ko.xz -> /lib/modules/5.14.0-503.29.1.el9_5.x86_64/extra/zfs.ko.xz
 33686877      0 drwxr-xr-x   2 root     root           34 Mar 24 21:42 /lib/modules/5.14.0-503.33.1.el9_5.x86_64/weak-updates/zfs
 33686881      0 lrwxrwxrwx   1 root     root           58 Mar 24 21:42 /lib/modules/5.14.0-503.33.1.el9_5.x86_64/weak-updates/zfs/zfs.ko -> /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko

In such cases it was not sufficient to simply re-install the kmod-zfs package. However, it was sufficient to uninstall and reinstall, which resulted in fewer weak-updates symbolic links.

@stuartthebruce
Copy link
Author

I saw this problem again upgrading a ZFS 2.3.1 system from kernel-5.14.0-503.33.1.el9_5.x86_64 to kernel-5.14.0-503.34.1.el9_5.x86_64,

[Thu Apr  3 15:03:06 2025] module spl: .gnu.linkonce.this_module section size must match the kernel's built struct module size at run time

and worked around it again by uninstalling and re-installing kmod-zfs-2.3.1-1.el9.x86_64

@stuartthebruce stuartthebruce reopened this Apr 3, 2025
@stuartthebruce
Copy link
Author

This test system had the same problem patching from kernel-5.14.0-503.34.1.el9_5.x86_64 to kernel-5.14.0-503.34.1.el9_5.x86_64. It really looks like something is problematic with weak-updates,

[root@ldas-pcdev15 ~]# find /lib/modules -name "*zfs*" -ls
 12783149   1560 -rw-r--r--   1 root     root      1594068 May  2  2024 /lib/modules/5.14.0-362.24.1.el9_3.0.1.x86_64/extra/zfs.ko.xz
  4194457   1548 -rw-r--r--   1 root     root      1582684 May  9  2024 /lib/modules/5.14.0-427.16.1.el9_4.x86_64/extra/zfs.ko.xz
 37930849   1560 -rw-r--r--   1 root     root      1593956 May 29  2024 /lib/modules/5.14.0-427.18.1.el9_4.x86_64/extra/zfs.ko.xz
   636005   1552 -rw-r--r--   1 root     root      1586468 Jun 10  2024 /lib/modules/5.14.0-427.20.1.el9_4.x86_64/extra/zfs.ko.xz
  9610426   1552 -rw-r--r--   1 root     root      1585236 Jun 19  2024 /lib/modules/5.14.0-427.20.1.el9_4.0.1.x86_64/extra/zfs.ko.xz
 33067800   1552 -rw-r--r--   1 root     root      1589084 Jun 20  2024 /lib/modules/5.14.0-427.22.1.el9_4.x86_64/extra/zfs.ko.xz
 12945925      0 drwxr-xr-x   2 root     root           34 Apr  4 16:19 /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs
 12945931  11908 -rw-r--r--   1 root     root     12191472 Mar 10 10:46 /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko
  9505288      0 drwxr-xr-x   2 root     root           34 Apr  4 16:19 /lib/modules/5.14.0-503.33.1.el9_5.x86_64/weak-updates/zfs
  9505292      0 lrwxrwxrwx   1 root     root           58 Apr  4 16:19 /lib/modules/5.14.0-503.33.1.el9_5.x86_64/weak-updates/zfs/zfs.ko -> /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko
   755035      0 drwxr-xr-x   2 root     root           34 Apr  4 16:19 /lib/modules/5.14.0-503.34.1.el9_5.x86_64/weak-updates/zfs
   755048      0 lrwxrwxrwx   1 root     root           58 Apr  4 16:19 /lib/modules/5.14.0-503.34.1.el9_5.x86_64/weak-updates/zfs/zfs.ko -> /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko
 38313921      0 drwxr-xr-x   2 root     root           34 Apr  4 16:19 /lib/modules/5.14.0-503.35.1.el9_5.x86_64/weak-updates/zfs
 38254387      0 lrwxrwxrwx   1 root     root           58 Apr  4 16:19 /lib/modules/5.14.0-503.35.1.el9_5.x86_64/weak-updates/zfs/zfs.ko -> /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/zfs/zfs.ko

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tonyhutter @stuartthebruce