Skip to content

zfs system hang without heavyload #17161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Popyllol opened this issue Mar 20, 2025 · 1 comment
Open

zfs system hang without heavyload #17161

Popyllol opened this issue Mar 20, 2025 · 1 comment
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@Popyllol
Copy link

System information

Type Version/Name
Distribution Name Ubuntu
Distribution Version 22.04.5 LTS
Kernel Version 5.15.0-135-generic
Architecture x86_64
OpenZFS Version zfs-2.1.5-1ubuntu6~22.04.5

Describe the problem you're observing

The system is running as a VM with a MariaDB on it, but the data files are on another filesystem. Only
the OS is on rpool/bpool.
Updated the kernel today from 5.15.0-132-generic to 5.15.0-135-generic. After 4,5 hours the kernel hung without an obvious reason. In vcenter there was no peak in any graph. MAybe it was too short but nothing to see.

Describe how to reproduce the problem

As this is the first problem after kernel upgrade and there are no obvious reasons I can't say how to reproduce.

Include any warning/errors/backtraces from the system logs

Mar 20 10:11:11 myhostname kernel: [13412.320768] INFO: task systemd-journal:1060 blocked for more than 120 seconds.
Mar 20 10:11:11 myhostname kernel: [13412.320861]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:11:11 myhostname kernel: [13412.320883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:11:11 myhostname kernel: [13412.320911] task:systemd-journal state:D stack:    0 pid: 1060 ppid:     1 flags:0x00004226
Mar 20 10:11:11 myhostname kernel: [13412.320915] Call Trace:
Mar 20 10:11:11 myhostname kernel: [13412.320917]  <TASK>
Mar 20 10:11:11 myhostname kernel: [13412.320919]  __schedule+0x24e/0x590
Mar 20 10:11:11 myhostname kernel: [13412.320926]  schedule+0x69/0x110
Mar 20 10:11:11 myhostname kernel: [13412.320927]  io_schedule+0x46/0x80
Mar 20 10:11:11 myhostname kernel: [13412.320929]  wait_on_page_bit_common+0x10c/0x3d0
Mar 20 10:11:11 myhostname kernel: [13412.320935]  ? filemap_invalidate_unlock_two+0x50/0x50
Mar 20 10:11:11 myhostname kernel: [13412.320939]  wait_on_page_bit+0x3f/0x50
Mar 20 10:11:11 myhostname kernel: [13412.320941]  wait_on_page_writeback+0x26/0x80
Mar 20 10:11:11 myhostname kernel: [13412.320943]  truncate_inode_pages_range+0x2ba/0x690
Mar 20 10:11:11 myhostname kernel: [13412.320948]  ? dmu_recv_end_sync+0x30f/0x550 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321106]  zfs_free_range+0x165/0x180 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321212]  zfs_freesp+0x251/0x2b0 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321322]  ? zfs_zaccess_common+0xa2/0x100 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321427]  ? zfs_zaccess+0x222/0x400 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321530]  zfs_space+0xbd/0x200 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321636]  zpl_fallocate_common+0xfe/0x250 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321737]  ? cp_new_stat+0x152/0x180
Mar 20 10:11:11 myhostname kernel: [13412.321741]  ? aa_file_perm+0x127/0x2a0
Mar 20 10:11:11 myhostname kernel: [13412.321745]  zpl_fallocate+0x2e/0x40 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.321846]  vfs_fallocate+0x152/0x300
Mar 20 10:11:11 myhostname kernel: [13412.321849]  __x64_sys_fallocate+0x43/0x70
Mar 20 10:11:11 myhostname kernel: [13412.321851]  x64_sys_call+0x17e2/0x1fa0
Mar 20 10:11:11 myhostname kernel: [13412.321854]  do_syscall_64+0x56/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.321856]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.321860]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:11:11 myhostname kernel: [13412.321862]  ? x64_sys_call+0xc4d/0x1fa0
Mar 20 10:11:11 myhostname kernel: [13412.321864]  ? do_syscall_64+0x63/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.321865]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.321867]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:11:11 myhostname kernel: [13412.321869]  ? x64_sys_call+0x1a81/0x1fa0
Mar 20 10:11:11 myhostname kernel: [13412.321871]  ? do_syscall_64+0x63/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.321872]  ? irqentry_exit_to_user_mode+0xe/0x20
Mar 20 10:11:11 myhostname kernel: [13412.321874]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.321875]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.321876]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.321877]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.321878]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.321879]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:11:11 myhostname kernel: [13412.321882] RIP: 0033:0x7f495516f80f
Mar 20 10:11:11 myhostname kernel: [13412.321884] RSP: 002b:00007ffe85e7c060 EFLAGS: 00000293 ORIG_RAX: 000000000000011d
Mar 20 10:11:11 myhostname kernel: [13412.321886] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f495516f80f
Mar 20 10:11:11 myhostname kernel: [13412.321888] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000027
Mar 20 10:11:11 myhostname kernel: [13412.321889] RBP: 000055f307f16020 R08: 0000000000000000 R09: 0000000000000000
Mar 20 10:11:11 myhostname kernel: [13412.321890] R10: 0000000000800000 R11: 0000000000000293 R12: 0000000000000000
Mar 20 10:11:11 myhostname kernel: [13412.321891] R13: 0000000000000027 R14: 00007ffe85e7c090 R15: 000055f307f16020
Mar 20 10:11:11 myhostname kernel: [13412.321899]  </TASK>
Mar 20 10:11:11 myhostname kernel: [13412.321900] INFO: task journal-offline:285647 blocked for more than 120 seconds.
Mar 20 10:11:11 myhostname kernel: [13412.321927]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:11:11 myhostname kernel: [13412.321948] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:11:11 myhostname kernel: [13412.321971] task:journal-offline state:D stack:    0 pid:285647 ppid:     1 flags:0x00000222
Mar 20 10:11:11 myhostname kernel: [13412.321974] Call Trace:
Mar 20 10:11:11 myhostname kernel: [13412.321975]  <TASK>
Mar 20 10:11:11 myhostname kernel: [13412.321976]  __schedule+0x24e/0x590
Mar 20 10:11:11 myhostname kernel: [13412.321978]  schedule+0x69/0x110
Mar 20 10:11:11 myhostname kernel: [13412.321980]  schedule_preempt_disabled+0xe/0x20
Mar 20 10:11:11 myhostname kernel: [13412.321981]  __mutex_lock.constprop.0+0x267/0x490
Mar 20 10:11:11 myhostname kernel: [13412.321984]  __mutex_lock_slowpath+0x13/0x20
Mar 20 10:11:11 myhostname kernel: [13412.321985]  mutex_lock+0x38/0x50
Mar 20 10:11:11 myhostname kernel: [13412.321987]  zil_commit_writer+0x23/0xa0 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.322093]  zil_commit_impl+0x3b/0x90 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.322198]  zil_commit+0x40/0x70 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.322309]  zpl_writepages+0xf0/0x190 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.322412]  do_writepages+0xd4/0x200
Mar 20 10:11:11 myhostname kernel: [13412.322415]  ? rseq_get_rseq_cs.isra.0+0x1b/0x230
Mar 20 10:11:11 myhostname kernel: [13412.322418]  filemap_fdatawrite_wbc+0x89/0xe0
Mar 20 10:11:11 myhostname kernel: [13412.322420]  filemap_write_and_wait_range+0x6f/0xe0
Mar 20 10:11:11 myhostname kernel: [13412.322423]  zpl_fsync+0x3d/0xa0 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.322541]  vfs_fsync_range+0x46/0x90
Mar 20 10:11:11 myhostname kernel: [13412.322544]  ? __fget_light+0x39/0x90
Mar 20 10:11:11 myhostname kernel: [13412.322548]  __x64_sys_fsync+0x38/0x70
Mar 20 10:11:11 myhostname kernel: [13412.322551]  x64_sys_call+0x1f72/0x1fa0
Mar 20 10:11:11 myhostname kernel: [13412.322555]  do_syscall_64+0x56/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.322557]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:11:11 myhostname kernel: [13412.322560]  ? x64_sys_call+0xb67/0x1fa0
Mar 20 10:11:11 myhostname kernel: [13412.322563]  ? do_syscall_64+0x63/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.322565]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.322567]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.322568]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.322570]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.322571]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.322572]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:11:11 myhostname kernel: [13412.322576] RIP: 0033:0x7f49551718ab
Mar 20 10:11:11 myhostname kernel: [13412.322578] RSP: 002b:00007f4950776c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar 20 10:11:11 myhostname kernel: [13412.322580] RAX: ffffffffffffffda RBX: 000055f307f0c500 RCX: 00007f49551718ab
Mar 20 10:11:11 myhostname kernel: [13412.322582] RDX: 0000000000000000 RSI: 00007f4955501a34 RDI: 0000000000000012
Mar 20 10:11:11 myhostname kernel: [13412.322583] RBP: 00007f49555032c0 R08: 0000000000000000 R09: 00007ffe85e8c02f
Mar 20 10:11:11 myhostname kernel: [13412.322584] R10: 00007f49550ea91a R11: 0000000000000293 R12: 0000000000000002
Mar 20 10:11:11 myhostname kernel: [13412.322585] R13: 0000000000000002 R14: 00007f49550ea7d0 R15: 00007ffe85e8c070
Mar 20 10:11:11 myhostname kernel: [13412.322587]  </TASK>
Mar 20 10:11:11 myhostname kernel: [13412.322588] INFO: task journal-offline:285648 blocked for more than 120 seconds.
Mar 20 10:11:11 myhostname kernel: [13412.322612]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:11:11 myhostname kernel: [13412.322633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:11:11 myhostname kernel: [13412.322656] task:journal-offline state:D stack:    0 pid:285648 ppid:     1 flags:0x00004222
Mar 20 10:11:11 myhostname kernel: [13412.322658] Call Trace:
Mar 20 10:11:11 myhostname kernel: [13412.322659]  <TASK>
Mar 20 10:11:11 myhostname kernel: [13412.322660]  __schedule+0x24e/0x590
Mar 20 10:11:11 myhostname kernel: [13412.322662]  schedule+0x69/0x110
Mar 20 10:11:11 myhostname kernel: [13412.322664]  cv_wait_common+0xf8/0x130 [spl]
Mar 20 10:11:11 myhostname kernel: [13412.322675]  ? wait_woken+0x70/0x70
Mar 20 10:11:11 myhostname kernel: [13412.322679]  __cv_wait+0x15/0x20 [spl]
Mar 20 10:11:11 myhostname kernel: [13412.322685]  zfs_rangelock_enter_reader+0xa1/0x1f0 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.322812]  zfs_rangelock_enter_impl+0x11d/0x130 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.322918]  zfs_rangelock_enter+0x11/0x20 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323039]  zfs_get_data+0xef/0x3e0 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323149]  zil_lwb_commit+0x2a0/0x370 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323257]  zil_process_commit_list+0x107/0x230 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323370]  zil_commit_writer+0x95/0xa0 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323495]  zil_commit_impl+0x3b/0x90 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323605]  zil_commit+0x40/0x70 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323710]  zpl_writepages+0xf0/0x190 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323822]  do_writepages+0xd4/0x200
Mar 20 10:11:11 myhostname kernel: [13412.323826]  filemap_fdatawrite_wbc+0x89/0xe0
Mar 20 10:11:11 myhostname kernel: [13412.323829]  filemap_write_and_wait_range+0x6f/0xe0
Mar 20 10:11:11 myhostname kernel: [13412.323834]  zpl_fsync+0x3d/0xa0 [zfs]
Mar 20 10:11:11 myhostname kernel: [13412.323959]  vfs_fsync_range+0x46/0x90
Mar 20 10:11:11 myhostname kernel: [13412.323961]  ? __fget_light+0x39/0x90
Mar 20 10:11:11 myhostname kernel: [13412.323963]  __x64_sys_fsync+0x38/0x70
Mar 20 10:11:11 myhostname kernel: [13412.323966]  x64_sys_call+0x1f72/0x1fa0
Mar 20 10:11:11 myhostname kernel: [13412.323968]  do_syscall_64+0x56/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.323970]  ? do_syscall_64+0x63/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.323971]  ? sigprocmask+0xac/0xf0
Mar 20 10:11:11 myhostname kernel: [13412.323975]  ? __x64_sys_rt_sigprocmask+0x77/0xd0
Mar 20 10:11:11 myhostname kernel: [13412.323978]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.323982]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:11:11 myhostname kernel: [13412.323983]  ? x64_sys_call+0x1e28/0x1fa0
Mar 20 10:11:11 myhostname kernel: [13412.323985]  ? do_syscall_64+0x63/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.323986]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:11:11 myhostname kernel: [13412.323988]  ? irqentry_exit_to_user_mode+0xe/0x20
Mar 20 10:11:11 myhostname kernel: [13412.323990]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.323991]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.323992]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.323993]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.323994]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:11:11 myhostname kernel: [13412.323995]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:11:11 myhostname kernel: [13412.323998] RIP: 0033:0x7f49551718ab
Mar 20 10:11:11 myhostname kernel: [13412.323999] RSP: 002b:00007f494e775c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar 20 10:11:11 myhostname kernel: [13412.324001] RAX: ffffffffffffffda RBX: 000055f307f43c30 RCX: 00007f49551718ab
Mar 20 10:11:11 myhostname kernel: [13412.324003] RDX: 0000000000000000 RSI: 00007f4955501a34 RDI: 0000000000000025
Mar 20 10:11:11 myhostname kernel: [13412.324004] RBP: 00007f49555032c0 R08: 0000000000000000 R09: 00007ffe85e8c02f
Mar 20 10:11:11 myhostname kernel: [13412.324004] R10: 00007f49550ea91a R11: 0000000000000293 R12: 0000000000000002
Mar 20 10:11:11 myhostname kernel: [13412.324005] R13: 0000000000000002 R14: 00007f49550ea7d0 R15: 00007ffe85e8c070
Mar 20 10:11:11 myhostname kernel: [13412.324007]  </TASK>
Mar 20 10:13:12 myhostname kernel: [13533.141362] INFO: task systemd-journal:1060 blocked for more than 241 seconds.
Mar 20 10:13:12 myhostname kernel: [13533.141400]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:13:12 myhostname kernel: [13533.141422] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:13:12 myhostname kernel: [13533.141444] task:systemd-journal state:D stack:    0 pid: 1060 ppid:     1 flags:0x00004226
Mar 20 10:13:12 myhostname kernel: [13533.141448] Call Trace:
Mar 20 10:13:12 myhostname kernel: [13533.141449]  <TASK>
Mar 20 10:13:12 myhostname kernel: [13533.141452]  __schedule+0x24e/0x590
Mar 20 10:13:12 myhostname kernel: [13533.141458]  schedule+0x69/0x110
Mar 20 10:13:12 myhostname kernel: [13533.141459]  io_schedule+0x46/0x80
Mar 20 10:13:12 myhostname kernel: [13533.141461]  wait_on_page_bit_common+0x10c/0x3d0
Mar 20 10:13:12 myhostname kernel: [13533.141468]  ? filemap_invalidate_unlock_two+0x50/0x50
Mar 20 10:13:12 myhostname kernel: [13533.141471]  wait_on_page_bit+0x3f/0x50
Mar 20 10:13:12 myhostname kernel: [13533.141473]  wait_on_page_writeback+0x26/0x80
Mar 20 10:13:12 myhostname kernel: [13533.141475]  truncate_inode_pages_range+0x2ba/0x690
Mar 20 10:13:12 myhostname kernel: [13533.141480]  ? dmu_recv_end_sync+0x30f/0x550 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.141605]  zfs_free_range+0x165/0x180 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.141711]  zfs_freesp+0x251/0x2b0 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.141814]  ? zfs_zaccess_common+0xa2/0x100 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.141918]  ? zfs_zaccess+0x222/0x400 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142032]  zfs_space+0xbd/0x200 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142134]  zpl_fallocate_common+0xfe/0x250 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142235]  ? cp_new_stat+0x152/0x180
Mar 20 10:13:12 myhostname kernel: [13533.142240]  ? aa_file_perm+0x127/0x2a0
Mar 20 10:13:12 myhostname kernel: [13533.142246]  zpl_fallocate+0x2e/0x40 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142347]  vfs_fallocate+0x152/0x300
Mar 20 10:13:12 myhostname kernel: [13533.142350]  __x64_sys_fallocate+0x43/0x70
Mar 20 10:13:12 myhostname kernel: [13533.142352]  x64_sys_call+0x17e2/0x1fa0
Mar 20 10:13:12 myhostname kernel: [13533.142357]  do_syscall_64+0x56/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.142359]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.142364]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:13:12 myhostname kernel: [13533.142366]  ? x64_sys_call+0xc4d/0x1fa0
Mar 20 10:13:12 myhostname kernel: [13533.142368]  ? do_syscall_64+0x63/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.142369]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.142371]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:13:12 myhostname kernel: [13533.142372]  ? x64_sys_call+0x1a81/0x1fa0
Mar 20 10:13:12 myhostname kernel: [13533.142374]  ? do_syscall_64+0x63/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.142375]  ? irqentry_exit_to_user_mode+0xe/0x20
Mar 20 10:13:12 myhostname kernel: [13533.142377]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.142380]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.142381]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.142382]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.142383]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.142384]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:13:12 myhostname kernel: [13533.142387] RIP: 0033:0x7f495516f80f
Mar 20 10:13:12 myhostname kernel: [13533.142389] RSP: 002b:00007ffe85e7c060 EFLAGS: 00000293 ORIG_RAX: 000000000000011d
Mar 20 10:13:12 myhostname kernel: [13533.142391] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f495516f80f
Mar 20 10:13:12 myhostname kernel: [13533.142392] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000027
Mar 20 10:13:12 myhostname kernel: [13533.142393] RBP: 000055f307f16020 R08: 0000000000000000 R09: 0000000000000000
Mar 20 10:13:12 myhostname kernel: [13533.142394] R10: 0000000000800000 R11: 0000000000000293 R12: 0000000000000000
Mar 20 10:13:12 myhostname kernel: [13533.142395] R13: 0000000000000027 R14: 00007ffe85e7c090 R15: 000055f307f16020
Mar 20 10:13:12 myhostname kernel: [13533.142397]  </TASK>
Mar 20 10:13:12 myhostname kernel: [13533.142399] INFO: task journal-offline:285647 blocked for more than 241 seconds.
Mar 20 10:13:12 myhostname kernel: [13533.142423]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:13:12 myhostname kernel: [13533.142444] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:13:12 myhostname kernel: [13533.142467] task:journal-offline state:D stack:    0 pid:285647 ppid:     1 flags:0x00000226
Mar 20 10:13:12 myhostname kernel: [13533.142470] Call Trace:
Mar 20 10:13:12 myhostname kernel: [13533.142471]  <TASK>
Mar 20 10:13:12 myhostname kernel: [13533.142472]  __schedule+0x24e/0x590
Mar 20 10:13:12 myhostname kernel: [13533.142474]  schedule+0x69/0x110
Mar 20 10:13:12 myhostname kernel: [13533.142475]  schedule_preempt_disabled+0xe/0x20
Mar 20 10:13:12 myhostname kernel: [13533.142477]  __mutex_lock.constprop.0+0x267/0x490
Mar 20 10:13:12 myhostname kernel: [13533.142479]  __mutex_lock_slowpath+0x13/0x20
Mar 20 10:13:12 myhostname kernel: [13533.142480]  mutex_lock+0x38/0x50
Mar 20 10:13:12 myhostname kernel: [13533.142482]  zil_commit_writer+0x23/0xa0 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142592]  zil_commit_impl+0x3b/0x90 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142698]  zil_commit+0x40/0x70 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142804]  zpl_writepages+0xf0/0x190 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.142906]  do_writepages+0xd4/0x200
Mar 20 10:13:12 myhostname kernel: [13533.142909]  ? rseq_get_rseq_cs.isra.0+0x1b/0x230
Mar 20 10:13:12 myhostname kernel: [13533.142912]  filemap_fdatawrite_wbc+0x89/0xe0
Mar 20 10:13:12 myhostname kernel: [13533.142914]  filemap_write_and_wait_range+0x6f/0xe0
Mar 20 10:13:12 myhostname kernel: [13533.142917]  zpl_fsync+0x3d/0xa0 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143021]  vfs_fsync_range+0x46/0x90
Mar 20 10:13:12 myhostname kernel: [13533.143024]  ? __fget_light+0x39/0x90
Mar 20 10:13:12 myhostname kernel: [13533.143027]  __x64_sys_fsync+0x38/0x70
Mar 20 10:13:12 myhostname kernel: [13533.143029]  x64_sys_call+0x1f72/0x1fa0
Mar 20 10:13:12 myhostname kernel: [13533.143031]  do_syscall_64+0x56/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.143033]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:13:12 myhostname kernel: [13533.143035]  ? x64_sys_call+0xb67/0x1fa0
Mar 20 10:13:12 myhostname kernel: [13533.143036]  ? do_syscall_64+0x63/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.143038]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.143039]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.143040]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.143041]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.143042]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.143043]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:13:12 myhostname kernel: [13533.143045] RIP: 0033:0x7f49551718ab
Mar 20 10:13:12 myhostname kernel: [13533.143047] RSP: 002b:00007f4950776c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar 20 10:13:12 myhostname kernel: [13533.143048] RAX: ffffffffffffffda RBX: 000055f307f0c500 RCX: 00007f49551718ab
Mar 20 10:13:12 myhostname kernel: [13533.143049] RDX: 0000000000000000 RSI: 00007f4955501a34 RDI: 0000000000000012
Mar 20 10:13:12 myhostname kernel: [13533.143050] RBP: 00007f49555032c0 R08: 0000000000000000 R09: 00007ffe85e8c02f
Mar 20 10:13:12 myhostname kernel: [13533.143051] R10: 00007f49550ea91a R11: 0000000000000293 R12: 0000000000000002
Mar 20 10:13:12 myhostname kernel: [13533.143052] R13: 0000000000000002 R14: 00007f49550ea7d0 R15: 00007ffe85e8c070
Mar 20 10:13:12 myhostname kernel: [13533.143053]  </TASK>
Mar 20 10:13:12 myhostname kernel: [13533.143054] INFO: task journal-offline:285648 blocked for more than 241 seconds.
Mar 20 10:13:12 myhostname kernel: [13533.143076]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:13:12 myhostname kernel: [13533.143103] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:13:12 myhostname kernel: [13533.143131] task:journal-offline state:D stack:    0 pid:285648 ppid:     1 flags:0x00004226
Mar 20 10:13:12 myhostname kernel: [13533.143133] Call Trace:
Mar 20 10:13:12 myhostname kernel: [13533.143135]  <TASK>
Mar 20 10:13:12 myhostname kernel: [13533.143136]  __schedule+0x24e/0x590
Mar 20 10:13:12 myhostname kernel: [13533.143138]  schedule+0x69/0x110
Mar 20 10:13:12 myhostname kernel: [13533.143141]  cv_wait_common+0xf8/0x130 [spl]
Mar 20 10:13:12 myhostname kernel: [13533.143151]  ? wait_woken+0x70/0x70
Mar 20 10:13:12 myhostname kernel: [13533.143155]  __cv_wait+0x15/0x20 [spl]
Mar 20 10:13:12 myhostname kernel: [13533.143161]  zfs_rangelock_enter_reader+0xa1/0x1f0 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143281]  zfs_rangelock_enter_impl+0x11d/0x130 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143386]  zfs_rangelock_enter+0x11/0x20 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143491]  zfs_get_data+0xef/0x3e0 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143595]  zil_lwb_commit+0x2a0/0x370 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143700]  zil_process_commit_list+0x107/0x230 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143805]  zil_commit_writer+0x95/0xa0 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.143909]  zil_commit_impl+0x3b/0x90 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.144013]  zil_commit+0x40/0x70 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.144121]  zpl_writepages+0xf0/0x190 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.144223]  do_writepages+0xd4/0x200
Mar 20 10:13:12 myhostname kernel: [13533.144226]  filemap_fdatawrite_wbc+0x89/0xe0
Mar 20 10:13:12 myhostname kernel: [13533.144229]  filemap_write_and_wait_range+0x6f/0xe0
Mar 20 10:13:12 myhostname kernel: [13533.144232]  zpl_fsync+0x3d/0xa0 [zfs]
Mar 20 10:13:12 myhostname kernel: [13533.144333]  vfs_fsync_range+0x46/0x90
Mar 20 10:13:12 myhostname kernel: [13533.144336]  ? __fget_light+0x39/0x90
Mar 20 10:13:12 myhostname kernel: [13533.144338]  __x64_sys_fsync+0x38/0x70
Mar 20 10:13:12 myhostname kernel: [13533.144340]  x64_sys_call+0x1f72/0x1fa0
Mar 20 10:13:12 myhostname kernel: [13533.144342]  do_syscall_64+0x56/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.144343]  ? do_syscall_64+0x63/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.144344]  ? sigprocmask+0xac/0xf0
Mar 20 10:13:12 myhostname kernel: [13533.144347]  ? __x64_sys_rt_sigprocmask+0x77/0xd0
Mar 20 10:13:12 myhostname kernel: [13533.144349]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.144352]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:13:12 myhostname kernel: [13533.144354]  ? x64_sys_call+0x1e28/0x1fa0
Mar 20 10:13:12 myhostname kernel: [13533.144356]  ? do_syscall_64+0x63/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.144357]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:13:12 myhostname kernel: [13533.144359]  ? irqentry_exit_to_user_mode+0xe/0x20
Mar 20 10:13:12 myhostname kernel: [13533.144361]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.144362]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.144363]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.144364]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.144365]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:13:12 myhostname kernel: [13533.144366]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:13:12 myhostname kernel: [13533.144368] RIP: 0033:0x7f49551718ab
Mar 20 10:13:12 myhostname kernel: [13533.144369] RSP: 002b:00007f494e775c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar 20 10:13:12 myhostname kernel: [13533.144370] RAX: ffffffffffffffda RBX: 000055f307f43c30 RCX: 00007f49551718ab
Mar 20 10:13:12 myhostname kernel: [13533.144371] RDX: 0000000000000000 RSI: 00007f4955501a34 RDI: 0000000000000025
Mar 20 10:13:12 myhostname kernel: [13533.144372] RBP: 00007f49555032c0 R08: 0000000000000000 R09: 00007ffe85e8c02f
Mar 20 10:13:12 myhostname kernel: [13533.144373] R10: 00007f49550ea91a R11: 0000000000000293 R12: 0000000000000002
Mar 20 10:13:12 myhostname kernel: [13533.144374] R13: 0000000000000002 R14: 00007f49550ea7d0 R15: 00007ffe85e8c070
Mar 20 10:13:12 myhostname kernel: [13533.144375]  </TASK>
Mar 20 10:15:13 myhostname kernel: [13653.961895] INFO: task systemd-journal:1060 blocked for more than 362 seconds.
Mar 20 10:15:13 myhostname kernel: [13653.961938]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:15:13 myhostname kernel: [13653.961965] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:15:13 myhostname kernel: [13653.961989] task:systemd-journal state:D stack:    0 pid: 1060 ppid:     1 flags:0x00004226
Mar 20 10:15:13 myhostname kernel: [13653.961992] Call Trace:
Mar 20 10:15:13 myhostname kernel: [13653.961994]  <TASK>
Mar 20 10:15:13 myhostname kernel: [13653.961997]  __schedule+0x24e/0x590
Mar 20 10:15:13 myhostname kernel: [13653.962026]  schedule+0x69/0x110
Mar 20 10:15:13 myhostname kernel: [13653.962027]  io_schedule+0x46/0x80
Mar 20 10:15:13 myhostname kernel: [13653.962029]  wait_on_page_bit_common+0x10c/0x3d0
Mar 20 10:15:13 myhostname kernel: [13653.962049]  ? filemap_invalidate_unlock_two+0x50/0x50
Mar 20 10:15:13 myhostname kernel: [13653.962055]  wait_on_page_bit+0x3f/0x50
Mar 20 10:15:13 myhostname kernel: [13653.962057]  wait_on_page_writeback+0x26/0x80
Mar 20 10:15:13 myhostname kernel: [13653.962059]  truncate_inode_pages_range+0x2ba/0x690
Mar 20 10:15:13 myhostname kernel: [13653.962064]  ? dmu_recv_end_sync+0x30f/0x550 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.962199]  zfs_free_range+0x165/0x180 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.962305]  zfs_freesp+0x251/0x2b0 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.962416]  ? zfs_zaccess_common+0xa2/0x100 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.962518]  ? zfs_zaccess+0x222/0x400 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.962623]  zfs_space+0xbd/0x200 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.962726]  zpl_fallocate_common+0xfe/0x250 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.962827]  ? cp_new_stat+0x152/0x180
Mar 20 10:15:13 myhostname kernel: [13653.962839]  ? aa_file_perm+0x127/0x2a0
Mar 20 10:15:13 myhostname kernel: [13653.962858]  zpl_fallocate+0x2e/0x40 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.963002]  vfs_fallocate+0x152/0x300
Mar 20 10:15:13 myhostname kernel: [13653.963007]  __x64_sys_fallocate+0x43/0x70
Mar 20 10:15:13 myhostname kernel: [13653.963009]  x64_sys_call+0x17e2/0x1fa0
Mar 20 10:15:13 myhostname kernel: [13653.963033]  do_syscall_64+0x56/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.963036]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.963051]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:15:13 myhostname kernel: [13653.963055]  ? x64_sys_call+0xc4d/0x1fa0
Mar 20 10:15:13 myhostname kernel: [13653.963057]  ? do_syscall_64+0x63/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.963059]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.963062]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:15:13 myhostname kernel: [13653.963064]  ? x64_sys_call+0x1a81/0x1fa0
Mar 20 10:15:13 myhostname kernel: [13653.963066]  ? do_syscall_64+0x63/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.963068]  ? irqentry_exit_to_user_mode+0xe/0x20
Mar 20 10:15:13 myhostname kernel: [13653.963070]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963073]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963074]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963076]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963077]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963079]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:15:13 myhostname kernel: [13653.963084] RIP: 0033:0x7f495516f80f
Mar 20 10:15:13 myhostname kernel: [13653.963090] RSP: 002b:00007ffe85e7c060 EFLAGS: 00000293 ORIG_RAX: 000000000000011d
Mar 20 10:15:13 myhostname kernel: [13653.963093] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f495516f80f
Mar 20 10:15:13 myhostname kernel: [13653.963094] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000027
Mar 20 10:15:13 myhostname kernel: [13653.963096] RBP: 000055f307f16020 R08: 0000000000000000 R09: 0000000000000000
Mar 20 10:15:13 myhostname kernel: [13653.963097] R10: 0000000000800000 R11: 0000000000000293 R12: 0000000000000000
Mar 20 10:15:13 myhostname kernel: [13653.963098] R13: 0000000000000027 R14: 00007ffe85e7c090 R15: 000055f307f16020
Mar 20 10:15:13 myhostname kernel: [13653.963102]  </TASK>
Mar 20 10:15:13 myhostname kernel: [13653.963104] INFO: task journal-offline:285647 blocked for more than 362 seconds.
Mar 20 10:15:13 myhostname kernel: [13653.963144]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:15:13 myhostname kernel: [13653.963176] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:15:13 myhostname kernel: [13653.963201] task:journal-offline state:D stack:    0 pid:285647 ppid:     1 flags:0x00000226
Mar 20 10:15:13 myhostname kernel: [13653.963204] Call Trace:
Mar 20 10:15:13 myhostname kernel: [13653.963207]  <TASK>
Mar 20 10:15:13 myhostname kernel: [13653.963208]  __schedule+0x24e/0x590
Mar 20 10:15:13 myhostname kernel: [13653.963211]  schedule+0x69/0x110
Mar 20 10:15:13 myhostname kernel: [13653.963214]  schedule_preempt_disabled+0xe/0x20
Mar 20 10:15:13 myhostname kernel: [13653.963216]  __mutex_lock.constprop.0+0x267/0x490
Mar 20 10:15:13 myhostname kernel: [13653.963219]  __mutex_lock_slowpath+0x13/0x20
Mar 20 10:15:13 myhostname kernel: [13653.963221]  mutex_lock+0x38/0x50
Mar 20 10:15:13 myhostname kernel: [13653.963224]  zil_commit_writer+0x23/0xa0 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.963363]  zil_commit_impl+0x3b/0x90 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.963470]  zil_commit+0x40/0x70 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.963575]  zpl_writepages+0xf0/0x190 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.963681]  do_writepages+0xd4/0x200
Mar 20 10:15:13 myhostname kernel: [13653.963685]  ? rseq_get_rseq_cs.isra.0+0x1b/0x230
Mar 20 10:15:13 myhostname kernel: [13653.963688]  filemap_fdatawrite_wbc+0x89/0xe0
Mar 20 10:15:13 myhostname kernel: [13653.963690]  filemap_write_and_wait_range+0x6f/0xe0
Mar 20 10:15:13 myhostname kernel: [13653.963693]  zpl_fsync+0x3d/0xa0 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.963795]  vfs_fsync_range+0x46/0x90
Mar 20 10:15:13 myhostname kernel: [13653.963798]  ? __fget_light+0x39/0x90
Mar 20 10:15:13 myhostname kernel: [13653.963801]  __x64_sys_fsync+0x38/0x70
Mar 20 10:15:13 myhostname kernel: [13653.963803]  x64_sys_call+0x1f72/0x1fa0
Mar 20 10:15:13 myhostname kernel: [13653.963806]  do_syscall_64+0x56/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.963807]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:15:13 myhostname kernel: [13653.963809]  ? x64_sys_call+0xb67/0x1fa0
Mar 20 10:15:13 myhostname kernel: [13653.963811]  ? do_syscall_64+0x63/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.963812]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963813]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963814]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963815]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963816]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.963817]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:15:13 myhostname kernel: [13653.963820] RIP: 0033:0x7f49551718ab
Mar 20 10:15:13 myhostname kernel: [13653.963821] RSP: 002b:00007f4950776c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar 20 10:15:13 myhostname kernel: [13653.963823] RAX: ffffffffffffffda RBX: 000055f307f0c500 RCX: 00007f49551718ab
Mar 20 10:15:13 myhostname kernel: [13653.963824] RDX: 0000000000000000 RSI: 00007f4955501a34 RDI: 0000000000000012
Mar 20 10:15:13 myhostname kernel: [13653.963825] RBP: 00007f49555032c0 R08: 0000000000000000 R09: 00007ffe85e8c02f
Mar 20 10:15:13 myhostname kernel: [13653.963826] R10: 00007f49550ea91a R11: 0000000000000293 R12: 0000000000000002
Mar 20 10:15:13 myhostname kernel: [13653.963827] R13: 0000000000000002 R14: 00007f49550ea7d0 R15: 00007ffe85e8c070
Mar 20 10:15:13 myhostname kernel: [13653.963828]  </TASK>
Mar 20 10:15:13 myhostname kernel: [13653.963829] INFO: task journal-offline:285648 blocked for more than 362 seconds.
Mar 20 10:15:13 myhostname kernel: [13653.963855]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:15:13 myhostname kernel: [13653.963875] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:15:13 myhostname kernel: [13653.963897] task:journal-offline state:D stack:    0 pid:285648 ppid:     1 flags:0x00004226
Mar 20 10:15:13 myhostname kernel: [13653.963900] Call Trace:
Mar 20 10:15:13 myhostname kernel: [13653.963901]  <TASK>
Mar 20 10:15:13 myhostname kernel: [13653.963901]  __schedule+0x24e/0x590
Mar 20 10:15:13 myhostname kernel: [13653.963903]  schedule+0x69/0x110
Mar 20 10:15:13 myhostname kernel: [13653.963905]  cv_wait_common+0xf8/0x130 [spl]
Mar 20 10:15:13 myhostname kernel: [13653.963916]  ? wait_woken+0x70/0x70
Mar 20 10:15:13 myhostname kernel: [13653.963929]  __cv_wait+0x15/0x20 [spl]
Mar 20 10:15:13 myhostname kernel: [13653.963935]  zfs_rangelock_enter_reader+0xa1/0x1f0 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964055]  zfs_rangelock_enter_impl+0x11d/0x130 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964161]  zfs_rangelock_enter+0x11/0x20 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964266]  zfs_get_data+0xef/0x3e0 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964377]  zil_lwb_commit+0x2a0/0x370 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964482]  zil_process_commit_list+0x107/0x230 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964587]  zil_commit_writer+0x95/0xa0 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964691]  zil_commit_impl+0x3b/0x90 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964795]  zil_commit+0x40/0x70 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.964899]  zpl_writepages+0xf0/0x190 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.965005]  do_writepages+0xd4/0x200
Mar 20 10:15:13 myhostname kernel: [13653.965008]  filemap_fdatawrite_wbc+0x89/0xe0
Mar 20 10:15:13 myhostname kernel: [13653.965011]  filemap_write_and_wait_range+0x6f/0xe0
Mar 20 10:15:13 myhostname kernel: [13653.965014]  zpl_fsync+0x3d/0xa0 [zfs]
Mar 20 10:15:13 myhostname kernel: [13653.965116]  vfs_fsync_range+0x46/0x90
Mar 20 10:15:13 myhostname kernel: [13653.965118]  ? __fget_light+0x39/0x90
Mar 20 10:15:13 myhostname kernel: [13653.965121]  __x64_sys_fsync+0x38/0x70
Mar 20 10:15:13 myhostname kernel: [13653.965123]  x64_sys_call+0x1f72/0x1fa0
Mar 20 10:15:13 myhostname kernel: [13653.965125]  do_syscall_64+0x56/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.965127]  ? do_syscall_64+0x63/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.965128]  ? sigprocmask+0xac/0xf0
Mar 20 10:15:13 myhostname kernel: [13653.965140]  ? __x64_sys_rt_sigprocmask+0x77/0xd0
Mar 20 10:15:13 myhostname kernel: [13653.965142]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.965145]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:15:13 myhostname kernel: [13653.965147]  ? x64_sys_call+0x1e28/0x1fa0
Mar 20 10:15:13 myhostname kernel: [13653.965150]  ? do_syscall_64+0x63/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.965151]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:15:13 myhostname kernel: [13653.965154]  ? irqentry_exit_to_user_mode+0xe/0x20
Mar 20 10:15:13 myhostname kernel: [13653.965156]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.965158]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.965159]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.965160]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.965161]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:15:13 myhostname kernel: [13653.965163]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:15:13 myhostname kernel: [13653.965165] RIP: 0033:0x7f49551718ab
Mar 20 10:15:13 myhostname kernel: [13653.965167] RSP: 002b:00007f494e775c50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a
Mar 20 10:15:13 myhostname kernel: [13653.965168] RAX: ffffffffffffffda RBX: 000055f307f43c30 RCX: 00007f49551718ab
Mar 20 10:15:13 myhostname kernel: [13653.965170] RDX: 0000000000000000 RSI: 00007f4955501a34 RDI: 0000000000000025
Mar 20 10:15:13 myhostname kernel: [13653.965170] RBP: 00007f49555032c0 R08: 0000000000000000 R09: 00007ffe85e8c02f
Mar 20 10:15:13 myhostname kernel: [13653.965172] R10: 00007f49550ea91a R11: 0000000000000293 R12: 0000000000000002
Mar 20 10:15:13 myhostname kernel: [13653.965173] R13: 0000000000000002 R14: 00007f49550ea7d0 R15: 00007ffe85e8c070
Mar 20 10:15:13 myhostname kernel: [13653.965174]  </TASK>
Mar 20 10:17:14 myhostname kernel: [13774.782524] INFO: task systemd-journal:1060 blocked for more than 483 seconds.
Mar 20 10:17:14 myhostname kernel: [13774.782560]       Tainted: P           O      5.15.0-135-generic #146-Ubuntu
Mar 20 10:17:14 myhostname kernel: [13774.782582] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 20 10:17:14 myhostname kernel: [13774.782605] task:systemd-journal state:D stack:    0 pid: 1060 ppid:     1 flags:0x00004226
Mar 20 10:17:14 myhostname kernel: [13774.782609] Call Trace:
Mar 20 10:17:14 myhostname kernel: [13774.782611]  <TASK>
Mar 20 10:17:14 myhostname kernel: [13774.782613]  __schedule+0x24e/0x590
Mar 20 10:17:14 myhostname kernel: [13774.782620]  schedule+0x69/0x110
Mar 20 10:17:14 myhostname kernel: [13774.782622]  io_schedule+0x46/0x80
Mar 20 10:17:14 myhostname kernel: [13774.782624]  wait_on_page_bit_common+0x10c/0x3d0
Mar 20 10:17:14 myhostname kernel: [13774.782629]  ? filemap_invalidate_unlock_two+0x50/0x50
Mar 20 10:17:14 myhostname kernel: [13774.782633]  wait_on_page_bit+0x3f/0x50
Mar 20 10:17:14 myhostname kernel: [13774.782635]  wait_on_page_writeback+0x26/0x80
Mar 20 10:17:14 myhostname kernel: [13774.782637]  truncate_inode_pages_range+0x2ba/0x690
Mar 20 10:17:14 myhostname kernel: [13774.782643]  ? dmu_recv_end_sync+0x30f/0x550 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.782775]  zfs_free_range+0x165/0x180 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.782891]  zfs_freesp+0x251/0x2b0 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.783012]  ? zfs_zaccess_common+0xa2/0x100 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.783116]  ? zfs_zaccess+0x222/0x400 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.783230]  zfs_space+0xbd/0x200 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.783333]  zpl_fallocate_common+0xfe/0x250 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.783434]  ? cp_new_stat+0x152/0x180
Mar 20 10:17:14 myhostname kernel: [13774.783439]  ? aa_file_perm+0x127/0x2a0
Mar 20 10:17:14 myhostname kernel: [13774.783443]  zpl_fallocate+0x2e/0x40 [zfs]
Mar 20 10:17:14 myhostname kernel: [13774.783552]  vfs_fallocate+0x152/0x300
Mar 20 10:17:14 myhostname kernel: [13774.783555]  __x64_sys_fallocate+0x43/0x70
Mar 20 10:17:14 myhostname kernel: [13774.783557]  x64_sys_call+0x17e2/0x1fa0
Mar 20 10:17:14 myhostname kernel: [13774.783562]  do_syscall_64+0x56/0xb0
Mar 20 10:17:14 myhostname kernel: [13774.783564]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:17:14 myhostname kernel: [13774.783568]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:17:14 myhostname kernel: [13774.783571]  ? x64_sys_call+0xc4d/0x1fa0
Mar 20 10:17:14 myhostname kernel: [13774.783573]  ? do_syscall_64+0x63/0xb0
Mar 20 10:17:14 myhostname kernel: [13774.783574]  ? exit_to_user_mode_prepare+0x37/0xb0
Mar 20 10:17:14 myhostname kernel: [13774.783576]  ? syscall_exit_to_user_mode+0x2c/0x50
Mar 20 10:17:14 myhostname kernel: [13774.783578]  ? x64_sys_call+0x1a81/0x1fa0
Mar 20 10:17:14 myhostname kernel: [13774.783580]  ? do_syscall_64+0x63/0xb0
Mar 20 10:17:14 myhostname kernel: [13774.783582]  ? irqentry_exit_to_user_mode+0xe/0x20
Mar 20 10:17:14 myhostname kernel: [13774.783584]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:17:14 myhostname kernel: [13774.783586]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:17:14 myhostname kernel: [13774.783587]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:17:14 myhostname kernel: [13774.783589]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:17:14 myhostname kernel: [13774.783590]  ? clear_bhb_loop+0x45/0xa0
Mar 20 10:17:14 myhostname kernel: [13774.783591]  entry_SYSCALL_64_after_hwframe+0x6c/0xd6
Mar 20 10:17:14 myhostname kernel: [13774.783597] RIP: 0033:0x7f495516f80f
Mar 20 10:17:14 myhostname kernel: [13774.783600] RSP: 002b:00007ffe85e7c060 EFLAGS: 00000293 ORIG_RAX: 000000000000011d
Mar 20 10:17:14 myhostname kernel: [13774.783602] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f495516f80f
Mar 20 10:17:14 myhostname kernel: [13774.783603] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000027
Mar 20 10:17:14 myhostname kernel: [13774.783605] RBP: 000055f307f16020 R08: 0000000000000000 R09: 0000000000000000
Mar 20 10:17:14 myhostname kernel: [13774.783606] R10: 0000000000800000 R11: 0000000000000293 R12: 0000000000000000
Mar 20 10:17:14 myhostname kernel: [13774.783607] R13: 0000000000000027 R14: 00007ffe85e7c090 R15: 000055f307f16020
Mar 20 10:17:14 myhostname kernel: [13774.783615]  </TASK>

I hope you can see anything from taking a look at the stack traces.
Greetings,
Lars
@Popyllol Popyllol added the Type: Defect Incorrect behavior (e.g. crash, hang) label Mar 20, 2025
@snajpa
Copy link
Contributor

snajpa commented Mar 23, 2025

OpenZFS Version zfs-2.1.5-1ubuntu6~22.04.5

That's a downstream version which should receive support from Canonical, I would say :) Hard to know what 1ubuntu6 means, but 2.1.5 is definitely old...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@snajpa @Popyllol