Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- =================================================================
- right before ssh stopped working, dmesg looked like this:
- =================================================================
- [ 8029.007177] INFO: task kswapd0:52 blocked for more than 120 seconds.
- [ 8029.007540] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8029.007992] kswapd0 D ffffffff81806240 0 52 2 0x00000000
- [ 8029.008000] ffff881014607a70 0000000000000046 0000000000000000 0000000100400039
- [ 8029.008008] ffff881014607fd8 ffff881014607fd8 ffff881014607fd8 00000000000137c0
- [ 8029.008015] ffff881018fd1700 ffff881014609700 ffff880d62fca2c0 ffff880f6014c928
- [ 8029.008021] Call Trace:
- [ 8029.008033] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8029.008039] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8029.008079] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8029.008085] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8029.008131] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8029.008173] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8029.008181] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8029.008221] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8029.008228] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8029.008233] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8029.008240] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8029.008245] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8029.008252] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8029.008257] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8029.008263] [<ffffffff8112ebea>] balance_pgdat+0x50a/0x6d0
- [ 8029.008268] [<ffffffff8112eed1>] kswapd+0x121/0x220
- [ 8029.008273] [<ffffffff8112edb0>] ? balance_pgdat+0x6d0/0x6d0
- [ 8029.008279] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8029.008286] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8029.008292] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8029.008297] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8029.008305] INFO: task spl_kmem_cache/:575 blocked for more than 120 seconds.
- [ 8029.008705] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8029.009162] spl_kmem_cache/ D ffffffff81806240 0 575 2 0x00000000
- [ 8029.009176] ffff881013f35520 0000000000000046 0000000000000000 000000010040003c
- [ 8029.009188] ffff881013f35fd8 ffff881013f35fd8 ffff881013f35fd8 00000000000137c0
- [ 8029.009195] ffffffff81c0d020 ffff881013ae5c00 ffff880d622f4a40 ffff880f6014c928
- [ 8029.009201] Call Trace:
- [ 8029.009206] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8029.009211] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8029.009237] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8029.009243] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8029.009281] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8029.009318] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8029.009324] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8029.009361] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8029.009367] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8029.009372] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8029.009377] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8029.009382] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8029.009388] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8029.009393] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8029.009398] [<ffffffff8112e188>] do_try_to_free_pages+0x268/0x2d0
- [ 8029.009403] [<ffffffff8112e4cb>] try_to_free_pages+0xab/0x170
- [ 8029.009410] [<ffffffff8112274e>] __alloc_pages_nodemask+0x59e/0x8f0
- [ 8029.009418] [<ffffffff81159d36>] alloc_pages_current+0xb6/0x120
- [ 8029.009423] [<ffffffff8111db1e>] __get_free_pages+0xe/0x40
- [ 8029.009430] [<ffffffff81045505>] pte_alloc_one_kernel+0x15/0x20
- [ 8029.009436] [<ffffffff8113d11b>] __pte_alloc_kernel+0x1b/0xe0
- [ 8029.009441] [<ffffffff8114bb7e>] vmap_pte_range+0x14e/0x170
- [ 8029.009445] [<ffffffff8114bcc2>] vmap_pud_range+0x122/0x1c0
- [ 8029.009450] [<ffffffff8114bdf6>] vmap_page_range_noflush+0x96/0xc0
- [ 8029.009454] [<ffffffff8114be52>] map_vm_area+0x32/0x50
- [ 8029.009459] [<ffffffff8114e458>] __vmalloc_area_node+0x118/0x1c0
- [ 8029.009473] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8029.009478] [<ffffffff8114e0f6>] __vmalloc_node_range+0x86/0xd0
- [ 8029.009488] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8029.009493] [<ffffffff8114e175>] __vmalloc_node+0x35/0x40
- [ 8029.009502] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8029.009507] [<ffffffff8114e332>] __vmalloc+0x22/0x30
- [ 8029.009516] [<ffffffffa013ee11>] kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8029.009526] [<ffffffffa013ee60>] spl_cache_grow_work+0x40/0x430 [spl]
- [ 8029.009539] [<ffffffffa0144626>] taskq_thread+0x236/0x4b0 [spl]
- [ 8029.009546] [<ffffffff81060910>] ? try_to_wake_up+0x200/0x200
- [ 8029.009557] [<ffffffffa01443f0>] ? task_done+0x160/0x160 [spl]
- [ 8029.009562] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8029.009568] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8029.009574] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8029.009579] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8148.823661] INFO: task kswapd0:52 blocked for more than 120 seconds.
- [ 8148.824022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8148.824472] kswapd0 D ffffffff81806240 0 52 2 0x00000000
- [ 8148.824479] ffff881014607a70 0000000000000046 0000000000000000 0000000100400039
- [ 8148.824487] ffff881014607fd8 ffff881014607fd8 ffff881014607fd8 00000000000137c0
- [ 8148.824493] ffff881018fd1700 ffff881014609700 ffff880d62fca2c0 ffff880f6014c928
- [ 8148.824500] Call Trace:
- [ 8148.824512] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8148.824517] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8148.824558] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8148.824564] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8148.824609] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8148.824651] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8148.824659] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8148.824699] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8148.824706] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8148.824711] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8148.824718] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8148.824723] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8148.824730] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8148.824736] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8148.824742] [<ffffffff8112ebea>] balance_pgdat+0x50a/0x6d0
- [ 8148.824748] [<ffffffff8112eed1>] kswapd+0x121/0x220
- [ 8148.824753] [<ffffffff8112edb0>] ? balance_pgdat+0x6d0/0x6d0
- [ 8148.824759] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8148.824766] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8148.824772] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8148.824777] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8148.824786] INFO: task spl_kmem_cache/:575 blocked for more than 120 seconds.
- [ 8148.825185] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8148.825642] spl_kmem_cache/ D ffffffff81806240 0 575 2 0x00000000
- [ 8148.825663] ffff881013f35520 0000000000000046 0000000000000000 000000010040003c
- [ 8148.825670] ffff881013f35fd8 ffff881013f35fd8 ffff881013f35fd8 00000000000137c0
- [ 8148.825676] ffffffff81c0d020 ffff881013ae5c00 ffff880d622f4a40 ffff880f6014c928
- [ 8148.825682] Call Trace:
- [ 8148.825687] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8148.825692] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8148.825718] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8148.825724] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8148.825762] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8148.825799] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8148.825806] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8148.825843] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8148.825849] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8148.825854] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8148.825859] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8148.825864] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8148.825870] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8148.825874] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8148.825880] [<ffffffff8112e188>] do_try_to_free_pages+0x268/0x2d0
- [ 8148.825885] [<ffffffff8112e4cb>] try_to_free_pages+0xab/0x170
- [ 8148.825892] [<ffffffff8112274e>] __alloc_pages_nodemask+0x59e/0x8f0
- [ 8148.825901] [<ffffffff81159d36>] alloc_pages_current+0xb6/0x120
- [ 8148.825906] [<ffffffff8111db1e>] __get_free_pages+0xe/0x40
- [ 8148.825913] [<ffffffff81045505>] pte_alloc_one_kernel+0x15/0x20
- [ 8148.825919] [<ffffffff8113d11b>] __pte_alloc_kernel+0x1b/0xe0
- [ 8148.825924] [<ffffffff8114bb7e>] vmap_pte_range+0x14e/0x170
- [ 8148.825928] [<ffffffff8114bcc2>] vmap_pud_range+0x122/0x1c0
- [ 8148.825933] [<ffffffff8114bdf6>] vmap_page_range_noflush+0x96/0xc0
- [ 8148.825938] [<ffffffff8114be52>] map_vm_area+0x32/0x50
- [ 8148.825943] [<ffffffff8114e458>] __vmalloc_area_node+0x118/0x1c0
- [ 8148.825957] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8148.825961] [<ffffffff8114e0f6>] __vmalloc_node_range+0x86/0xd0
- [ 8148.825971] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8148.825976] [<ffffffff8114e175>] __vmalloc_node+0x35/0x40
- [ 8148.825986] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8148.825991] [<ffffffff8114e332>] __vmalloc+0x22/0x30
- [ 8148.826000] [<ffffffffa013ee11>] kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8148.826010] [<ffffffffa013ee60>] spl_cache_grow_work+0x40/0x430 [spl]
- [ 8148.826023] [<ffffffffa0144626>] taskq_thread+0x236/0x4b0 [spl]
- [ 8148.826031] [<ffffffff81060910>] ? try_to_wake_up+0x200/0x200
- [ 8148.826042] [<ffffffffa01443f0>] ? task_done+0x160/0x160 [spl]
- [ 8148.826047] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8148.826053] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8148.826058] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8148.826064] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8508.273079] INFO: task kswapd0:52 blocked for more than 120 seconds.
- [ 8508.273442] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8508.273891] kswapd0 D ffffffff81806240 0 52 2 0x00000000
- [ 8508.273898] ffff881014607a70 0000000000000046 0000000000000000 0000000000000001
- [ 8508.273906] ffff881014607fd8 ffff881014607fd8 ffff881014607fd8 00000000000137c0
- [ 8508.273913] ffff881018859700 ffff881014609700 ffff880d6afcf080 ffff880f6014ca68
- [ 8508.273919] Call Trace:
- [ 8508.273931] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8508.273937] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8508.273977] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8508.273983] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8508.274029] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8508.274070] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8508.274078] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8508.274117] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8508.274124] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8508.274128] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8508.274135] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8508.274140] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8508.274147] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8508.274152] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8508.274157] [<ffffffff8112ebea>] balance_pgdat+0x50a/0x6d0
- [ 8508.274163] [<ffffffff8112eed1>] kswapd+0x121/0x220
- [ 8508.274168] [<ffffffff8112edb0>] ? balance_pgdat+0x6d0/0x6d0
- [ 8508.274174] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8508.274181] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8508.274186] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8508.274192] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8508.274200] INFO: task spl_kmem_cache/:575 blocked for more than 120 seconds.
- [ 8508.274601] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8508.275058] spl_kmem_cache/ D ffffffff81806240 0 575 2 0x00000000
- [ 8508.275079] ffff881013f35520 0000000000000046 0000000000000000 0000000000000001
- [ 8508.275085] ffff881013f35fd8 ffff881013f35fd8 ffff881013f35fd8 00000000000137c0
- [ 8508.275092] ffffffff81c0d020 ffff881013ae5c00 ffff880d6a584a40 ffff880f6014ca68
- [ 8508.275098] Call Trace:
- [ 8508.275103] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8508.275108] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8508.275135] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8508.275140] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8508.275178] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8508.275215] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8508.275222] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8508.275259] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8508.275264] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8508.275269] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8508.275275] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8508.275280] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8508.275285] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8508.275290] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8508.275296] [<ffffffff8112e188>] do_try_to_free_pages+0x268/0x2d0
- [ 8508.275301] [<ffffffff8112e4cb>] try_to_free_pages+0xab/0x170
- [ 8508.275308] [<ffffffff8112274e>] __alloc_pages_nodemask+0x59e/0x8f0
- [ 8508.275316] [<ffffffff81159d36>] alloc_pages_current+0xb6/0x120
- [ 8508.275321] [<ffffffff8111db1e>] __get_free_pages+0xe/0x40
- [ 8508.275328] [<ffffffff81045505>] pte_alloc_one_kernel+0x15/0x20
- [ 8508.275334] [<ffffffff8113d11b>] __pte_alloc_kernel+0x1b/0xe0
- [ 8508.275338] [<ffffffff8114bb7e>] vmap_pte_range+0x14e/0x170
- [ 8508.275343] [<ffffffff8114bcc2>] vmap_pud_range+0x122/0x1c0
- [ 8508.275348] [<ffffffff8114bdf6>] vmap_page_range_noflush+0x96/0xc0
- [ 8508.275352] [<ffffffff8114be52>] map_vm_area+0x32/0x50
- [ 8508.275357] [<ffffffff8114e458>] __vmalloc_area_node+0x118/0x1c0
- [ 8508.275371] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8508.275376] [<ffffffff8114e0f6>] __vmalloc_node_range+0x86/0xd0
- [ 8508.275387] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8508.275392] [<ffffffff8114e175>] __vmalloc_node+0x35/0x40
- [ 8508.275401] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8508.275406] [<ffffffff8114e332>] __vmalloc+0x22/0x30
- [ 8508.275416] [<ffffffffa013ee11>] kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8508.275426] [<ffffffffa013ee60>] spl_cache_grow_work+0x40/0x430 [spl]
- [ 8508.275438] [<ffffffffa0144626>] taskq_thread+0x236/0x4b0 [spl]
- [ 8508.275445] [<ffffffff81060910>] ? try_to_wake_up+0x200/0x200
- [ 8508.275456] [<ffffffffa01443f0>] ? task_done+0x160/0x160 [spl]
- [ 8508.275461] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8508.275467] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8508.275473] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8508.275479] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8628.089535] INFO: task kswapd0:52 blocked for more than 120 seconds.
- [ 8628.089896] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8628.090363] kswapd0 D ffffffff81806240 0 52 2 0x00000000
- [ 8628.090370] ffff881014607a70 0000000000000046 0000000000000000 0000000000000001
- [ 8628.090378] ffff881014607fd8 ffff881014607fd8 ffff881014607fd8 00000000000137c0
- [ 8628.090385] ffff881018859700 ffff881014609700 ffff880d6afcf080 ffff880f6014ca68
- [ 8628.090392] Call Trace:
- [ 8628.090404] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8628.090410] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8628.090449] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8628.090455] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8628.090500] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8628.090542] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8628.090551] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8628.090590] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8628.090598] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8628.090603] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8628.090609] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8628.090614] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8628.090621] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8628.090627] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8628.090633] [<ffffffff8112ebea>] balance_pgdat+0x50a/0x6d0
- [ 8628.090638] [<ffffffff8112eed1>] kswapd+0x121/0x220
- [ 8628.090643] [<ffffffff8112edb0>] ? balance_pgdat+0x6d0/0x6d0
- [ 8628.090651] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8628.090658] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8628.090663] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8628.090669] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8628.090678] INFO: task spl_kmem_cache/:575 blocked for more than 120 seconds.
- [ 8628.091078] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8628.091534] spl_kmem_cache/ D ffffffff81806240 0 575 2 0x00000000
- [ 8628.091547] ffff881013f35520 0000000000000046 0000000000000000 0000000000000001
- [ 8628.091554] ffff881013f35fd8 ffff881013f35fd8 ffff881013f35fd8 00000000000137c0
- [ 8628.091560] ffffffff81c0d020 ffff881013ae5c00 ffff880d6a584a40 ffff880f6014ca68
- [ 8628.091566] Call Trace:
- [ 8628.091571] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8628.091576] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8628.091603] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8628.091608] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8628.091647] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8628.091683] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8628.091690] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8628.091727] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8628.091732] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8628.091738] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8628.091743] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8628.091748] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8628.091754] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8628.091759] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8628.091764] [<ffffffff8112e188>] do_try_to_free_pages+0x268/0x2d0
- [ 8628.091770] [<ffffffff8112e4cb>] try_to_free_pages+0xab/0x170
- [ 8628.091776] [<ffffffff8112274e>] __alloc_pages_nodemask+0x59e/0x8f0
- [ 8628.091785] [<ffffffff81159d36>] alloc_pages_current+0xb6/0x120
- [ 8628.091790] [<ffffffff8111db1e>] __get_free_pages+0xe/0x40
- [ 8628.091797] [<ffffffff81045505>] pte_alloc_one_kernel+0x15/0x20
- [ 8628.091802] [<ffffffff8113d11b>] __pte_alloc_kernel+0x1b/0xe0
- [ 8628.091807] [<ffffffff8114bb7e>] vmap_pte_range+0x14e/0x170
- [ 8628.091811] [<ffffffff8114bcc2>] vmap_pud_range+0x122/0x1c0
- [ 8628.091816] [<ffffffff8114bdf6>] vmap_page_range_noflush+0x96/0xc0
- [ 8628.091821] [<ffffffff8114be52>] map_vm_area+0x32/0x50
- [ 8628.091826] [<ffffffff8114e458>] __vmalloc_area_node+0x118/0x1c0
- [ 8628.091839] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8628.091844] [<ffffffff8114e0f6>] __vmalloc_node_range+0x86/0xd0
- [ 8628.091854] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8628.091859] [<ffffffff8114e175>] __vmalloc_node+0x35/0x40
- [ 8628.091869] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8628.091874] [<ffffffff8114e332>] __vmalloc+0x22/0x30
- [ 8628.091883] [<ffffffffa013ee11>] kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8628.091893] [<ffffffffa013ee60>] spl_cache_grow_work+0x40/0x430 [spl]
- [ 8628.091905] [<ffffffffa0144626>] taskq_thread+0x236/0x4b0 [spl]
- [ 8628.091912] [<ffffffff81060910>] ? try_to_wake_up+0x200/0x200
- [ 8628.091923] [<ffffffffa01443f0>] ? task_done+0x160/0x160 [spl]
- [ 8628.091928] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8628.091934] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8628.091940] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8628.091946] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8747.906029] INFO: task kswapd0:52 blocked for more than 120 seconds.
- [ 8747.906391] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8747.927827] kswapd0 D ffffffff81806240 0 52 2 0x00000000
- [ 8747.927834] ffff881014607a70 0000000000000046 0000000000000000 0000000000000001
- [ 8747.927842] ffff881014607fd8 ffff881014607fd8 ffff881014607fd8 00000000000137c0
- [ 8747.927849] ffff881018859700 ffff881014609700 ffff880d6afcf080 ffff880f6014ca68
- [ 8747.927861] Call Trace:
- [ 8747.927876] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8747.927885] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8747.927929] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8747.927940] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8747.927989] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8747.928035] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8747.928048] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8747.928091] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8747.928101] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8747.928109] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8747.928120] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8747.928130] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8747.928141] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8747.928151] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8747.928160] [<ffffffff8112ebea>] balance_pgdat+0x50a/0x6d0
- [ 8747.928169] [<ffffffff8112eed1>] kswapd+0x121/0x220
- [ 8747.928178] [<ffffffff8112edb0>] ? balance_pgdat+0x6d0/0x6d0
- [ 8747.928190] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8747.928201] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8747.928211] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8747.928221] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [ 8747.928232] INFO: task spl_kmem_cache/:575 blocked for more than 120 seconds.
- [ 8747.939343] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
- [ 8747.961940] spl_kmem_cache/ D ffffffff81806240 0 575 2 0x00000000
- [ 8747.961961] ffff881013f35520 0000000000000046 0000000000000000 0000000000000001
- [ 8747.961968] ffff881013f35fd8 ffff881013f35fd8 ffff881013f35fd8 00000000000137c0
- [ 8747.961975] ffffffff81c0d020 ffff881013ae5c00 ffff880d6a584a40 ffff880f6014ca68
- [ 8747.961981] Call Trace:
- [ 8747.961987] [<ffffffff8165f2af>] schedule+0x3f/0x60
- [ 8747.961996] [<ffffffff816600b7>] __mutex_lock_slowpath+0xd7/0x150
- [ 8747.962028] [<ffffffffa01e25b0>] ? dbuf_rele_and_unlock+0x1b0/0x210 [zfs]
- [ 8747.962038] [<ffffffff8165fcca>] mutex_lock+0x2a/0x50
- [ 8747.962081] [<ffffffffa02748dc>] zfs_zinactive+0x5c/0xd0 [zfs]
- [ 8747.962124] [<ffffffffa026f3bc>] zfs_inactive+0x7c/0x230 [zfs]
- [ 8747.962134] [<ffffffff811272ae>] ? truncate_pagecache+0x5e/0x70
- [ 8747.962176] [<ffffffffa0283a28>] zpl_evict_inode+0x28/0x30 [zfs]
- [ 8747.962186] [<ffffffff81194e98>] evict+0xb8/0x1c0
- [ 8747.962195] [<ffffffff81195581>] dispose_list+0x41/0x50
- [ 8747.962204] [<ffffffff8166119f>] ? _raw_spin_trylock+0xf/0x30
- [ 8747.962213] [<ffffffff81195fc8>] prune_icache_sb+0x168/0x330
- [ 8747.962223] [<ffffffff8117d6ae>] prune_super+0xee/0x1a0
- [ 8747.962232] [<ffffffff8112b8f4>] shrink_slab+0x154/0x300
- [ 8747.962242] [<ffffffff8112e188>] do_try_to_free_pages+0x268/0x2d0
- [ 8747.962251] [<ffffffff8112e4cb>] try_to_free_pages+0xab/0x170
- [ 8747.962262] [<ffffffff8112274e>] __alloc_pages_nodemask+0x59e/0x8f0
- [ 8747.962274] [<ffffffff81159d36>] alloc_pages_current+0xb6/0x120
- [ 8747.962283] [<ffffffff8111db1e>] __get_free_pages+0xe/0x40
- [ 8747.962295] [<ffffffff81045505>] pte_alloc_one_kernel+0x15/0x20
- [ 8747.962306] [<ffffffff8113d11b>] __pte_alloc_kernel+0x1b/0xe0
- [ 8747.962314] [<ffffffff8114bb7e>] vmap_pte_range+0x14e/0x170
- [ 8747.962323] [<ffffffff8114bcc2>] vmap_pud_range+0x122/0x1c0
- [ 8747.962331] [<ffffffff8114bdf6>] vmap_page_range_noflush+0x96/0xc0
- [ 8747.962340] [<ffffffff8114be52>] map_vm_area+0x32/0x50
- [ 8747.962350] [<ffffffff8114e458>] __vmalloc_area_node+0x118/0x1c0
- [ 8747.962368] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8747.962377] [<ffffffff8114e0f6>] __vmalloc_node_range+0x86/0xd0
- [ 8747.962392] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8747.962401] [<ffffffff8114e175>] __vmalloc_node+0x35/0x40
- [ 8747.962414] [<ffffffffa013ee11>] ? kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8747.962423] [<ffffffff8114e332>] __vmalloc+0x22/0x30
- [ 8747.962437] [<ffffffffa013ee11>] kv_alloc.isra.8+0x51/0x60 [spl]
- [ 8747.962451] [<ffffffffa013ee60>] spl_cache_grow_work+0x40/0x430 [spl]
- [ 8747.962468] [<ffffffffa0144626>] taskq_thread+0x236/0x4b0 [spl]
- [ 8747.962480] [<ffffffff81060910>] ? try_to_wake_up+0x200/0x200
- [ 8747.962495] [<ffffffffa01443f0>] ? task_done+0x160/0x160 [spl]
- [ 8747.962504] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [ 8747.962515] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [ 8747.962524] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [ 8747.962534] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- =================================================================
- After I could ssh into the box again, more appeared looking like this:
- =================================================================
- [109011.999713] spl_kmem_cache/: page allocation failure: order:0, mode:0x20
- [109011.999715] Pid: 575, comm: spl_kmem_cache/ Tainted: P C O 3.2.0-55-generic #85-Ubuntu
- [109011.999716] Call Trace:
- [109011.999721] [<ffffffff8111eaf6>] warn_alloc_failed+0xf6/0x150
- [109011.999723] [<ffffffff81122887>] __alloc_pages_nodemask+0x6d7/0x8f0
- [109011.999726] [<ffffffff81159d36>] alloc_pages_current+0xb6/0x120
- [109011.999729] [<ffffffff81162eb6>] allocate_slab+0x196/0x1a0
- [109011.999731] [<ffffffff81162ef0>] new_slab+0x30/0x180
- [109011.999734] [<ffffffff8164d68e>] __slab_alloc+0x165/0x269
- [109011.999737] [<ffffffff811a33a4>] ? __bdi_start_writeback+0x44/0x110
- [109011.999740] [<ffffffff8117cb31>] ? put_super+0x31/0x40
- [109011.999741] [<ffffffff8117cc12>] ? drop_super+0x22/0x30
- [109011.999743] [<ffffffff811a33a4>] ? __bdi_start_writeback+0x44/0x110
- [109011.999745] [<ffffffff811656e6>] kmem_cache_alloc_trace+0x136/0x140
- [109011.999748] [<ffffffff8112b976>] ? shrink_slab+0x1d6/0x300
- [109011.999749] [<ffffffff811a33a4>] __bdi_start_writeback+0x44/0x110
- [109011.999752] [<ffffffff811a5800>] wakeup_flusher_threads+0x90/0xb0
- [109011.999753] [<ffffffff8112dff0>] do_try_to_free_pages+0xd0/0x2d0
- [109011.999755] [<ffffffff8112e4cb>] try_to_free_pages+0xab/0x170
- [109011.999757] [<ffffffff8112274e>] __alloc_pages_nodemask+0x59e/0x8f0
- [109011.999760] [<ffffffff8101bdf9>] ? sched_clock+0x9/0x10
- [109011.999762] [<ffffffff81159d36>] alloc_pages_current+0xb6/0x120
- [109011.999763] [<ffffffff8111db1e>] __get_free_pages+0xe/0x40
- [109011.999771] [<ffffffffa013edef>] kv_alloc.isra.8+0x2f/0x60 [spl]
- [109011.999775] [<ffffffffa013ee60>] spl_cache_grow_work+0x40/0x430 [spl]
- [109011.999779] [<ffffffffa0144626>] taskq_thread+0x236/0x4b0 [spl]
- [109011.999782] [<ffffffff81060910>] ? try_to_wake_up+0x200/0x200
- [109011.999785] [<ffffffffa01443f0>] ? task_done+0x160/0x160 [spl]
- [109011.999788] [<ffffffff8108b67c>] kthread+0x8c/0xa0
- [109011.999791] [<ffffffff8166b974>] kernel_thread_helper+0x4/0x10
- [109011.999793] [<ffffffff8108b5f0>] ? flush_kthread_worker+0xa0/0xa0
- [109011.999794] [<ffffffff8166b970>] ? gs_change+0x13/0x13
- [109011.999795] Mem-Info:
- [109011.999796] Node 0 DMA per-cpu:
- [109011.999797] CPU 0: hi: 0, btch: 1 usd: 0
- [109011.999798] CPU 1: hi: 0, btch: 1 usd: 0
- [109011.999799] CPU 2: hi: 0, btch: 1 usd: 0
- [109011.999800] CPU 3: hi: 0, btch: 1 usd: 0
- [109011.999801] CPU 4: hi: 0, btch: 1 usd: 0
- [109011.999802] CPU 5: hi: 0, btch: 1 usd: 0
- [109011.999803] CPU 6: hi: 0, btch: 1 usd: 0
- [109011.999804] CPU 7: hi: 0, btch: 1 usd: 0
- [109011.999805] Node 0 DMA32 per-cpu:
- [109011.999806] CPU 0: hi: 186, btch: 31 usd: 0
- [109011.999807] CPU 1: hi: 186, btch: 31 usd: 0
- [109011.999808] CPU 2: hi: 186, btch: 31 usd: 0
- [109011.999809] CPU 3: hi: 186, btch: 31 usd: 0
- [109011.999810] CPU 4: hi: 186, btch: 31 usd: 0
- [109011.999811] CPU 5: hi: 186, btch: 31 usd: 0
- [109011.999812] CPU 6: hi: 186, btch: 31 usd: 0
- [109011.999813] CPU 7: hi: 186, btch: 31 usd: 0
- [109011.999813] Node 0 Normal per-cpu:
- [109011.999814] CPU 0: hi: 186, btch: 31 usd: 0
- [109011.999815] CPU 1: hi: 186, btch: 31 usd: 0
- [109011.999816] CPU 2: hi: 186, btch: 31 usd: 0
- [109011.999817] CPU 3: hi: 186, btch: 31 usd: 0
- [109011.999818] CPU 4: hi: 186, btch: 31 usd: 0
- [109011.999819] CPU 5: hi: 186, btch: 31 usd: 0
- [109011.999820] CPU 6: hi: 186, btch: 31 usd: 0
- [109011.999821] CPU 7: hi: 186, btch: 31 usd: 0
- [109011.999823] active_anon:8806 inactive_anon:1891 isolated_anon:32
- [109011.999824] active_file:0 inactive_file:55 isolated_file:0
- [109011.999824] unevictable:0 dirty:1 writeback:0 unstable:0
- [109011.999825] free:0 slab_reclaimable:44815 slab_unreclaimable:1247993
- [109011.999825] mapped:7 shmem:141 pagetables:909 bounce:0
- [109011.999826] Node 0 DMA free:0kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15636kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:15892kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
- [109011.999831] lowmem_reserve[]: 0 1936 64432 64432
- [109011.999833] Node 0 DMA32 free:0kB min:2028kB low:2532kB high:3040kB active_anon:0kB inactive_anon:4kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1982644kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:5644kB slab_unreclaimable:398420kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:36615988 all_unreclaimable? no
- [109011.999838] lowmem_reserve[]: 0 0 62496 62496
- [109011.999839] Node 0 Normal free:0kB min:65536kB low:81920kB high:98304kB active_anon:35224kB inactive_anon:7560kB active_file:0kB inactive_file:220kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:63995904kB mlocked:0kB dirty:4kB writeback:0kB mapped:24kB shmem:564kB slab_reclaimable:173616kB slab_unreclaimable:4577660kB kernel_stack:3112kB pagetables:3636kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:13106092728 all_unreclaimable? no
- [109011.999845] lowmem_reserve[]: 0 0 0 0
- [109011.999846] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
- [109011.999851] Node 0 DMA32: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
- [109011.999855] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
- [109011.999859] 225 total pagecache pages
- [109011.999860] 0 pages in swap cache
- [109011.999861] Swap cache stats: add 0, delete 0, find 0/0
- [109011.999861] Free swap = 488124kB
- [109011.999862] Total swap = 488124kB
- [109012.094249] 16777200 pages RAM
- [109012.094250] 290233 pages reserved
- [109012.094251] 954 pages shared
- [109012.094252] 12878528 pages non-shared
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement