[ 30 minutes after reboot ]
[ BIOS 1.2.11 ]
 19:14:07 server2 kernel: ------------[ cut here ]------------
 19:14:07 server2 kernel: WARNING: CPU: 38 PID: 0 at net/sched/sch_generic.c:300 dev_watchdog+0x248/0x260
 19:14:07 server2 kernel: NETDEV WATCHDOG: em2 (bnxt_en): transmit queue 0 timed out
 19:14:07 server2 kernel: Modules linked in: mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase nf_log_ipv4 nf_log_common xt_LOG xt_recent dell_rbu xt_tcpmss nf_conntrack_ipv4 nf_defrag_ipv4 iptable_mangle xt_conntrack nf_conntrack iptable_filter vfat fat skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel iTCO_wdt iTCO_vendor_support lrw gf128mul glue_helper ablk_helper dcdbas cryptd ipmi_ssif sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler nfit acpi_pad acpi_power_meter libnvdimm i2c_i801 mei_me mei lpc_ich shpchp tcp_htcp ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci
 19:14:07 server2 kernel: libahci tg3 libata bnxt_en i2c_core ptp pps_core megaraid_sas(OE)
 19:14:07 server2 kernel: CPU: 38 PID: 0 Comm: swapper/38 Tainted: G           OE  ------------   3.10.0-693.21.1.el7.x86_64 #1
 19:14:07 server2 kernel: Hardware name: Dell Inc. PowerEdge R640/008R9M, BIOS 1.2.11 10/19/2017
 19:14:07 server2 kernel: Call Trace:
 19:14:07 server2 kernel: <IRQ>  [<ffffffff816ae7c8>] dump_stack+0x19/0x1b
 19:14:07 server2 kernel: [<ffffffff8108ae58>] __warn+0xd8/0x100
 19:14:07 server2 kernel: [<ffffffff8108aedf>] warn_slowpath_fmt+0x5f/0x80
 19:14:07 server2 kernel: [<ffffffff815b8bf8>] dev_watchdog+0x248/0x260
 19:14:07 server2 kernel: [<ffffffff815b89b0>] ? dev_deactivate_queue.constprop.33+0x60/0x60
 19:14:07 server2 kernel: [<ffffffff8109a878>] call_timer_fn+0x38/0x110
 19:14:07 server2 kernel: [<ffffffff815b89b0>] ? dev_deactivate_queue.constprop.33+0x60/0x60
 19:14:07 server2 kernel: [<ffffffff8109cd9d>] run_timer_softirq+0x22d/0x310
 19:14:07 server2 kernel: [<ffffffff81094035>] __do_softirq+0xf5/0x280
 19:14:07 server2 kernel: [<ffffffff816c3afc>] call_softirq+0x1c/0x30
 19:14:07 server2 kernel: [<ffffffff8102d435>] do_softirq+0x65/0xa0
 19:14:07 server2 kernel: [<ffffffff810943b5>] irq_exit+0x105/0x110
 19:14:07 server2 kernel: [<ffffffff816c4e78>] smp_apic_timer_interrupt+0x48/0x60
 19:14:07 server2 kernel: [<ffffffff816c1732>] apic_timer_interrupt+0x162/0x170
 19:14:07 server2 kernel: <EOI>  [<ffffffff81530b07>] ? cpuidle_enter_state+0x57/0xd0
 19:14:07 server2 kernel: [<ffffffff81530c5e>] cpuidle_idle_call+0xde/0x230
 19:14:07 server2 kernel: [<ffffffff81034f8e>] arch_cpu_idle+0xe/0x40
 19:14:07 server2 kernel: [<ffffffff810eb6da>] cpu_startup_entry+0x14a/0x1c0
 19:14:07 server2 kernel: [<ffffffff81052222>] start_secondary+0x1f2/0x270
 19:14:07 server2 kernel: ---[ end trace 2f384ae091eea882 ]---
 19:14:07 server2 kernel: bnxt_en 0000:19:00.1 em2: TX timeout detected, starting reset task!
 19:14:08 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:08 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free tx failed. rc:-1
 19:14:08 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcb6} len:0
 19:14:09 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:09 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free tx failed. rc:-1
 19:14:10 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcb7} len:0
 19:14:10 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:10 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free tx failed. rc:-1
 19:14:11 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcb8} len:0
 19:14:11 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:11 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:11 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:11 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:12 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcb9} len:0
 19:14:12 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:12 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:13 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcba} len:0
 19:14:13 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:13 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:14 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcbb} len:0
 19:14:14 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:14 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:15 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcbc} len:0
 19:14:15 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:15 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:16 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcbd} len:0
 19:14:16 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:16 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:17 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcbe} len:0
 19:14:17 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:17 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:18 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcbf} len:0
 19:14:18 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:18 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:19 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc0} len:0
 19:14:19 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:19 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:20 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc1} len:0
 19:14:20 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:20 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:21 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc2} len:0
 19:14:21 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:21 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:22 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc3} len:0
 19:14:22 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:22 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:23 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc4} len:0
 19:14:23 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:23 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:24 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc5} len:0
 19:14:24 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:24 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:25 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc6} len:0
 19:14:25 server2 kernel: bnxt_en 0000:19:00.1 em2: Resp cmpl intr err msg: 0x51
 19:14:25 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free rx failed. rc:-1
 19:14:26 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xcde} len:0
 19:14:26 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:26 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc7} len:0
 19:14:26 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xcdf} len:0
 19:14:26 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:27 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc8} len:0
 19:14:27 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xce0} len:0
 19:14:27 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:28 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcc9} len:0
 19:14:28 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xce1} len:0
 19:14:28 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:29 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcca} len:0
 19:14:29 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xce2} len:0
 19:14:29 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:30 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xccb} len:0
 19:14:30 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xce3} len:0
 19:14:30 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:31 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xccc} len:0
 19:14:31 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xce4} len:0
 19:14:31 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:32 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xccd} len:0
 19:14:32 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x51 0xce5} len:0
 19:14:32 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm_ring_free cp failed. rc:-1
 19:14:33 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcce} len:0
 19:14:33 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x61 0xce6} len:0
 19:14:34 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xccf} len:0
 19:14:34 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x50 0xce7} len:0
 19:14:34 server2 kernel: bnxt_en 0000:19:00.1 em2: Invalid ring
 19:14:34 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm ring alloc failure rc: ffffffff
 19:14:35 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd0} len:0
 19:14:35 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x44 0xce8} len:0
 19:14:35 server2 kernel: bnxt_en 0000:19:00.1 em2: hwrm vnic set tpa failure rc for vnic 0: ffffffff
 19:14:36 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd1} len:0
 19:14:36 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0xb1 0xce9} len:0
 19:14:36 server2 kernel: bnxt_en 0000:19:00.1 em2: bnxt_init_nic err: ffffffff
 19:14:36 server2 kernel: bnxt_en 0000:19:00.1 em2: nic open fail (rc: ffffffff)
 19:14:37 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd2} len:0
 19:14:37 server2 kernel: bnxt_en 0000:19:00.1 em2: Error (timeout: 500) msg {0x20 0xcea} len:0
 19:14:38 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd3} len:0
 19:14:39 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd4} len:0
 19:14:40 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd5} len:0
 19:14:41 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd6} len:0
 19:14:42 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd7} len:0
 19:14:43 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd8} len:0
 19:14:44 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcd9} len:0
 19:14:45 server2 kernel: bnxt_en 0000:19:00.0 em1: Error (timeout: 500) msg {0x23 0xcda} len:0
[...]


[ after 20 minutes of reboot ]
[ BIOS 1.2.11 ]
 23:10:50 server1 kernel: ------------[ cut here ]------------
 23:10:50 server1 kernel: WARNING: CPU: 40 PID: 0 at net/sched/sch_generic.c:300 dev_watchdog+0x248/0x260
 23:10:50 server1 kernel: NETDEV WATCHDOG: em2 (bnxt_en): transmit queue 7 timed out
 23:10:50 server1 kernel: Modules linked in: mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase nf_log_ipv4 nf_log_common xt_LOG xt_recent xt_tcpmss dell_rbu nf_conntrack_ipv4 nf_defrag_ipv4 iptable_mangle xt_conntrack nf_conntrack iptable_filter vfat skx_edac fat edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support dcdbas ipmi_ssif pcspkr sg ipmi_si ipmi_devintf ipmi_msghandler nfit acpi_power_meter libnvdimm mei_me mei acpi_pad i2c_i801 lpc_ich shpchp tcp_htcp ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3
 23:10:50 server1 kernel: bnxt_en ahci libahci ptp libata i2c_core pps_core megaraid_sas(OE)
 23:10:50 server1 kernel: CPU: 40 PID: 0 Comm: swapper/40 Tainted: G           OE  ------------   3.10.0-693.21.1.el7.x86_64 #1
 23:10:50 server1 kernel: Hardware name: Dell Inc. PowerEdge R640/008R9M, BIOS 1.2.11 10/19/2017
 23:10:50 server1 kernel: Call Trace:
 23:10:50 server1 kernel: <IRQ>  [<ffffffff816ae7c8>] dump_stack+0x19/0x1b
 23:10:50 server1 kernel: [<ffffffff8108ae58>] __warn+0xd8/0x100
 23:10:50 server1 kernel: [<ffffffff8108aedf>] warn_slowpath_fmt+0x5f/0x80
 23:10:50 server1 kernel: [<ffffffff815b8bf8>] dev_watchdog+0x248/0x260
 23:10:50 server1 kernel: [<ffffffff815b89b0>] ? dev_deactivate_queue.constprop.33+0x60/0x60
 23:10:50 server1 kernel: [<ffffffff8109a878>] call_timer_fn+0x38/0x110
 23:10:50 server1 kernel: [<ffffffff815b89b0>] ? dev_deactivate_queue.constprop.33+0x60/0x60
 23:10:50 server1 kernel: [<ffffffff8109cd9d>] run_timer_softirq+0x22d/0x310
 23:10:50 server1 kernel: [<ffffffff81094035>] __do_softirq+0xf5/0x280
 23:10:50 server1 kernel: [<ffffffff816c3afc>] call_softirq+0x1c/0x30
 23:10:50 server1 kernel: [<ffffffff8102d435>] do_softirq+0x65/0xa0
 23:10:50 server1 kernel: [<ffffffff810943b5>] irq_exit+0x105/0x110
 23:10:50 server1 kernel: [<ffffffff816c4e78>] smp_apic_timer_interrupt+0x48/0x60
 23:10:50 server1 kernel: [<ffffffff816c1732>] apic_timer_interrupt+0x162/0x170
 23:10:50 server1 kernel: <EOI>  [<ffffffff81530b07>] ? cpuidle_enter_state+0x57/0xd0
 23:10:50 server1 kernel: [<ffffffff81530c5e>] cpuidle_idle_call+0xde/0x230
 23:10:50 server1 kernel: [<ffffffff81034f8e>] arch_cpu_idle+0xe/0x40
 23:10:50 server1 kernel: [<ffffffff810eb6da>] cpu_startup_entry+0x14a/0x1c0
 23:10:50 server1 kernel: [<ffffffff81052222>] start_secondary+0x1f2/0x270
 23:10:50 server1 kernel: ---[ end trace 5e119893f17b1c74 ]---
 23:10:50 server1 kernel: bnxt_en 0000:19:00.1 em2: TX timeout detected, starting reset task!
 23:16:50 server1 kernel: bnxt_en 0000:19:00.1 em2: TX timeout detected, starting reset task!
 23:24:10 server1 kernel: bnxt_en 0000:19:00.1 em2: TX timeout detected, starting reset task!