API tools faq
paste
Guest User

Untitled

a guest
May 14th, 2024
120
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 20.33 KB | None | 0 0
raw download clone embed print report
  1. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: Internal error detected:
  2. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[00]: ffffffff
  3. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[01]: ffffffff
  4. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[02]: ffffffff
  5. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[03]: ffffffff
  6. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[04]: ffffffff
  7. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[05]: ffffffff
  8. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[06]: ffffffff
  9. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[07]: ffffffff
  10. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[08]: ffffffff
  11. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[09]: ffffffff
  12. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[0a]: ffffffff
  13. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[0b]: ffffffff
  14. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[0c]: ffffffff
  15. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[0d]: ffffffff
  16. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[0e]: ffffffff
  17. May 13 02:27:49 VHOST1 kernel: mlx4_core 0000:02:00.0: buf[0f]: ffffffff
  18. May 13 02:27:50 VHOST1 corosync[1927]: [KNET ] link: host: 2 link: 0 is down
  19. May 13 02:27:50 VHOST1 corosync[1927]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
  20. May 13 02:27:50 VHOST1 corosync[1927]: [KNET ] host: host: 2 has no active links
  21. May 13 02:27:51 VHOST1 corosync[1927]: [TOTEM ] Token has not been received in 2737 ms
  22. May 13 02:27:52 VHOST1 corosync[1927]: [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms), waiting 4380ms for consensus.
  23. May 13 02:27:56 VHOST1 corosync[1927]: [QUORUM] Sync members[1]: 1
  24. May 13 02:27:56 VHOST1 corosync[1927]: [QUORUM] Sync left[1]: 2
  25. May 13 02:27:56 VHOST1 corosync[1927]: [VOTEQ ] waiting for quorum device Qdevice poll (but maximum for 30000 ms)
  26. May 13 02:27:56 VHOST1 corosync[1927]: [TOTEM ] A new membership (1.1095) was formed. Members left: 2
  27. May 13 02:27:56 VHOST1 corosync[1927]: [TOTEM ] Failed to receive the leave message. failed: 2
  28. May 13 02:27:56 VHOST1 pmxcfs[1761]: [dcdb] notice: members: 1/1761
  29. May 13 02:27:56 VHOST1 pmxcfs[1761]: [status] notice: members: 1/1761
  30. May 13 02:27:58 VHOST1 kernel: mlx4_core 0000:02:00.0: mlx4_cmd_post:cmd_pending failed
  31. May 13 02:27:58 VHOST1 kernel: mlx4_core 0000:02:00.0: Could not post command 0x49: ret=-5, in_param=0x0, in_mod=0x2, op_mod=0x0
  32. May 13 02:27:58 VHOST1 kernel: mlx4_core 0000:02:00.0: device is going to be reset
  33. May 13 02:27:58 VHOST1 kernel: mlx4_core 0000:02:00.0: crdump: FW doesn't support health buffer access, skipping
  34. May 13 02:28:01 VHOST1 CRON[62598]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
  35. May 13 02:28:01 VHOST1 CRON[62599]: (root) CMD ( /root/autopower.sh)
  36. May 13 02:28:01 VHOST1 CRON[62598]: pam_unix(cron:session): session closed for user root
  37. May 13 02:28:01 VHOST1 pmxcfs[1761]: [dcdb] notice: cpg_send_message retry 10
  38. May 13 02:28:02 VHOST1 pmxcfs[1761]: [dcdb] notice: cpg_send_message retry 20
  39. May 13 02:28:03 VHOST1 pmxcfs[1761]: [dcdb] notice: cpg_send_message retry 30
  40. May 13 02:28:03 VHOST1 corosync-qdevice[1948]: Server didn't send echo reply message on time
  41. May 13 02:28:03 VHOST1 corosync[1927]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
  42. May 13 02:28:03 VHOST1 corosync[1927]: [QUORUM] Members[1]: 1
  43. May 13 02:28:03 VHOST1 corosync[1927]: [MAIN ] Completed service synchronization, ready to provide service.
  44. May 13 02:28:03 VHOST1 pmxcfs[1761]: [status] notice: node lost quorum
  45. May 13 02:28:03 VHOST1 pmxcfs[1761]: [dcdb] crit: received write while not quorate - trigger resync
  46. May 13 02:28:03 VHOST1 pmxcfs[1761]: [dcdb] notice: cpg_send_message retried 32 times
  47. May 13 02:28:03 VHOST1 pmxcfs[1761]: [dcdb] crit: leaving CPG group
  48. May 13 02:28:03 VHOST1 pve-ha-lrm[2708]: lost lock 'ha_agent_VHOST1_lock - cfs lock update failed - Operation not permitted
  49. May 13 02:28:03 VHOST1 pve-ha-lrm[2708]: status change active => lost_agent_lock
  50. May 13 02:28:04 VHOST1 pmxcfs[1761]: [dcdb] notice: start cluster connection
  51. May 13 02:28:04 VHOST1 pmxcfs[1761]: [dcdb] crit: cpg_join failed: 14
  52. May 13 02:28:04 VHOST1 pmxcfs[1761]: [dcdb] crit: can't initialize service
  53. May 13 02:28:04 VHOST1 pve-ha-crm[2138]: lost lock 'ha_manager_lock - cfs lock update failed - Device or resource busy
  54. May 13 02:28:04 VHOST1 pve-ha-crm[2138]: status change master => lost_manager_lock
  55. May 13 02:28:04 VHOST1 pve-ha-crm[2138]: watchdog closed (disabled)
  56. May 13 02:28:04 VHOST1 pve-ha-crm[2138]: status change lost_manager_lock => wait_for_quorum
  57. May 13 02:28:08 VHOST1 kernel: mlx4_core 0000:02:00.0: Failed to obtain HW semaphore, aborting
  58. May 13 02:28:08 VHOST1 kernel: mlx4_core 0000:02:00.0: Fail to reset HCA
  59. May 13 02:28:08 VHOST1 kernel: ------------[ cut here ]------------
  60. May 13 02:28:08 VHOST1 kernel: kernel BUG at drivers/net/ethernet/mellanox/mlx4/catas.c:191!
  61. May 13 02:28:08 VHOST1 kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
  62. May 13 02:28:08 VHOST1 kernel: CPU: 4 PID: 52956 Comm: kworker/u64:2 Tainted: P OE 6.5.13-5-pve #1
  63. May 13 02:28:08 VHOST1 kernel: Hardware name: iXsystems 4236Q/X9DRH-7TF/7F/iTF/iF, BIOS 3.3 07/13/2018
  64. May 13 02:28:08 VHOST1 kernel: Workqueue: mlx4_health catas_reset [mlx4_core]
  65. May 13 02:28:08 VHOST1 kernel: RIP: 0010:mlx4_enter_error_state.part.0+0x232/0x310 [mlx4_core]
  66. May 13 02:28:08 VHOST1 kernel: Code: 03 48 c7 c6 50 fe 7e c0 48 8b 38 48 81 c7 d0 00 00 00 e8 21 db 4e ef 48 8b 03 48 8b 10 83 ba c8 00 00 00 01 0f 85 6f fe ff ff <0f> 0b 31 c0 31 f6 48 8d 55 d0 66 89 45 d0 e8 eb bf 2e ef 85 c0 75
  67. May 13 02:28:08 VHOST1 kernel: RSP: 0018:ffffb2ff059c3df8 EFLAGS: 00010246
  68. May 13 02:28:08 VHOST1 kernel: RAX: ffff90de000f8600 RBX: ffff90c69c5401a0 RCX: 0000000000000000
  69. May 13 02:28:08 VHOST1 kernel: RDX: ffff90c688a7f000 RSI: 0000000000000000 RDI: 0000000000000000
  70. May 13 02:28:08 VHOST1 kernel: RBP: ffffb2ff059c3e28 R08: 0000000000000000 R09: 0000000000000000
  71. May 13 02:28:08 VHOST1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff90de000f8660
  72. May 13 02:28:08 VHOST1 kernel: R13: ffff90de000fd900 R14: ffff90de000fd905 R15: ffff90c6922c9080
  73. May 13 02:28:08 VHOST1 kernel: FS: 0000000000000000(0000) GS:ffff90dd9fb00000(0000) knlGS:0000000000000000
  74. May 13 02:28:08 VHOST1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  75. May 13 02:28:08 VHOST1 kernel: CR2: 00005f8c21fef000 CR3: 000000016915c002 CR4: 00000000001726e0
  76. May 13 02:28:08 VHOST1 kernel: Call Trace:
  77. May 13 02:28:08 VHOST1 kernel: <TASK>
  78. May 13 02:28:08 VHOST1 kernel: ? show_regs+0x6d/0x80
  79. May 13 02:28:08 VHOST1 kernel: ? die+0x37/0xa0
  80. May 13 02:28:08 VHOST1 kernel: ? do_trap+0xd4/0xf0
  81. May 13 02:28:08 VHOST1 kernel: ? do_error_trap+0x71/0xb0
  82. May 13 02:28:08 VHOST1 kernel: ? mlx4_enter_error_state.part.0+0x232/0x310 [mlx4_core]
  83. May 13 02:28:08 VHOST1 kernel: ? exc_invalid_op+0x52/0x80
  84. May 13 02:28:08 VHOST1 kernel: ? mlx4_enter_error_state.part.0+0x232/0x310 [mlx4_core]
  85. May 13 02:28:08 VHOST1 kernel: ? asm_exc_invalid_op+0x1b/0x20
  86. May 13 02:28:08 VHOST1 kernel: ? mlx4_enter_error_state.part.0+0x232/0x310 [mlx4_core]
  87. May 13 02:28:08 VHOST1 kernel: ? mlx4_enter_error_state.part.0+0x279/0x310 [mlx4_core]
  88. May 13 02:28:08 VHOST1 kernel: catas_reset+0x28/0xa0 [mlx4_core]
  89. May 13 02:28:08 VHOST1 kernel: process_one_work+0x23e/0x450
  90. May 13 02:28:08 VHOST1 kernel: worker_thread+0x50/0x3f0
  91. May 13 02:28:08 VHOST1 kernel: ? __pfx_worker_thread+0x10/0x10
  92. May 13 02:28:08 VHOST1 kernel: kthread+0xf2/0x120
  93. May 13 02:28:08 VHOST1 kernel: ? __pfx_kthread+0x10/0x10
  94. May 13 02:28:08 VHOST1 kernel: ret_from_fork+0x47/0x70
  95. May 13 02:28:08 VHOST1 kernel: ? __pfx_kthread+0x10/0x10
  96. May 13 02:28:08 VHOST1 kernel: ret_from_fork_asm+0x1b/0x30
  97. May 13 02:28:08 VHOST1 kernel: </TASK>
  98. May 13 02:28:08 VHOST1 kernel: Modules linked in: tcp_diag inet_diag rbd libceph rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables sctp ip6_udp_tunnel udp_tunnel msr nf_tables nvme_fabrics xt_nat xt_tcpudp xt_conntrack>
  99. May 13 02:28:08 VHOST1 kernel: intel_cstate ipmi_devintf pcspkr drm_kms_helper mei usbserial ioatdma ipmi_msghandler input_leds joydev mac_hid vhost_net vhost vhost_iotlb tap vhci_hcd usbip_core vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 zfs(>
  100. May 13 02:28:08 VHOST1 kernel: ---[ end trace 0000000000000000 ]---
  101. May 13 02:28:08 VHOST1 kernel: RIP: 0010:mlx4_enter_error_state.part.0+0x232/0x310 [mlx4_core]
  102. May 13 02:28:08 VHOST1 kernel: Code: 03 48 c7 c6 50 fe 7e c0 48 8b 38 48 81 c7 d0 00 00 00 e8 21 db 4e ef 48 8b 03 48 8b 10 83 ba c8 00 00 00 01 0f 85 6f fe ff ff <0f> 0b 31 c0 31 f6 48 8d 55 d0 66 89 45 d0 e8 eb bf 2e ef 85 c0 75
  103. May 13 02:28:08 VHOST1 kernel: RSP: 0018:ffffb2ff059c3df8 EFLAGS: 00010246
  104. May 13 02:28:08 VHOST1 kernel: RAX: ffff90de000f8600 RBX: ffff90c69c5401a0 RCX: 0000000000000000
  105. May 13 02:28:08 VHOST1 kernel: RDX: ffff90c688a7f000 RSI: 0000000000000000 RDI: 0000000000000000
  106. May 13 02:28:08 VHOST1 kernel: RBP: ffffb2ff059c3e28 R08: 0000000000000000 R09: 0000000000000000
  107. May 13 02:28:08 VHOST1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff90de000f8660
  108. May 13 02:28:08 VHOST1 kernel: R13: ffff90de000fd900 R14: ffff90de000fd905 R15: ffff90c6922c9080
  109. May 13 02:28:08 VHOST1 kernel: FS: 0000000000000000(0000) GS:ffff90dd9fb00000(0000) knlGS:0000000000000000
  110. May 13 02:28:08 VHOST1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  111. May 13 02:28:08 VHOST1 kernel: CR2: 00005f8c21fef000 CR3: 000000016915c002 CR4: 00000000001726e0
  112. May 13 02:28:09 VHOST1 pvescheduler[62579]: replication: cfs-lock 'file-replication_cfg' error: no quorum!
  113. May 13 02:28:09 VHOST1 pvescheduler[62580]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
  114. May 13 02:28:09 VHOST1 pvestatd[2067]: storage 'VMs-Archive' is not online
  115. May 13 02:28:10 VHOST1 pmxcfs[1761]: [dcdb] notice: members: 1/1761
  116. May 13 02:28:10 VHOST1 pmxcfs[1761]: [dcdb] notice: all data is up to date
  117. May 13 02:28:10 VHOST1 kernel: libceph: mon1 (1)172.16.0.2:6789 session lost, hunting for new mon
  118. May 13 02:28:12 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:12.032-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  119. May 13 02:28:12 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:12.032-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  120. May 13 02:28:12 VHOST1 kernel: libceph: mon0 (1)172.16.0.1:6789 socket closed (con state OPEN)
  121. May 13 02:28:13 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:13.076-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  122. May 13 02:28:13 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:13.076-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  123. May 13 02:28:14 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:14.116-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  124. May 13 02:28:14 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:14.116-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  125. May 13 02:28:14 VHOST1 corosync-qdevice[1948]: Connect timeout
  126. May 13 02:28:15 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:15.144-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  127. May 13 02:28:15 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:15.144-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  128. May 13 02:28:16 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:16.188-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  129. May 13 02:28:16 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:16.188-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  130. May 13 02:28:16 VHOST1 kernel: ------------[ cut here ]------------
  131. May 13 02:28:16 VHOST1 kernel: NETDEV WATCHDOG: enp2s0 (mlx4_core): transmit queue 0 timed out 19476 ms
  132. May 13 02:28:16 VHOST1 kernel: WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x260/0x270
  133. May 13 02:28:16 VHOST1 kernel: Modules linked in: tcp_diag inet_diag rbd libceph rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables sctp ip6_udp_tunnel udp_tunnel msr nf_tables nvme_fabrics xt_nat xt_tcpudp xt_conntrack>
  134. May 13 02:28:16 VHOST1 kernel: intel_cstate ipmi_devintf pcspkr drm_kms_helper mei usbserial ioatdma ipmi_msghandler input_leds joydev mac_hid vhost_net vhost vhost_iotlb tap vhci_hcd usbip_core vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 zfs(>
  135. May 13 02:28:16 VHOST1 kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: P D OE 6.5.13-5-pve #1
  136. May 13 02:28:16 VHOST1 kernel: Hardware name: iXsystems 4236Q/X9DRH-7TF/7F/iTF/iF, BIOS 3.3 07/13/2018
  137. May 13 02:28:16 VHOST1 kernel: RIP: 0010:dev_watchdog+0x260/0x270
  138. May 13 02:28:16 VHOST1 kernel: Code: ff ff 48 89 df c6 05 68 ef 77 01 01 e8 19 7d f9 ff 44 8b 45 cc 44 89 f9 48 89 de 48 89 c2 48 c7 c7 98 bc c3 b0 e8 b0 74 33 ff <0f> 0b e9 1d ff ff ff 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90
  139. May 13 02:28:16 VHOST1 kernel: RSP: 0018:ffffb2fe883f8e40 EFLAGS: 00010246
  140. May 13 02:28:16 VHOST1 kernel: RAX: 0000000000000000 RBX: ffff90c6b38a0000 RCX: 0000000000000000
  141. May 13 02:28:16 VHOST1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
  142. May 13 02:28:16 VHOST1 kernel: RBP: ffffb2fe883f8e78 R08: 0000000000000000 R09: 0000000000000000
  143. May 13 02:28:16 VHOST1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff90c6b38a04c8
  144. May 13 02:28:16 VHOST1 kernel: R13: ffff90c6b38a041c R14: 0000000000000000 R15: 0000000000000000
  145. May 13 02:28:16 VHOST1 kernel: FS: 0000000000000000(0000) GS:ffff90dd9fb40000(0000) knlGS:0000000000000000
  146. May 13 02:28:16 VHOST1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  147. May 13 02:28:16 VHOST1 kernel: CR2: 00005f8bfdf5b400 CR3: 00000002492cc001 CR4: 00000000001726e0
  148. May 13 02:28:16 VHOST1 kernel: Call Trace:
  149. May 13 02:28:16 VHOST1 kernel: <IRQ>
  150. May 13 02:28:16 VHOST1 kernel: ? show_regs+0x6d/0x80
  151. May 13 02:28:16 VHOST1 kernel: ? __warn+0x89/0x160
  152. May 13 02:28:16 VHOST1 kernel: ? dev_watchdog+0x260/0x270
  153. May 13 02:28:16 VHOST1 kernel: ? report_bug+0x17e/0x1b0
  154. May 13 02:28:16 VHOST1 kernel: ? irq_work_queue+0x2f/0x70
  155. May 13 02:28:16 VHOST1 kernel: ? handle_bug+0x46/0x90
  156. May 13 02:28:16 VHOST1 kernel: ? exc_invalid_op+0x18/0x80
  157. May 13 02:28:16 VHOST1 kernel: ? asm_exc_invalid_op+0x1b/0x20
  158. May 13 02:28:16 VHOST1 kernel: ? dev_watchdog+0x260/0x270
  159. May 13 02:28:16 VHOST1 kernel: ? __pfx_dev_watchdog+0x10/0x10
  160. May 13 02:28:16 VHOST1 kernel: call_timer_fn+0x2c/0x160
  161. May 13 02:28:16 VHOST1 kernel: ? __pfx_dev_watchdog+0x10/0x10
  162. May 13 02:28:16 VHOST1 kernel: __run_timers+0x259/0x310
  163. May 13 02:28:16 VHOST1 kernel: run_timer_softirq+0x1d/0x40
  164. May 13 02:28:16 VHOST1 kernel: __do_softirq+0xd4/0x303
  165. May 13 02:28:16 VHOST1 kernel: __irq_exit_rcu+0x75/0xa0
  166. May 13 02:28:16 VHOST1 kernel: irq_exit_rcu+0xe/0x20
  167. May 13 02:28:16 VHOST1 kernel: sysvec_apic_timer_interrupt+0x92/0xd0
  168. May 13 02:28:16 VHOST1 kernel: </IRQ>
  169. May 13 02:28:16 VHOST1 kernel: <TASK>
  170. May 13 02:28:16 VHOST1 kernel: asm_sysvec_apic_timer_interrupt+0x1b/0x20
  171. May 13 02:28:16 VHOST1 kernel: RIP: 0010:cpuidle_enter_state+0xce/0x470
  172. May 13 02:28:16 VHOST1 kernel: Code: c5 0f ff e8 64 f6 ff ff 8b 53 04 49 89 c6 0f 1f 44 00 00 31 ff e8 c2 c1 0e ff 80 7d d7 00 0f 85 e7 01 00 00 fb 0f 1f 44 00 00 <45> 85 ff 0f 88 83 01 00 00 49 63 d7 4c 89 f1 48 8d 04 52 48 8d 04
  173. May 13 02:28:16 VHOST1 kernel: RSP: 0018:ffffb2fe8821fe50 EFLAGS: 00000246
  174. May 13 02:28:16 VHOST1 kernel: RAX: 0000000000000000 RBX: ffffd2e67ef72100 RCX: 0000000000000000
  175. May 13 02:28:16 VHOST1 kernel: RDX: 0000000000000005 RSI: 0000000000000000 RDI: 0000000000000000
  176. May 13 02:28:16 VHOST1 kernel: RBP: ffffb2fe8821fe88 R08: 0000000000000000 R09: 0000000000000000
  177. May 13 02:28:16 VHOST1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
  178. May 13 02:28:16 VHOST1 kernel: R13: ffffffffb166a2e0 R14: 0000050b9c4629e2 R15: 0000000000000002
  179. May 13 02:28:16 VHOST1 kernel: cpuidle_enter+0x2e/0x50
  180. May 13 02:28:16 VHOST1 kernel: call_cpuidle+0x23/0x60
  181. May 13 02:28:16 VHOST1 kernel: do_idle+0x202/0x260
  182. May 13 02:28:16 VHOST1 kernel: cpu_startup_entry+0x2a/0x30
  183. May 13 02:28:16 VHOST1 kernel: start_secondary+0x119/0x140
  184. May 13 02:28:16 VHOST1 kernel: secondary_startup_64_no_verify+0x190/0x19b
  185. May 13 02:28:16 VHOST1 kernel: </TASK>
  186. May 13 02:28:16 VHOST1 kernel: ---[ end trace 0000000000000000 ]---
  187. May 13 02:28:16 VHOST1 kernel: mlx4_en: enp2s0: TX timeout on queue: 0, QP: 0x208, CQ: 0xc2, Cons: 0xde997, Prod: 0xdea72
  188. May 13 02:28:17 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:17.152-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  189. May 13 02:28:17 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:17.152-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  190. May 13 02:28:18 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:18.164-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  191. May 13 02:28:18 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:18.164-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  192. May 13 02:28:19 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:19.192-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  193. May 13 02:28:19 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:19.192-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  194. May 13 02:28:20 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:20.204-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.2:6804 osd.1 since back 2024-05-13T02:27:45.861141-0400 front 2024-05-13T02:27:45.861154-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  195. May 13 02:28:20 VHOST1 ceph-osd[2128]: 2024-05-13T02:28:20.204-0400 7aa1e1cce6c0 -1 osd.0 1300 heartbeat_check: no reply from 172.16.0.3:6804 osd.3 since back 2024-05-13T02:27:45.861227-0400 front 2024-05-13T02:27:45.861245-0400 (oldest deadline 2024-05-13T02:28:11.160966-0400)
  196. -- Boot 0708190a48e64b92b41a74e0a12e8ca4 --
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!