Dec 03 04:00:36 ceph06 bash[2643372]: debug     -3> 2023-12-03T04:00:36.686+0000 7f08520e2700 -1 bdev(0x55f02a28a400 /var/lib/ceph/osd/ceph-56/block) _aio_thread got r=-1 ((1) Operation not permitted)
Dec 03 04:00:36 ceph06 bash[2643372]: debug     -2> 2023-12-03T04:00:36.686+0000 7f084f0dc700  3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5.
Dec 03 04:00:36 ceph06 bash[2643372]: debug     -1> 2023-12-03T04:00:36.690+0000 7f08520e2700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f08520e2700 time 2023-12-03T04:00:36.689614+0000
Dec 03 04:00:36 ceph06 bash[2643372]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14/rpm/el8/BUILD/ceph-16.2.14/src/blk/kernel/KernelDevice.cc: 604: ceph_abort_msg("Unexpected IO error. This may suggest HW issue. Please check your dmesg!")
Dec 03 04:00:36 ceph06 bash[2643372]:  ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)
Dec 03 04:00:36 ceph06 bash[2643372]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x55f01d9493fa]
Dec 03 04:00:36 ceph06 bash[2643372]:  2: (KernelDevice::_aio_thread()+0x1285) [0x55f01e4b5c15]
Dec 03 04:00:36 ceph06 bash[2643372]:  3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f01e4c0ee1]
Dec 03 04:00:36 ceph06 bash[2643372]:  4: /lib64/libpthread.so.0(+0x81ca) [0x7f085e2fe1ca]
Dec 03 04:00:36 ceph06 bash[2643372]:  5: clone()
Dec 03 04:00:36 ceph06 bash[2643372]: debug      0> 2023-12-03T04:00:36.690+0000 7f08520e2700 -1 *** Caught signal (Aborted) **
Dec 03 04:00:36 ceph06 bash[2643372]:  in thread 7f08520e2700 thread_name:bstore_aio
Dec 03 04:00:36 ceph06 bash[2643372]:  ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)
Dec 03 04:00:36 ceph06 bash[2643372]:  1: /lib64/libpthread.so.0(+0x12cf0) [0x7f085e308cf0]
Dec 03 04:00:36 ceph06 bash[2643372]:  2: gsignal()
Dec 03 04:00:36 ceph06 bash[2643372]:  3: abort()
Dec 03 04:00:36 ceph06 bash[2643372]:  4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x55f01d9494cb]
Dec 03 04:00:36 ceph06 bash[2643372]:  5: (KernelDevice::_aio_thread()+0x1285) [0x55f01e4b5c15]
Dec 03 04:00:36 ceph06 bash[2643372]:  6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f01e4c0ee1]
Dec 03 04:00:36 ceph06 bash[2643372]:  7: /lib64/libpthread.so.0(+0x81ca) [0x7f085e2fe1ca]
Dec 03 04:00:36 ceph06 bash[2643372]:  8: clone()