API tools faq
paste
KervyN

osd139log

KervyN
Sep 13th, 2022
61
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 73.86 KB | None | 0 0
raw download clone embed print report
  1. 2022-09-12T20:10:35.653+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:10:35.653638) [db/memtable_list.cc:380] [default] Level-0 commit table #81077 started
  2. 2022-09-12T20:10:35.653+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:10:35.654229) [db/memtable_list.cc:428] [default] Level-0 commit table #81077: memtable #1 done
  3. 2022-09-12T20:10:35.653+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:10:35.654246) EVENT_LOG_v1 {"time_micros": 1663006235654238, "job": 3336, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [3, 4, 32, 107, 0, 0, 0], "immutable_memtables": 0}
  4. 2022-09-12T20:10:35.653+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:10:35.654303) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[3 4 32 107 0 0 0] max score 0.87
  5.  
  6. 2022-09-12T20:11:26.648+0200 7fb3df27c700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
  7. 2022-09-12T20:11:26.648+0200 7fb3df27c700 4 rocksdb: [db/db_impl.cc:778]
  8. ** DB Stats **
  9. Uptime(secs): 532203.3 total, 600.0 interval
  10. Cumulative writes: 311M writes, 1766M keys, 311M commit groups, 1.0 writes per commit group, ingest: 463.64 GB, 0.89 MB/s
  11. Cumulative WAL: 311M writes, 136M syncs, 2.29 writes per sync, written: 463.64 GB, 0.89 MB/s
  12. Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
  13. Interval writes: 417K writes, 2318K keys, 417K commit groups, 1.0 writes per commit group, ingest: 624.64 MB, 1.04 MB/s
  14. Interval WAL: 417K writes, 182K syncs, 2.29 writes per sync, written: 0.61 MB, 1.04 MB/s
  15. Interval stall: 00:00:0.000 H:M:S, 0.0 percent
  16.  
  17. ** Compaction Stats [default] **
  18. Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  19. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  20. L0 3/0 115.50 MB 0.8 0.0 0.0 0.0 73.3 73.3 0.0 1.0 0.0 77.1 972.30 800.42 2071 0.469 0 0
  21. L1 4/0 152.46 MB 0.6 166.3 73.1 93.1 130.4 37.2 0.0 1.8 118.6 93.0 1436.12 1126.16 517 2.778 1312M 129M
  22. L2 32/0 1.63 GB 0.9 188.0 37.2 150.9 163.8 12.9 0.1 4.4 76.8 66.9 2505.52 2082.69 575 4.357 2931M 93M
  23. L3 107/0 6.74 GB 0.3 29.2 11.5 17.7 17.7 0.0 0.0 1.5 135.7 82.4 220.17 167.20 165 1.334 450M 318M
  24. Sum 146/0 8.63 GB 0.0 383.5 121.8 261.7 385.1 123.4 0.1 5.3 76.5 76.8 5134.11 4176.46 3328 1.543 4694M 541M
  25. Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 75.9 1.52 1.21 3 0.507 0 0
  26.  
  27. ** Compaction Stats [default] **
  28. Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  29. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  30. Low 0/0 0.00 KB 0.0 383.5 121.8 261.7 311.9 50.2 0.0 0.0 94.4 76.7 4161.81 3376.04 1257 3.311 4694M 541M
  31. High 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.2 73.2 0.0 0.0 0.0 77.1 971.59 800.42 2070 0.469 0 0
  32. User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 78.9 0.71 0.00 1 0.706 0 0
  33. Uptime(secs): 532203.3 total, 600.0 interval
  34. Flush(GB): cumulative 73.254, interval 0.113
  35. AddFile(GB): cumulative 0.000, interval 0.000
  36. AddFile(Total Files): cumulative 0, interval 0
  37. AddFile(L0 Files): cumulative 0, interval 0
  38. AddFile(Keys): cumulative 0, interval 0
  39. Cumulative compaction: 385.13 GB write, 0.74 MB/s write, 383.47 GB read, 0.74 MB/s read, 5134.1 seconds
  40. Interval compaction: 0.11 GB write, 0.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 1.5 seconds
  41. Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
  42.  
  43. ** File Read Latency Histogram By Level [default] **
  44.  
  45. ** Compaction Stats [default] **
  46. Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  47. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  48. L0 3/0 115.50 MB 0.8 0.0 0.0 0.0 73.3 73.3 0.0 1.0 0.0 77.1 972.30 800.42 2071 0.469 0 0
  49. L1 4/0 152.46 MB 0.6 166.3 73.1 93.1 130.4 37.2 0.0 1.8 118.6 93.0 1436.12 1126.16 517 2.778 1312M 129M
  50. L2 32/0 1.63 GB 0.9 188.0 37.2 150.9 163.8 12.9 0.1 4.4 76.8 66.9 2505.52 2082.69 575 4.357 2931M 93M
  51. L3 107/0 6.74 GB 0.3 29.2 11.5 17.7 17.7 0.0 0.0 1.5 135.7 82.4 220.17 167.20 165 1.334 450M 318M
  52. Sum 146/0 8.63 GB 0.0 383.5 121.8 261.7 385.1 123.4 0.1 5.3 76.5 76.8 5134.11 4176.46 3328 1.543 4694M 541M
  53. Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
  54.  
  55. ** Compaction Stats [default] **
  56. Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  57. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  58. Low 0/0 0.00 KB 0.0 383.5 121.8 261.7 311.9 50.2 0.0 0.0 94.4 76.7 4161.81 3376.04 1257 3.311 4694M 541M
  59. High 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.2 73.2 0.0 0.0 0.0 77.1 971.59 800.42 2070 0.469 0 0
  60. User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 78.9 0.71 0.00 1 0.706 0 0
  61. Uptime(secs): 532203.3 total, 0.0 interval
  62. Flush(GB): cumulative 73.254, interval 0.000
  63. AddFile(GB): cumulative 0.000, interval 0.000
  64. AddFile(Total Files): cumulative 0, interval 0
  65. AddFile(L0 Files): cumulative 0, interval 0
  66. AddFile(Keys): cumulative 0, interval 0
  67. Cumulative compaction: 385.13 GB write, 0.74 MB/s write, 383.47 GB read, 0.74 MB/s read, 5134.1 seconds
  68. Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
  69. Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
  70.  
  71. ** File Read Latency Histogram By Level [default] **
  72.  
  73. 2022-09-12T20:12:25.231+0200 7fb3de27a700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5775856320ns
  74. 2022-09-12T20:12:25.236+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5780599097ns, txc = 0x556fef5f1500
  75. 2022-09-12T20:14:03.747+0200 7fb3de27a700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 9166442101ns
  76. 2022-09-12T20:14:03.748+0200 7fb3d1260700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5786712591ns
  77. 2022-09-12T20:14:03.749+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 9168947897ns, txc = 0x556fa6128000
  78. 2022-09-12T20:14:03.752+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5791061540ns, txc = 0x556fe33c5200
  79. 2022-09-12T20:16:42.839+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 81074 from recycle list
  80.  
  81. 2022-09-12T20:16:42.839+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #81078. Immutable memtables: 0.
  82.  
  83. 2022-09-12T20:16:42.839+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:16:42.840595) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
  84. 2022-09-12T20:16:42.839+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 3337] Flushing memtable with next log file: 81078
  85.  
  86. 2022-09-12T20:16:42.839+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006602840657, "job": 3337, "event": "flush_started", "num_memtables": 1, "num_entries": 886819, "num_deletes": 152894, "total_data_size": 242699599, "memory_usage": 260049360, "flush_reason": "Write Buffer Full"}
  87. 2022-09-12T20:16:42.839+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 3337] Level-0 flush table #81079: started
  88. 2022-09-12T20:16:43.353+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006603353899, "cf_name": "default", "job": 3337, "event": "table_file_creation", "file_number": 81079, "file_size": 44186142, "table_properties": {"data_size": 42578349, "index_size": 702537, "filter_size": 904389, "raw_key_size": 16640331, "raw_average_key_size": 46, "raw_value_size": 36308932, "raw_average_value_size": 100, "num_data_blocks": 10430, "num_entries": 361742, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  89. 2022-09-12T20:16:43.353+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 3337] Level-0 flush table #81079: 44186142 bytes OK
  90. 2022-09-12T20:16:43.353+0200 7fb3e7a8d700 4 rocksdb: [db/db_impl_files.cc:194] adding log 81076 to recycle list
  91.  
  92. 2022-09-12T20:16:43.353+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:16:43.353945) [db/memtable_list.cc:380] [default] Level-0 commit table #81079 started
  93. 2022-09-12T20:16:43.353+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:16:43.354499) [db/memtable_list.cc:428] [default] Level-0 commit table #81079: memtable #1 done
  94. 2022-09-12T20:16:43.353+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:16:43.354520) EVENT_LOG_v1 {"time_micros": 1663006603354506, "job": 3337, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [4, 4, 32, 107, 0, 0, 0], "immutable_memtables": 0}
  95. 2022-09-12T20:16:43.353+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:16:43.354547) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[4 4 32 107 0 0 0] max score 1.00
  96.  
  97. 2022-09-12T20:16:43.354+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 3338] Compacting 4@0 + 4@1 files to L1, score 1.00
  98. 2022-09-12T20:16:43.354+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 3337 Base level 0, inputs: [81079(42MB) 81077(35MB) 81075(38MB) 81073(40MB)], [81056(65MB) 81057(65MB) 81059(16MB) 81060(4187KB)]
  99.  
  100. 2022-09-12T20:16:43.354+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006603355162, "job": 3338, "event": "compaction_started", "compaction_reason": "LevelL0FilesNum", "files_L0": [81079, 81077, 81075, 81073], "files_L1": [81056, 81057, 81059, 81060], "score": 1, "input_data_size": 325167072}
  101. 2022-09-12T20:16:43.781+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3338] Generated table #81080: 154492 keys, 68946548 bytes
  102. 2022-09-12T20:16:43.781+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006603782387, "cf_name": "default", "job": 3338, "event": "table_file_creation", "file_number": 81080, "file_size": 68946548, "table_properties": {"data_size": 67111186, "index_size": 1448255, "filter_size": 386245, "raw_key_size": 13492807, "raw_average_key_size": 87, "raw_value_size": 62175651, "raw_average_value_size": 402, "num_data_blocks": 16466, "num_entries": 154492, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  103. 2022-09-12T20:16:44.221+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3338] Generated table #81081: 147259 keys, 68921905 bytes
  104. 2022-09-12T20:16:44.221+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006604222260, "cf_name": "default", "job": 3338, "event": "table_file_creation", "file_number": 81081, "file_size": 68921905, "table_properties": {"data_size": 67111127, "index_size": 1441719, "filter_size": 368197, "raw_key_size": 12862281, "raw_average_key_size": 87, "raw_value_size": 62294559, "raw_average_value_size": 423, "num_data_blocks": 16409, "num_entries": 147259, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  105. 2022-09-12T20:16:44.841+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3338] Generated table #81082: 572763 keys, 69711528 bytes
  106. 2022-09-12T20:16:44.841+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006604842772, "cf_name": "default", "job": 3338, "event": "table_file_creation", "file_number": 81082, "file_size": 69711528, "table_properties": {"data_size": 67110867, "index_size": 1167794, "filter_size": 1432005, "raw_key_size": 30717279, "raw_average_key_size": 53, "raw_value_size": 57461619, "raw_average_value_size": 100, "num_data_blocks": 16555, "num_entries": 572763, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  107. 2022-09-12T20:16:45.601+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3338] Generated table #81083: 688761 keys, 45410536 bytes
  108. 2022-09-12T20:16:45.601+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605602438, "cf_name": "default", "job": 3338, "event": "table_file_creation", "file_number": 81083, "file_size": 45410536, "table_properties": {"data_size": 43072366, "index_size": 615382, "filter_size": 1721925, "raw_key_size": 28572317, "raw_average_key_size": 41, "raw_value_size": 32028215, "raw_average_value_size": 46, "num_data_blocks": 10674, "num_entries": 688761, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  109. 2022-09-12T20:16:45.634+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3338] Generated table #81084: 25722 keys, 3120940 bytes
  110. 2022-09-12T20:16:45.634+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605635255, "cf_name": "default", "job": 3338, "event": "table_file_creation", "file_number": 81084, "file_size": 3120940, "table_properties": {"data_size": 2995020, "index_size": 60736, "filter_size": 64325, "raw_key_size": 3208798, "raw_average_key_size": 124, "raw_value_size": 1098888, "raw_average_value_size": 42, "num_data_blocks": 752, "num_entries": 25722, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  111. 2022-09-12T20:16:45.647+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 3338] Compacted 4@0 + 4@1 files to L1 => 256111457 bytes
  112. 2022-09-12T20:16:45.648+0200 7fb3e8a8f700 4 rocksdb: (Original Log Time 2022/09/12-20:16:45.648880) [db/compaction_job.cc:768] [default] compacted to: files[0 5 32 107 0 0 0] max score 1.10, MB/sec: 142.6 rd, 112.3 wr, level 1, files in(4, 4) out(5) MB in(157.6, 152.5) out(244.2), read-write-amplify(3.5) write-amplify(1.5) OK, records in: 1877734, records dropped: 222324 output_compression: NoCompression
  113.  
  114. 2022-09-12T20:16:45.648+0200 7fb3e8a8f700 4 rocksdb: (Original Log Time 2022/09/12-20:16:45.648911) EVENT_LOG_v1 {"time_micros": 1663006605648898, "job": 3338, "event": "compaction_finished", "compaction_time_micros": 2280467, "compaction_time_cpu_micros": 1744555, "output_level": 1, "num_output_files": 5, "total_output_size": 256111457, "num_input_records": 1811321, "num_output_records": 1588997, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 32, 107, 0, 0, 0]}
  115. 2022-09-12T20:16:45.648+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 3339] Compacting 1@1 + 7@2 files to L2, score 1.10
  116. 2022-09-12T20:16:45.648+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 3338 Base level 1, inputs: [81082(66MB)], [81061(66MB) 81062(66MB) 81063(66MB) 81064(66MB) 81065(56MB) 81066(66MB) 81067(68MB)]
  117.  
  118. 2022-09-12T20:16:45.648+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605649054, "job": 3339, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [81082], "files_L2": [81061, 81062, 81063, 81064, 81065, 81066, 81067], "score": 1.10406, "input_data_size": 546912233}
  119. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661273, "job": 0, "event": "table_file_deletion", "file_number": 81079}
  120. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661341, "job": 0, "event": "table_file_deletion", "file_number": 81077}
  121. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661410, "job": 0, "event": "table_file_deletion", "file_number": 81075}
  122. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661461, "job": 0, "event": "table_file_deletion", "file_number": 81073}
  123. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661507, "job": 0, "event": "table_file_deletion", "file_number": 81060}
  124. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661550, "job": 0, "event": "table_file_deletion", "file_number": 81059}
  125. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661607, "job": 0, "event": "table_file_deletion", "file_number": 81057}
  126. 2022-09-12T20:16:45.660+0200 7fb3d0a5f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006605661652, "job": 0, "event": "table_file_deletion", "file_number": 81056}
  127. 2022-09-12T20:16:46.198+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81085: 255981 keys, 69251607 bytes
  128. 2022-09-12T20:16:46.198+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006606199646, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81085, "file_size": 69251607, "table_properties": {"data_size": 67111481, "index_size": 1499195, "filter_size": 640069, "raw_key_size": 22409448, "raw_average_key_size": 87, "raw_value_size": 60122765, "raw_average_value_size": 234, "num_data_blocks": 16685, "num_entries": 255981, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  129. 2022-09-12T20:16:46.657+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81086: 243014 keys, 69215905 bytes
  130. 2022-09-12T20:16:46.657+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006606658608, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81086, "file_size": 69215905, "table_properties": {"data_size": 67112420, "index_size": 1495066, "filter_size": 607557, "raw_key_size": 21268087, "raw_average_key_size": 87, "raw_value_size": 60363841, "raw_average_value_size": 248, "num_data_blocks": 16659, "num_entries": 243014, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  131. 2022-09-12T20:16:47.087+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81087: 245312 keys, 69214265 bytes
  132. 2022-09-12T20:16:47.087+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006607088538, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81087, "file_size": 69214265, "table_properties": {"data_size": 67111465, "index_size": 1488621, "filter_size": 613317, "raw_key_size": 21470280, "raw_average_key_size": 87, "raw_value_size": 60322908, "raw_average_value_size": 245, "num_data_blocks": 16642, "num_entries": 245312, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  133. 2022-09-12T20:16:47.686+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81088: 248767 keys, 69224622 bytes
  134. 2022-09-12T20:16:47.686+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006607687554, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81088, "file_size": 69224622, "table_properties": {"data_size": 67112524, "index_size": 1489215, "filter_size": 622021, "raw_key_size": 21773955, "raw_average_key_size": 87, "raw_value_size": 60253369, "raw_average_value_size": 242, "num_data_blocks": 16649, "num_entries": 248767, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  135. 2022-09-12T20:16:48.087+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81089: 216060 keys, 59850764 bytes
  136. 2022-09-12T20:16:48.087+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006608088437, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81089, "file_size": 59850764, "table_properties": {"data_size": 58017394, "index_size": 1292280, "filter_size": 540229, "raw_key_size": 18912594, "raw_average_key_size": 87, "raw_value_size": 52060636, "raw_average_value_size": 240, "num_data_blocks": 14387, "num_entries": 216060, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  137. 2022-09-12T20:16:48.577+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81090: 253598 keys, 69242962 bytes
  138. 2022-09-12T20:16:48.577+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006608578540, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81090, "file_size": 69242962, "table_properties": {"data_size": 67111578, "index_size": 1496469, "filter_size": 634053, "raw_key_size": 22202732, "raw_average_key_size": 87, "raw_value_size": 60188551, "raw_average_value_size": 237, "num_data_blocks": 16635, "num_entries": 253598, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  139. 2022-09-12T20:16:49.738+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81091: 1507724 keys, 72008019 bytes
  140. 2022-09-12T20:16:49.739+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006609739823, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81091, "file_size": 72008019, "table_properties": {"data_size": 67109447, "index_size": 1128295, "filter_size": 3769413, "raw_key_size": 80062671, "raw_average_key_size": 53, "raw_value_size": 43231304, "raw_average_value_size": 28, "num_data_blocks": 16651, "num_entries": 1507724, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  141. 2022-09-12T20:16:50.192+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 3339] Generated table #81092: 436451 keys, 31122832 bytes
  142. 2022-09-12T20:16:50.192+0200 7fb3e8a8f700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610193542, "cf_name": "default", "job": 3339, "event": "table_file_creation", "file_number": 81092, "file_size": 31122832, "table_properties": {"data_size": 29569094, "index_size": 461736, "filter_size": 1091141, "raw_key_size": 22445489, "raw_average_key_size": 51, "raw_value_size": 22807395, "raw_average_value_size": 52, "num_data_blocks": 7343, "num_entries": 436451, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  143. 2022-09-12T20:16:50.216+0200 7fb3e8a8f700 4 rocksdb: [db/compaction_job.cc:1395] [default] [JOB 3339] Compacted 1@1 + 7@2 files to L2 => 509130976 bytes
  144. 2022-09-12T20:16:50.217+0200 7fb3e8a8f700 4 rocksdb: (Original Log Time 2022/09/12-20:16:50.217940) [db/compaction_job.cc:768] [default] compacted to: files[0 4 33 107 0 0 0] max score 0.91, MB/sec: 120.3 rd, 112.0 wr, level 2, files in(1, 7) out(8) MB in(66.5, 455.1) out(485.5), read-write-amplify(15.1) write-amplify(7.3) OK, records in: 3526832, records dropped: 119925 output_compression: NoCompression
  145.  
  146. 2022-09-12T20:16:50.217+0200 7fb3e8a8f700 4 rocksdb: (Original Log Time 2022/09/12-20:16:50.217969) EVENT_LOG_v1 {"time_micros": 1663006610217957, "job": 3339, "event": "compaction_finished", "compaction_time_micros": 4544520, "compaction_time_cpu_micros": 3267596, "output_level": 2, "num_output_files": 8, "total_output_size": 509130976, "num_input_records": 3526832, "num_output_records": 3406907, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 4, 33, 107, 0, 0, 0]}
  147. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494262, "job": 0, "event": "table_file_deletion", "file_number": 81082}
  148. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494346, "job": 0, "event": "table_file_deletion", "file_number": 81067}
  149. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494403, "job": 0, "event": "table_file_deletion", "file_number": 81066}
  150. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494455, "job": 0, "event": "table_file_deletion", "file_number": 81065}
  151. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494505, "job": 0, "event": "table_file_deletion", "file_number": 81064}
  152. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494562, "job": 0, "event": "table_file_deletion", "file_number": 81063}
  153. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494634, "job": 0, "event": "table_file_deletion", "file_number": 81062}
  154. 2022-09-12T20:16:50.492+0200 7fb3d3a65700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006610494680, "job": 0, "event": "table_file_deletion", "file_number": 81061}
  155. 2022-09-12T20:17:10.392+0200 7fb3d2a63700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5627652584ns
  156. 2022-09-12T20:17:10.395+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5630732414ns, txc = 0x556f19fcec00
  157. 2022-09-12T20:17:32.689+0200 7fb3e339d700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 12144897459ns
  158. 2022-09-12T20:17:32.690+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 12145396652ns, txc = 0x556fef5f1200
  159. 2022-09-12T20:18:50.619+0200 7fb3e339d700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5791010397ns
  160. 2022-09-12T20:18:50.620+0200 7fb3d4266700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 6442116892ns
  161. 2022-09-12T20:18:50.620+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5792658552ns, txc = 0x556ed14f5b00
  162. 2022-09-12T20:18:50.623+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 6445286403ns, txc = 0x556fcc3f4600
  163. 2022-09-12T20:19:36.222+0200 7fb3d025e700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5256981182ns
  164. 2022-09-12T20:19:36.223+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5258103612ns, txc = 0x556f146ebb00
  165. 2022-09-12T20:19:36.223+0200 7fb3d2a63700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5248382432ns
  166. 2022-09-12T20:19:36.230+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5255365150ns, txc = 0x556f1213b200
  167. 2022-09-12T20:19:39.789+0200 7fb3de27a700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 10822703396ns
  168. 2022-09-12T20:19:39.791+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 10824298979ns, txc = 0x556fcdc4e000
  169. 2022-09-12T20:21:26.649+0200 7fb3df27c700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
  170. 2022-09-12T20:21:26.649+0200 7fb3df27c700 4 rocksdb: [db/db_impl.cc:778]
  171. ** DB Stats **
  172. Uptime(secs): 532803.3 total, 600.0 interval
  173. Cumulative writes: 311M writes, 1767M keys, 311M commit groups, 1.0 writes per commit group, ingest: 463.98 GB, 0.89 MB/s
  174. Cumulative WAL: 311M writes, 136M syncs, 2.29 writes per sync, written: 463.98 GB, 0.89 MB/s
  175. Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
  176. Interval writes: 219K writes, 1354K keys, 219K commit groups, 1.0 writes per commit group, ingest: 348.21 MB, 0.58 MB/s
  177. Interval WAL: 219K writes, 85K syncs, 2.58 writes per sync, written: 0.34 MB, 0.58 MB/s
  178. Interval stall: 00:00:0.000 H:M:S, 0.0 percent
  179.  
  180. ** Compaction Stats [default] **
  181. Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  182. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  183. L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.3 73.3 0.0 1.0 0.0 77.2 972.81 800.82 2072 0.470 0 0
  184. L1 4/0 177.76 MB 0.7 166.6 73.3 93.3 130.6 37.3 0.0 1.8 118.6 93.0 1438.40 1127.90 518 2.777 1314M 129M
  185. L2 33/0 1.66 GB 0.9 188.5 37.2 151.3 164.2 12.9 0.1 4.4 76.9 67.0 2510.07 2085.96 576 4.358 2934M 93M
  186. L3 107/0 6.74 GB 0.3 29.2 11.5 17.7 17.7 0.0 0.0 1.5 135.7 82.4 220.17 167.20 165 1.334 450M 318M
  187. Sum 144/0 8.57 GB 0.0 384.3 122.0 262.3 385.9 123.6 0.1 5.3 76.5 76.9 5141.45 4181.87 3331 1.544 4700M 542M
  188. Int 0/0 0.00 KB 0.0 0.8 0.2 0.6 0.8 0.2 0.0 18.3 113.3 105.2 7.34 5.41 3 2.446 5404K 342K
  189.  
  190. ** Compaction Stats [default] **
  191. Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  192. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  193. Low 0/0 0.00 KB 0.0 384.3 122.0 262.3 312.6 50.3 0.0 0.0 94.4 76.8 4168.63 3381.06 1259 3.311 4700M 542M
  194. High 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.2 73.2 0.0 0.0 0.0 77.2 972.11 800.82 2071 0.469 0 0
  195. User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 78.9 0.71 0.00 1 0.706 0 0
  196. Uptime(secs): 532803.3 total, 600.0 interval
  197. Flush(GB): cumulative 73.295, interval 0.041
  198. AddFile(GB): cumulative 0.000, interval 0.000
  199. AddFile(Total Files): cumulative 0, interval 0
  200. AddFile(L0 Files): cumulative 0, interval 0
  201. AddFile(Keys): cumulative 0, interval 0
  202. Cumulative compaction: 385.88 GB write, 0.74 MB/s write, 384.28 GB read, 0.74 MB/s read, 5141.4 seconds
  203. Interval compaction: 0.75 GB write, 1.29 MB/s write, 0.81 GB read, 1.39 MB/s read, 7.3 seconds
  204. Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
  205.  
  206. ** File Read Latency Histogram By Level [default] **
  207.  
  208. ** Compaction Stats [default] **
  209. Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  210. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  211. L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.3 73.3 0.0 1.0 0.0 77.2 972.81 800.82 2072 0.470 0 0
  212. L1 4/0 177.76 MB 0.7 166.6 73.3 93.3 130.6 37.3 0.0 1.8 118.6 93.0 1438.40 1127.90 518 2.777 1314M 129M
  213. L2 33/0 1.66 GB 0.9 188.5 37.2 151.3 164.2 12.9 0.1 4.4 76.9 67.0 2510.07 2085.96 576 4.358 2934M 93M
  214. L3 107/0 6.74 GB 0.3 29.2 11.5 17.7 17.7 0.0 0.0 1.5 135.7 82.4 220.17 167.20 165 1.334 450M 318M
  215. Sum 144/0 8.57 GB 0.0 384.3 122.0 262.3 385.9 123.6 0.1 5.3 76.5 76.9 5141.45 4181.87 3331 1.544 4700M 542M
  216. Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
  217.  
  218. ** Compaction Stats [default] **
  219. Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  220. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  221. Low 0/0 0.00 KB 0.0 384.3 122.0 262.3 312.6 50.3 0.0 0.0 94.4 76.8 4168.63 3381.06 1259 3.311 4700M 542M
  222. High 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.2 73.2 0.0 0.0 0.0 77.2 972.11 800.82 2071 0.469 0 0
  223. User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 78.9 0.71 0.00 1 0.706 0 0
  224. Uptime(secs): 532803.3 total, 0.0 interval
  225. Flush(GB): cumulative 73.295, interval 0.000
  226. AddFile(GB): cumulative 0.000, interval 0.000
  227. AddFile(Total Files): cumulative 0, interval 0
  228. AddFile(L0 Files): cumulative 0, interval 0
  229. AddFile(Keys): cumulative 0, interval 0
  230. Cumulative compaction: 385.88 GB write, 0.74 MB/s write, 384.28 GB read, 0.74 MB/s read, 5141.4 seconds
  231. Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
  232. Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
  233.  
  234. ** File Read Latency Histogram By Level [default] **
  235.  
  236. 2022-09-12T20:22:39.586+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 81076 from recycle list
  237.  
  238. 2022-09-12T20:22:39.586+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #81093. Immutable memtables: 0.
  239.  
  240. 2022-09-12T20:22:39.587+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:22:39.589090) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
  241. 2022-09-12T20:22:39.587+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 3340] Flushing memtable with next log file: 81093
  242.  
  243. 2022-09-12T20:22:39.587+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006959589128, "job": 3340, "event": "flush_started", "num_memtables": 1, "num_entries": 817799, "num_deletes": 141821, "total_data_size": 242915527, "memory_usage": 260050368, "flush_reason": "Write Buffer Full"}
  244. 2022-09-12T20:22:39.587+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 3340] Level-0 flush table #81094: started
  245. 2022-09-12T20:22:40.019+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663006960020838, "cf_name": "default", "job": 3340, "event": "table_file_creation", "file_number": 81094, "file_size": 43876506, "table_properties": {"data_size": 42321789, "index_size": 721397, "filter_size": 832453, "raw_key_size": 15923232, "raw_average_key_size": 47, "raw_value_size": 36491485, "raw_average_value_size": 109, "num_data_blocks": 10382, "num_entries": 332933, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  246. 2022-09-12T20:22:40.019+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 3340] Level-0 flush table #81094: 43876506 bytes OK
  247. 2022-09-12T20:22:40.019+0200 7fb3e7a8d700 4 rocksdb: [db/db_impl_files.cc:194] adding log 81078 to recycle list
  248.  
  249. 2022-09-12T20:22:40.019+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:22:40.020884) [db/memtable_list.cc:380] [default] Level-0 commit table #81094 started
  250. 2022-09-12T20:22:40.019+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:22:40.021426) [db/memtable_list.cc:428] [default] Level-0 commit table #81094: memtable #1 done
  251. 2022-09-12T20:22:40.019+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:22:40.021448) EVENT_LOG_v1 {"time_micros": 1663006960021439, "job": 3340, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [1, 4, 33, 107, 0, 0, 0], "immutable_memtables": 0}
  252. 2022-09-12T20:22:40.019+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:22:40.021472) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[1 4 33 107 0 0 0] max score 0.91
  253.  
  254. 2022-09-12T20:23:54.721+0200 7fb3d2262700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5694308431ns
  255. 2022-09-12T20:23:54.721+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5694678371ns, txc = 0x556fc6b71500
  256. 2022-09-12T20:23:54.722+0200 7fb3e339d700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 13532438922ns
  257. 2022-09-12T20:23:54.722+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 13532966476ns, txc = 0x556edfcf2f00
  258. 2022-09-12T20:24:13.465+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8.8335604c (undecoded) ondisk+write+known_if_redirected e786358)
  259. 2022-09-12T20:24:13.465+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 3 slow requests (by type [ 'delayed' : 3 ] most affected pool [ 'rbd' : 3 ])
  260. 2022-09-12T20:24:14.512+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8.8335604c (undecoded) ondisk+write+known_if_redirected e786358)
  261. 2022-09-12T20:24:14.512+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 4 slow requests (by type [ 'delayed' : 4 ] most affected pool [ 'rbd' : 4 ])
  262. 2022-09-12T20:24:15.479+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8.8335604c (undecoded) ondisk+write+known_if_redirected e786358)
  263. 2022-09-12T20:24:15.479+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 4 slow requests (by type [ 'delayed' : 4 ] most affected pool [ 'rbd' : 4 ])
  264. 2022-09-12T20:24:16.524+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8.8335604c (undecoded) ondisk+write+known_if_redirected e786358)
  265. 2022-09-12T20:24:16.524+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 6 slow requests (by type [ 'delayed' : 6 ] most affected pool [ 'rbd' : 6 ])
  266. 2022-09-12T20:24:17.551+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 7 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8.8335604c (undecoded) ondisk+write+known_if_redirected e786358)
  267. 2022-09-12T20:24:17.551+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 7 slow requests (by type [ 'delayed' : 7 ] most affected pool [ 'rbd' : 7 ])
  268. 2022-09-12T20:24:20.061+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  269. 2022-09-12T20:24:20.061+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  270. 2022-09-12T20:24:21.052+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  271. 2022-09-12T20:24:21.052+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  272. 2022-09-12T20:24:22.061+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  273. 2022-09-12T20:24:22.061+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  274. 2022-09-12T20:24:23.105+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  275. 2022-09-12T20:24:23.105+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  276. 2022-09-12T20:24:24.111+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  277. 2022-09-12T20:24:24.111+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  278. 2022-09-12T20:24:26.334+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 73 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8.8335604c (undecoded) ondisk+write+known_if_redirected e786358)
  279. 2022-09-12T20:24:26.334+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 73 slow requests (by type [ 'delayed' : 73 ] most affected pool [ 'rbd' : 73 ])
  280. 2022-09-12T20:24:27.360+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  281. 2022-09-12T20:24:27.360+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  282. 2022-09-12T20:24:28.327+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  283. 2022-09-12T20:24:28.327+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  284. 2022-09-12T20:24:29.377+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  285. 2022-09-12T20:24:29.377+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  286. 2022-09-12T20:24:30.421+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  287. 2022-09-12T20:24:30.421+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  288. 2022-09-12T20:24:32.990+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  289. 2022-09-12T20:24:32.990+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  290. 2022-09-12T20:24:34.018+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  291. 2022-09-12T20:24:34.018+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  292. 2022-09-12T20:24:35.039+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  293. 2022-09-12T20:24:35.039+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  294. 2022-09-12T20:24:36.062+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8.40ef364 (undecoded) ondisk+read+known_if_redirected e786358)
  295. 2022-09-12T20:24:36.062+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  296. 2022-09-12T20:24:37.102+0200 7fb3eca97700 -1 osd.139 786357 get_health_metrics reporting 78 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8.8335604c (undecoded) ondisk+write+known_if_redirected e786358)
  297. 2022-09-12T20:24:37.102+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 78 slow requests (by type [ 'delayed' : 78 ] most affected pool [ 'rbd' : 78 ])
  298. 2022-09-12T20:24:42.249+0200 7fb3e339d700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 47526929326ns
  299. 2022-09-12T20:24:42.250+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 47527588098ns, txc = 0x556f969a6600
  300. 2022-09-12T20:24:42.252+0200 7fb3d2262700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 5174138198ns
  301. 2022-09-12T20:24:42.254+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5176359133ns, txc = 0x556f199e4900
  302. 2022-09-12T20:24:43.213+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  303. 2022-09-12T20:24:43.213+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  304. 2022-09-12T20:24:44.260+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  305. 2022-09-12T20:24:44.260+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  306. 2022-09-12T20:24:45.244+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  307. 2022-09-12T20:24:45.244+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  308. 2022-09-12T20:24:46.267+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  309. 2022-09-12T20:24:46.267+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  310. 2022-09-12T20:24:47.286+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  311. 2022-09-12T20:24:47.286+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  312. 2022-09-12T20:24:50.059+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  313. 2022-09-12T20:24:50.059+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  314. 2022-09-12T20:24:51.101+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  315. 2022-09-12T20:24:51.101+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  316. 2022-09-12T20:24:52.134+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  317. 2022-09-12T20:24:52.134+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  318. 2022-09-12T20:24:53.118+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  319. 2022-09-12T20:24:53.118+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  320. 2022-09-12T20:24:54.117+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  321. 2022-09-12T20:24:54.117+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  322. 2022-09-12T20:24:55.387+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  323. 2022-09-12T20:24:55.387+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  324. 2022-09-12T20:24:56.383+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  325. 2022-09-12T20:24:56.384+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  326. 2022-09-12T20:24:57.349+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  327. 2022-09-12T20:24:57.349+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  328. 2022-09-12T20:24:58.394+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  329. 2022-09-12T20:24:58.394+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  330. 2022-09-12T20:24:59.386+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  331. 2022-09-12T20:24:59.386+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  332. 2022-09-12T20:25:00.370+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  333. 2022-09-12T20:25:00.371+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  334. 2022-09-12T20:25:01.738+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  335. 2022-09-12T20:25:01.738+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  336. 2022-09-12T20:25:02.753+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  337. 2022-09-12T20:25:02.753+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  338. 2022-09-12T20:25:03.781+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  339. 2022-09-12T20:25:03.781+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  340. 2022-09-12T20:25:04.824+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  341. 2022-09-12T20:25:04.825+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  342. 2022-09-12T20:25:05.801+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  343. 2022-09-12T20:25:05.801+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  344. 2022-09-12T20:25:07.630+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  345. 2022-09-12T20:25:07.630+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  346. 2022-09-12T20:25:08.665+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  347. 2022-09-12T20:25:08.665+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  348. 2022-09-12T20:25:09.630+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  349. 2022-09-12T20:25:09.630+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  350. 2022-09-12T20:25:10.610+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  351. 2022-09-12T20:25:10.610+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  352. 2022-09-12T20:25:11.613+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  353. 2022-09-12T20:25:11.613+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  354. 2022-09-12T20:25:12.608+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  355. 2022-09-12T20:25:12.608+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  356. 2022-09-12T20:25:13.568+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  357. 2022-09-12T20:25:13.568+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  358. 2022-09-12T20:25:14.531+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  359. 2022-09-12T20:25:14.531+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  360. 2022-09-12T20:25:15.495+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  361. 2022-09-12T20:25:15.495+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  362. 2022-09-12T20:25:16.473+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  363. 2022-09-12T20:25:16.473+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  364. 2022-09-12T20:25:17.428+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  365. 2022-09-12T20:25:17.428+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  366. 2022-09-12T20:25:18.570+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  367. 2022-09-12T20:25:18.570+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  368. 2022-09-12T20:25:19.456+0200 7fb3de27a700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency slow operation observed for submit_transact, latency = 36458839766ns
  369. 2022-09-12T20:25:19.456+0200 7fb3e6885700 0 bluestore(/var/lib/ceph/osd/ceph-139) log_latency_fn slow operation observed for _txc_committed_kv, latency = 36459426271ns, txc = 0x556fb73d7b00
  370. 2022-09-12T20:25:19.527+0200 7fb3eca97700 -1 osd.139 786359 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.168285149.0:674682041 8.364 8:26cf7020:::rbd_data.c8f486969792d4.00000000000025c3:head [read 2019328~4096] snapc 0=[] ondisk+read+known_if_redirected e786358)
  371. 2022-09-12T20:25:19.527+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 256 slow requests (by type [ 'delayed' : 256 ] most affected pool [ 'rbd' : 255 ])
  372. 2022-09-12T20:25:20.540+0200 7fb3eca97700 -1 osd.139 786366 get_health_metrics reporting 9 slow ops, oldest is osd_op(client.164411599.0:6530786 8.4c 8:3206acc1:::rbd_data.8a9a87e481ee14.00000000000004e4:head [write 2523136~32768 in=32768b] snapc 0=[] ondisk+write+known_if_redirected e786358)
  373. 2022-09-12T20:25:20.540+0200 7fb3eca97700 0 log_channel(cluster) log [WRN] : 9 slow requests (by type [ 'waiting for sub ops' : 9 ] most affected pool [ 'rbd' : 9 ])
  374. 2022-09-12T20:28:42.238+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 81078 from recycle list
  375.  
  376. 2022-09-12T20:28:42.238+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #81095. Immutable memtables: 0.
  377.  
  378. 2022-09-12T20:28:42.238+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:28:42.239336) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
  379. 2022-09-12T20:28:42.238+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 3341] Flushing memtable with next log file: 81095
  380.  
  381. 2022-09-12T20:28:42.238+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663007322239380, "job": 3341, "event": "flush_started", "num_memtables": 1, "num_entries": 814943, "num_deletes": 141952, "total_data_size": 244453023, "memory_usage": 260051664, "flush_reason": "Write Buffer Full"}
  382. 2022-09-12T20:28:42.238+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 3341] Level-0 flush table #81096: started
  383. 2022-09-12T20:28:42.709+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663007322710356, "cf_name": "default", "job": 3341, "event": "table_file_creation", "file_number": 81096, "file_size": 37766714, "table_properties": {"data_size": 36437568, "index_size": 588882, "filter_size": 739397, "raw_key_size": 14494729, "raw_average_key_size": 49, "raw_value_size": 31372579, "raw_average_value_size": 106, "num_data_blocks": 8876, "num_entries": 295737, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  384. 2022-09-12T20:28:42.709+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 3341] Level-0 flush table #81096: 37766714 bytes OK
  385. 2022-09-12T20:28:42.711+0200 7fb3e7a8d700 4 rocksdb: [db/db_impl_files.cc:194] adding log 81093 to recycle list
  386.  
  387. 2022-09-12T20:28:42.711+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:28:42.710425) [db/memtable_list.cc:380] [default] Level-0 commit table #81096 started
  388. 2022-09-12T20:28:42.711+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:28:42.711642) [db/memtable_list.cc:428] [default] Level-0 commit table #81096: memtable #1 done
  389. 2022-09-12T20:28:42.711+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:28:42.711691) EVENT_LOG_v1 {"time_micros": 1663007322711674, "job": 3341, "event": "flush_finished", "output_compression": "NoCompression", "lsm_state": [2, 4, 33, 107, 0, 0, 0], "immutable_memtables": 0}
  390. 2022-09-12T20:28:42.711+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:28:42.711795) [db/db_impl_compaction_flush.cc:201] [default] Level summary: files[2 4 33 107 0 0 0] max score 0.91
  391.  
  392. 2022-09-12T20:31:26.650+0200 7fb3df27c700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS -------
  393. 2022-09-12T20:31:26.650+0200 7fb3df27c700 4 rocksdb: [db/db_impl.cc:778]
  394. ** DB Stats **
  395. Uptime(secs): 533403.3 total, 600.0 interval
  396. Cumulative writes: 311M writes, 1768M keys, 311M commit groups, 1.0 writes per commit group, ingest: 464.40 GB, 0.89 MB/s
  397. Cumulative WAL: 311M writes, 136M syncs, 2.29 writes per sync, written: 464.40 GB, 0.89 MB/s
  398. Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
  399. Interval writes: 275K writes, 1512K keys, 275K commit groups, 1.0 writes per commit group, ingest: 435.14 MB, 0.73 MB/s
  400. Interval WAL: 275K writes, 114K syncs, 2.40 writes per sync, written: 0.42 MB, 0.73 MB/s
  401. Interval stall: 00:00:0.000 H:M:S, 0.0 percent
  402.  
  403. ** Compaction Stats [default] **
  404. Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  405. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  406. L0 2/0 77.86 MB 0.5 0.0 0.0 0.0 73.4 73.4 0.0 1.0 0.0 77.2 973.72 801.55 2074 0.469 0 0
  407. L1 4/0 177.76 MB 0.7 166.6 73.3 93.3 130.6 37.3 0.0 1.8 118.6 93.0 1438.40 1127.90 518 2.777 1314M 129M
  408. L2 33/0 1.66 GB 0.9 188.5 37.2 151.3 164.2 12.9 0.1 4.4 76.9 67.0 2510.07 2085.96 576 4.358 2934M 93M
  409. L3 107/0 6.74 GB 0.3 29.2 11.5 17.7 17.7 0.0 0.0 1.5 135.7 82.4 220.17 167.20 165 1.334 450M 318M
  410. Sum 146/0 8.65 GB 0.0 384.3 122.0 262.3 386.0 123.7 0.1 5.3 76.5 76.9 5142.35 4182.60 3333 1.543 4700M 542M
  411. Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 1.0 0.0 86.2 0.90 0.73 2 0.451 0 0
  412.  
  413. ** Compaction Stats [default] **
  414. Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  415. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  416. Low 0/0 0.00 KB 0.0 384.3 122.0 262.3 312.6 50.3 0.0 0.0 94.4 76.8 4168.63 3381.06 1259 3.311 4700M 542M
  417. High 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.3 73.3 0.0 0.0 0.0 77.2 973.01 801.55 2073 0.469 0 0
  418. User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 78.9 0.71 0.00 1 0.706 0 0
  419. Uptime(secs): 533403.3 total, 600.0 interval
  420. Flush(GB): cumulative 73.371, interval 0.076
  421. AddFile(GB): cumulative 0.000, interval 0.000
  422. AddFile(Total Files): cumulative 0, interval 0
  423. AddFile(L0 Files): cumulative 0, interval 0
  424. AddFile(Keys): cumulative 0, interval 0
  425. Cumulative compaction: 385.96 GB write, 0.74 MB/s write, 384.28 GB read, 0.74 MB/s read, 5142.3 seconds
  426. Interval compaction: 0.08 GB write, 0.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.9 seconds
  427. Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
  428.  
  429. ** File Read Latency Histogram By Level [default] **
  430.  
  431. ** Compaction Stats [default] **
  432. Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  433. ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  434. L0 2/0 77.86 MB 0.5 0.0 0.0 0.0 73.4 73.4 0.0 1.0 0.0 77.2 973.72 801.55 2074 0.469 0 0
  435. L1 4/0 177.76 MB 0.7 166.6 73.3 93.3 130.6 37.3 0.0 1.8 118.6 93.0 1438.40 1127.90 518 2.777 1314M 129M
  436. L2 33/0 1.66 GB 0.9 188.5 37.2 151.3 164.2 12.9 0.1 4.4 76.9 67.0 2510.07 2085.96 576 4.358 2934M 93M
  437. L3 107/0 6.74 GB 0.3 29.2 11.5 17.7 17.7 0.0 0.0 1.5 135.7 82.4 220.17 167.20 165 1.334 450M 318M
  438. Sum 146/0 8.65 GB 0.0 384.3 122.0 262.3 386.0 123.7 0.1 5.3 76.5 76.9 5142.35 4182.60 3333 1.543 4700M 542M
  439. Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
  440.  
  441. ** Compaction Stats [default] **
  442. Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
  443. -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  444. Low 0/0 0.00 KB 0.0 384.3 122.0 262.3 312.6 50.3 0.0 0.0 94.4 76.8 4168.63 3381.06 1259 3.311 4700M 542M
  445. High 0/0 0.00 KB 0.0 0.0 0.0 0.0 73.3 73.3 0.0 0.0 0.0 77.2 973.01 801.55 2073 0.469 0 0
  446. User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.1 0.1 0.0 0.0 0.0 78.9 0.71 0.00 1 0.706 0 0
  447. Uptime(secs): 533403.3 total, 0.0 interval
  448. Flush(GB): cumulative 73.371, interval 0.000
  449. AddFile(GB): cumulative 0.000, interval 0.000
  450. AddFile(Total Files): cumulative 0, interval 0
  451. AddFile(L0 Files): cumulative 0, interval 0
  452. AddFile(Keys): cumulative 0, interval 0
  453. Cumulative compaction: 385.96 GB write, 0.74 MB/s write, 384.28 GB read, 0.74 MB/s read, 5142.3 seconds
  454. Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
  455. Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
  456.  
  457. ** File Read Latency Histogram By Level [default] **
  458.  
  459. 2022-09-12T20:32:49.902+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1436] reusing log 81093 from recycle list
  460.  
  461. 2022-09-12T20:32:49.902+0200 7fb3e187b700 4 rocksdb: [db/db_impl_write.cc:1470] [default] New memtable created with log file: #81097. Immutable memtables: 0.
  462.  
  463. 2022-09-12T20:32:49.903+0200 7fb3e7a8d700 4 rocksdb: (Original Log Time 2022/09/12-20:32:49.904554) [db/db_impl_compaction_flush.cc:2073] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0
  464. 2022-09-12T20:32:49.903+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:322] [default] [JOB 3342] Flushing memtable with next log file: 81097
  465.  
  466. 2022-09-12T20:32:49.903+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663007569904614, "job": 3342, "event": "flush_started", "num_memtables": 1, "num_entries": 773751, "num_deletes": 131543, "total_data_size": 244888607, "memory_usage": 260049208, "flush_reason": "Write Buffer Full"}
  467. 2022-09-12T20:32:49.903+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:351] [default] [JOB 3342] Level-0 flush table #81098: started
  468. 2022-09-12T20:32:50.333+0200 7fb3e7a8d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1663007570334683, "cf_name": "default", "job": 3342, "event": "table_file_creation", "file_number": 81098, "file_size": 33556534, "table_properties": {"data_size": 32371211, "index_size": 521731, "filter_size": 662725, "raw_key_size": 13210684, "raw_average_key_size": 49, "raw_value_size": 27871311, "raw_average_value_size": 105, "num_data_blocks": 7842, "num_entries": 265070, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}}
  469. 2022-09-12T20:32:50.333+0200 7fb3e7a8d700 4 rocksdb: [db/flush_job.cc:392] [default] [JOB 3342] Level-0 flush table #81098: 33556534 bytes OK
  470. 2022-09-12T20:32:50.334+0200 7fb3e7a8d700 4 rocksdb: [db/db_impl_files.cc:194] adding log 81095 to recycle list
  471.  
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!