API tools faq
paste
Guest User

DISABLED speculative

a guest
Aug 4th, 2026
13
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 218.73 KB | None | 0 0
raw download clone embed print report
  1. (APIServer pid=48) INFO 08-03 22:50:38 [api_utils.py:339]
  2. (APIServer pid=48) INFO 08-03 22:50:38 [api_utils.py:339] █ █ █▄ ▄█
  3. (APIServer pid=48) INFO 08-03 22:50:38 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.2.dev0+g752a3a504.d20260803
  4. (APIServer pid=48) INFO 08-03 22:50:38 [api_utils.py:339] █▄█▀ █ █ █ █ model poolside/Laguna-S-2.1-NVFP4
  5. (APIServer pid=48) INFO 08-03 22:50:38 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
  6. (APIServer pid=48) INFO 08-03 22:50:38 [api_utils.py:339]
  7. (APIServer pid=48) INFO 08-03 22:50:38 [api_utils.py:273] non-default args: {'model_tag': 'poolside/Laguna-S-2.1-NVFP4', 'default_chat_template_kwargs': {'enable_thinking': True}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'poolside_v1', 'host': '0.0.0.0', 'port': 8888, 'model': 'poolside/Laguna-S-2.1-NVFP4', 'max_model_len': 262144, 'reasoning_parser': 'poolside_v1', 'master_addr': '192.168.9.21', 'master_port': 25000, 'nnodes': 2, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.85, 'max_num_batched_tokens': 4096, 'max_num_seqs': 8}
  8. (APIServer pid=48) INFO 08-03 22:50:38 [arg_utils.py:772] HF_HUB_OFFLINE is True, replace model_id [poolside/Laguna-S-2.1-NVFP4] to model_path [/cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669]
  9. (APIServer pid=48) WARNING 08-03 22:50:38 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_BASE_DIR
  10. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  11. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  12. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  13. (APIServer pid=48) INFO 08-03 22:50:43 [model.py:619] Resolved architecture: LagunaForCausalLM
  14. (APIServer pid=48) INFO 08-03 22:50:43 [model.py:1776] Using max model len 262144
  15. (APIServer pid=48) INFO 08-03 22:50:44 [arg_utils.py:2026] Inferred data_parallel_rank 0 from node_rank 0
  16. (APIServer pid=48) INFO 08-03 22:50:44 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4096.
  17. (APIServer pid=48) INFO 08-03 22:50:44 [vllm.py:1042] Asynchronous scheduling is enabled.
  18. (APIServer pid=48) INFO 08-03 22:50:44 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
  19. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  20. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  21. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  22. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  23. (APIServer pid=48) [transformers] The tokenizer you are loading from '/cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
  24. (APIServer pid=48) WARNING 08-03 22:50:44 [vllm.py:1486] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implemented the `reasoning_start_str` and `reasoning_end_str`.
  25. (APIServer pid=48) INFO 08-03 22:50:44 [compilation.py:312] Enabled custom fusions: act_quant
  26. (EngineCore pid=139) INFO 08-03 22:50:48 [core.py:114] Initializing a V1 LLM engine (v0.25.2.dev0+g752a3a504.d20260803) with config: model='/cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669', speculative_config=None, tokenizer='/cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='poolside_v1', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')
  27. (EngineCore pid=139) INFO 08-03 22:50:48 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=192.168.9.21, mq_connect_ip=192.168.9.21 (local), world_size=2, local_world_size=1
  28. (Worker pid=165) INFO 08-03 22:50:52 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://192.168.9.21:25000 backend=nccl
  29. (Worker pid=165) INFO 08-03 22:51:19 [pynccl.py:113] vLLM is using nccl==2.30.7
  30. (Worker pid=165) WARNING 08-03 22:51:21 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.1 not supported, communicator is not available.
  31. (Worker pid=165) INFO 08-03 22:51:21 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
  32. (Worker pid=165) INFO 08-03 22:51:23 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
  33. (Worker pid=165) INFO 08-03 22:51:23 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
  34. (Worker pid=165) INFO 08-03 22:51:23 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
  35. (Worker_TP0 pid=165) INFO 08-03 22:51:23 [gpu_model_runner.py:5209] Starting to load model /cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669...
  36. (Worker_TP0 pid=165) INFO 08-03 22:51:24 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
  37. (Worker_TP0 pid=165) INFO 08-03 22:51:24 [flash_attn.py:718] Using FlashAttention version 2
  38. (Worker_TP0 pid=165) INFO 08-03 22:51:24 [nvfp4.py:285] Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION'].
  39. (Worker_TP0 pid=165) INFO 08-03 22:51:25 [unquantized.py:262] Using FlashInfer CUTLASS Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].
  40. (Worker_TP0 pid=165) INFO 08-03 22:51:26 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 92.85 GiB. Available RAM: 59.51 GiB.
  41. (Worker_TP0 pid=165) INFO 08-03 22:51:26 [weight_utils.py:879] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (92.85 GiB) exceeds 90% of available RAM (59.51 GiB).
  42. (Worker_TP0 pid=165)
  43. Loading safetensors checkpoint shards: 0% Completed | 0/49 [00:00<?, ?it/s]
  44. (Worker_TP0 pid=165)
  45. Loading safetensors checkpoint shards: 2% Completed | 1/49 [00:00<00:36, 1.33it/s]
  46. (Worker_TP0 pid=165)
  47. Loading safetensors checkpoint shards: 4% Completed | 2/49 [00:02<00:52, 1.12s/it]
  48. (Worker_TP0 pid=165)
  49. Loading safetensors checkpoint shards: 6% Completed | 3/49 [00:04<01:16, 1.66s/it]
  50. (Worker_TP0 pid=165)
  51. Loading safetensors checkpoint shards: 8% Completed | 4/49 [00:19<05:21, 7.14s/it]
  52. (Worker_TP0 pid=165)
  53. Loading safetensors checkpoint shards: 10% Completed | 5/49 [00:36<07:50, 10.69s/it]
  54. (Worker_TP0 pid=165)
  55. Loading safetensors checkpoint shards: 12% Completed | 6/49 [00:54<09:17, 12.96s/it]
  56. (Worker_TP0 pid=165)
  57. Loading safetensors checkpoint shards: 14% Completed | 7/49 [01:11<10:06, 14.44s/it]
  58. (Worker_TP0 pid=165)
  59. Loading safetensors checkpoint shards: 16% Completed | 8/49 [01:27<10:08, 14.83s/it]
  60. (Worker_TP0 pid=165)
  61. Loading safetensors checkpoint shards: 18% Completed | 9/49 [01:44<10:15, 15.39s/it]
  62. (Worker_TP0 pid=165)
  63. Loading safetensors checkpoint shards: 20% Completed | 10/49 [02:00<10:17, 15.84s/it]
  64. (Worker_TP0 pid=165)
  65. Loading safetensors checkpoint shards: 22% Completed | 11/49 [02:12<09:08, 14.44s/it]
  66. (Worker_TP0 pid=165)
  67. Loading safetensors checkpoint shards: 24% Completed | 12/49 [02:20<07:46, 12.61s/it]
  68. (Worker_TP0 pid=165)
  69. Loading safetensors checkpoint shards: 27% Completed | 13/49 [02:28<06:37, 11.06s/it]
  70. (Worker_TP0 pid=165)
  71. Loading safetensors checkpoint shards: 29% Completed | 14/49 [02:36<05:54, 10.14s/it]
  72. (Worker_TP0 pid=165)
  73. Loading safetensors checkpoint shards: 31% Completed | 15/49 [02:44<05:22, 9.49s/it]
  74. (Worker_TP0 pid=165)
  75. Loading safetensors checkpoint shards: 33% Completed | 16/49 [02:51<04:53, 8.90s/it]
  76. (Worker_TP0 pid=165)
  77. Loading safetensors checkpoint shards: 35% Completed | 17/49 [02:59<04:37, 8.69s/it]
  78. (Worker_TP0 pid=165)
  79. Loading safetensors checkpoint shards: 37% Completed | 18/49 [03:07<04:18, 8.32s/it]
  80. (Worker_TP0 pid=165)
  81. Loading safetensors checkpoint shards: 39% Completed | 19/49 [03:15<04:08, 8.28s/it]
  82. (Worker_TP0 pid=165)
  83. Loading safetensors checkpoint shards: 41% Completed | 20/49 [03:23<04:00, 8.28s/it]
  84. (Worker_TP0 pid=165)
  85. Loading safetensors checkpoint shards: 43% Completed | 21/49 [03:31<03:51, 8.26s/it]
  86. (Worker_TP0 pid=165)
  87. Loading safetensors checkpoint shards: 45% Completed | 22/49 [03:40<03:42, 8.23s/it]
  88. (Worker_TP0 pid=165)
  89. Loading safetensors checkpoint shards: 47% Completed | 23/49 [03:47<03:29, 8.06s/it]
  90. (Worker_TP0 pid=165)
  91. Loading safetensors checkpoint shards: 49% Completed | 24/49 [03:56<03:22, 8.12s/it]
  92. (Worker_TP0 pid=165)
  93. Loading safetensors checkpoint shards: 51% Completed | 25/49 [04:04<03:14, 8.10s/it]
  94. (Worker_TP0 pid=165)
  95. Loading safetensors checkpoint shards: 53% Completed | 26/49 [04:12<03:06, 8.12s/it]
  96. (Worker_TP0 pid=165)
  97. Loading safetensors checkpoint shards: 55% Completed | 27/49 [04:20<02:59, 8.15s/it]
  98. (Worker_TP0 pid=165)
  99. Loading safetensors checkpoint shards: 57% Completed | 28/49 [04:28<02:52, 8.21s/it]
  100. (Worker_TP0 pid=165)
  101. Loading safetensors checkpoint shards: 59% Completed | 29/49 [04:36<02:42, 8.14s/it]
  102. (Worker_TP0 pid=165)
  103. Loading safetensors checkpoint shards: 61% Completed | 30/49 [04:44<02:33, 8.10s/it]
  104. (Worker_TP0 pid=165)
  105. Loading safetensors checkpoint shards: 63% Completed | 31/49 [04:53<02:26, 8.11s/it]
  106. (Worker_TP0 pid=165)
  107. Loading safetensors checkpoint shards: 65% Completed | 32/49 [05:00<02:16, 8.04s/it]
  108. (Worker_TP0 pid=165)
  109. Loading safetensors checkpoint shards: 67% Completed | 33/49 [05:08<02:07, 7.97s/it]
  110. (Worker_TP0 pid=165)
  111. Loading safetensors checkpoint shards: 69% Completed | 34/49 [05:16<01:59, 7.95s/it]
  112. (Worker_TP0 pid=165)
  113. Loading safetensors checkpoint shards: 71% Completed | 35/49 [05:24<01:51, 7.97s/it]
  114. (Worker_TP0 pid=165)
  115. Loading safetensors checkpoint shards: 73% Completed | 36/49 [05:32<01:44, 8.02s/it]
  116. (Worker_TP0 pid=165)
  117. Loading safetensors checkpoint shards: 76% Completed | 37/49 [05:40<01:36, 8.01s/it]
  118. (Worker_TP0 pid=165)
  119. Loading safetensors checkpoint shards: 78% Completed | 38/49 [05:48<01:27, 7.93s/it]
  120. (Worker_TP0 pid=165)
  121. Loading safetensors checkpoint shards: 80% Completed | 39/49 [05:56<01:19, 7.94s/it]
  122. (Worker_TP0 pid=165)
  123. Loading safetensors checkpoint shards: 82% Completed | 40/49 [06:04<01:12, 8.01s/it]
  124. (Worker_TP0 pid=165)
  125. Loading safetensors checkpoint shards: 84% Completed | 41/49 [06:12<01:03, 7.99s/it]
  126. (Worker_TP0 pid=165)
  127. Loading safetensors checkpoint shards: 86% Completed | 42/49 [06:20<00:55, 7.99s/it]
  128. (Worker_TP0 pid=165)
  129. Loading safetensors checkpoint shards: 88% Completed | 43/49 [06:27<00:47, 7.84s/it]
  130. (Worker_TP0 pid=165)
  131. Loading safetensors checkpoint shards: 90% Completed | 44/49 [06:35<00:39, 7.83s/it]
  132. (Worker_TP0 pid=165)
  133. Loading safetensors checkpoint shards: 92% Completed | 45/49 [06:43<00:31, 7.88s/it]
  134. (Worker_TP0 pid=165)
  135. Loading safetensors checkpoint shards: 94% Completed | 46/49 [06:51<00:23, 7.69s/it]
  136. (Worker_TP0 pid=165)
  137. Loading safetensors checkpoint shards: 96% Completed | 47/49 [06:58<00:15, 7.55s/it]
  138. (Worker_TP0 pid=165)
  139. Loading safetensors checkpoint shards: 98% Completed | 48/49 [07:05<00:07, 7.44s/it]
  140. (Worker_TP0 pid=165)
  141. Loading safetensors checkpoint shards: 100% Completed | 49/49 [07:08<00:00, 6.17s/it]
  142. (Worker_TP0 pid=165)
  143. Loading safetensors checkpoint shards: 100% Completed | 49/49 [07:08<00:00, 8.75s/it]
  144. (Worker_TP0 pid=165)
  145. (Worker_TP0 pid=165) INFO 08-03 22:58:35 [default_loader.py:430] Loading weights took 428.72 seconds
  146. (Worker_TP0 pid=165) INFO 08-03 22:58:35 [nvfp4.py:543] Using MoEPrepareAndFinalizeNoDPEPModular
  147. (Worker_TP0 pid=165) INFO 08-03 22:58:35 [unquantized.py:334] Using MoEPrepareAndFinalizeNoDPEPModular
  148. (Worker_TP0 pid=165) INFO 08-03 22:58:36 [gpu_model_runner.py:5306] Model loading took 46.91 GiB memory and 431.829826 seconds
  149. (Worker_TP0 pid=165) INFO 08-03 22:58:42 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/8e143b869f/rank_0_0/backbone for vLLM's torch.compile
  150. (Worker_TP0 pid=165) INFO 08-03 22:58:42 [backends.py:1148] Dynamo bytecode transform time: 6.39 s
  151. (Worker_TP0 pid=165) [rank0]:W0803 22:58:44.237000 165 torch/_inductor/utils.py:1731] Not enough SMs to use max_autotune_gemm mode
  152. (Worker_TP0 pid=165) INFO 08-03 22:58:49 [backends.py:378] Cache the graph of compile range (1, 4096) for later use
  153. (Worker_TP0 pid=165) INFO 08-03 22:59:05 [backends.py:393] Compiling a graph for compile range (1, 4096) takes 21.96 s
  154. (Worker_TP0 pid=165) INFO 08-03 22:59:10 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/7d47e56b466554a017361f3961e5e87470c9ad4f5c053d61be339aca75603d67/rank_0_0/model
  155. (Worker_TP0 pid=165) INFO 08-03 22:59:10 [monitor.py:53] torch.compile took 34.42 s in total
  156. (Worker_TP0 pid=165) INFO 08-03 22:59:14 [monitor.py:81] Initial profiling/warmup run took 3.37 s
  157. (Worker_TP0 pid=165) INFO 08-03 22:59:19 [gpu_model_runner.py:6534] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
  158. (EngineCore pid=139) INFO 08-03 22:59:37 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
  159. (Worker_TP0 pid=165) INFO 08-03 23:00:09 [gpu_model_runner.py:6639] Estimated CUDA graph memory: 0.90 GiB total
  160. (Worker_TP0 pid=165) INFO 08-03 23:00:10 [gpu_worker.py:531] Freed 0.00 GiB before KV cache sizing; non-torch profile increase is 3.82 GiB.
  161. (Worker_TP0 pid=165) INFO 08-03 23:00:10 [gpu_worker.py:569] Available KV cache memory: 51.1 GiB
  162. (Worker_TP0 pid=165) INFO 08-03 23:00:10 [gpu_worker.py:584] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.8500 is equivalent to --gpu-memory-utilization=0.8426 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.8574. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
  163. (EngineCore pid=139) INFO 08-03 23:00:10 [kv_cache_utils.py:2146] GPU KV cache size: 2,113,609 tokens
  164. (EngineCore pid=139) INFO 08-03 23:00:10 [kv_cache_utils.py:2147] Maximum concurrency for 262,144 tokens per request: 8.06x
  165. (Worker_TP0 pid=165) INFO 08-03 23:00:11 [gpu_worker.py:739] Cleared 0.15 GiB of cached CUDA allocator memory before KV cache allocation.
  166. (Worker_TP0 pid=165) INFO 08-03 23:00:26 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
  167. (Worker_TP0 pid=165) INFO 08-03 23:00:26 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
  168. (Worker_TP0 pid=165) 2026-08-03 23:00:26,621 - INFO - autotuner.py:829 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
  169. (Worker_TP0 pid=165)
  170. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 0%| | 0/21 [00:00<?, ?profile/s]2026-08-03 23:00:30,567 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  171. (Worker_TP0 pid=165)
  172. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 5%|▍ | 1/21 [00:03<01:09, 3.45s/profile]2026-08-03 23:00:30,648 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  173. (Worker_TP0 pid=165) 2026-08-03 23:00:30,752 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  174. (Worker_TP0 pid=165)
  175. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 14%|█▍ | 3/21 [00:03<00:17, 1.04profile/s]2026-08-03 23:00:30,908 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  176. (Worker_TP0 pid=165)
  177. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 19%|█▉ | 4/21 [00:03<00:11, 1.46profile/s]2026-08-03 23:00:31,158 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  178. (Worker_TP0 pid=165)
  179. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 24%|██▍ | 5/21 [00:04<00:08, 1.85profile/s]2026-08-03 23:00:31,525 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  180. (Worker_TP0 pid=165)
  181. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 29%|██▊ | 6/21 [00:04<00:07, 2.06profile/s]2026-08-03 23:00:31,985 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  182. (Worker_TP0 pid=165)
  183. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 33%|███▎ | 7/21 [00:04<00:06, 2.10profile/s]2026-08-03 23:00:32,490 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  184. (Worker_TP0 pid=165)
  185. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 38%|███▊ | 8/21 [00:05<00:06, 2.06profile/s]2026-08-03 23:00:32,998 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  186. (Worker_TP0 pid=165)
  187. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 43%|████▎ | 9/21 [00:05<00:05, 2.03profile/s]2026-08-03 23:00:33,530 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  188. (Worker_TP0 pid=165)
  189. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 48%|████▊ | 10/21 [00:06<00:05, 1.98profile/s]2026-08-03 23:00:34,068 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  190. (Worker_TP0 pid=165)
  191. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 52%|█████▏ | 11/21 [00:06<00:05, 1.94profile/s]2026-08-03 23:00:34,614 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  192. (Worker_TP0 pid=165)
  193. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 57%|█████▋ | 12/21 [00:07<00:04, 1.91profile/s]2026-08-03 23:00:35,192 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  194. (Worker_TP0 pid=165)
  195. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 62%|██████▏ | 13/21 [00:08<00:04, 1.85profile/s]2026-08-03 23:00:35,782 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  196. (Worker_TP0 pid=165)
  197. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 67%|██████▋ | 14/21 [00:08<00:03, 1.80profile/s]2026-08-03 23:00:36,398 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  198. (Worker_TP0 pid=165)
  199. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 71%|███████▏ | 15/21 [00:09<00:03, 1.74profile/s]2026-08-03 23:00:37,010 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  200. (Worker_TP0 pid=165)
  201. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 76%|███████▌ | 16/21 [00:09<00:02, 1.71profile/s]2026-08-03 23:00:37,660 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  202. (Worker_TP0 pid=165)
  203. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 81%|████████ | 17/21 [00:10<00:02, 1.65profile/s]2026-08-03 23:00:38,336 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  204. (Worker_TP0 pid=165)
  205. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 86%|████████▌ | 18/21 [00:11<00:01, 1.60profile/s]2026-08-03 23:00:39,077 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  206. (Worker_TP0 pid=165)
  207. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 90%|█████████ | 19/21 [00:11<00:01, 1.51profile/s]2026-08-03 23:00:39,867 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  208. (Worker_TP0 pid=165)
  209. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 95%|█████████▌| 20/21 [00:12<00:00, 1.43profile/s]2026-08-03 23:00:41,116 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  210. (Worker_TP0 pid=165)
  211. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:14<00:00, 1.16profile/s]
  212. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:14<00:00, 1.50profile/s]
  213. (Worker_TP0 pid=165)
  214. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 0%| | 0/21 [00:00<?, ?profile/s]2026-08-03 23:00:41,333 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  215. (Worker_TP0 pid=165)
  216. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 5%|▍ | 1/21 [00:00<00:04, 4.78profile/s]2026-08-03 23:00:41,489 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  217. (Worker_TP0 pid=165)
  218. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 10%|▉ | 2/21 [00:00<00:03, 5.61profile/s]2026-08-03 23:00:41,670 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  219. (Worker_TP0 pid=165)
  220. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 14%|█▍ | 3/21 [00:00<00:03, 5.58profile/s]2026-08-03 23:00:41,887 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  221. (Worker_TP0 pid=165)
  222. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 19%|█▉ | 4/21 [00:00<00:03, 5.15profile/s]2026-08-03 23:00:42,205 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  223. (Worker_TP0 pid=165)
  224. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 24%|██▍ | 5/21 [00:01<00:03, 4.19profile/s]2026-08-03 23:00:42,639 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  225. (Worker_TP0 pid=165)
  226. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 29%|██▊ | 6/21 [00:01<00:04, 3.27profile/s]2026-08-03 23:00:43,153 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  227. (Worker_TP0 pid=165)
  228. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 33%|███▎ | 7/21 [00:02<00:05, 2.68profile/s]2026-08-03 23:00:43,698 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  229. (Worker_TP0 pid=165)
  230. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 38%|███▊ | 8/21 [00:02<00:05, 2.34profile/s]2026-08-03 23:00:44,243 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  231. (Worker_TP0 pid=165)
  232. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 43%|████▎ | 9/21 [00:03<00:05, 2.15profile/s]2026-08-03 23:00:44,828 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  233. (Worker_TP0 pid=165)
  234. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 48%|████▊ | 10/21 [00:03<00:05, 1.99profile/s]2026-08-03 23:00:45,468 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  235. (Worker_TP0 pid=165)
  236. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 52%|█████▏ | 11/21 [00:04<00:05, 1.84profile/s]2026-08-03 23:00:46,147 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  237. (Worker_TP0 pid=165)
  238. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 57%|█████▋ | 12/21 [00:05<00:05, 1.71profile/s]2026-08-03 23:00:46,860 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  239. (Worker_TP0 pid=165)
  240. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 62%|██████▏ | 13/21 [00:05<00:04, 1.60profile/s]2026-08-03 23:00:47,629 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  241. (Worker_TP0 pid=165)
  242. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 67%|██████▋ | 14/21 [00:06<00:04, 1.50profile/s]2026-08-03 23:00:48,442 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  243. (Worker_TP0 pid=165)
  244. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 71%|███████▏ | 15/21 [00:07<00:04, 1.41profile/s]2026-08-03 23:00:49,341 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  245. (Worker_TP0 pid=165)
  246. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 76%|███████▌ | 16/21 [00:08<00:03, 1.30profile/s]2026-08-03 23:00:50,319 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  247. (Worker_TP0 pid=165)
  248. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 81%|████████ | 17/21 [00:09<00:03, 1.20profile/s]2026-08-03 23:00:51,426 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  249. (Worker_TP0 pid=165)
  250. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 86%|████████▌ | 18/21 [00:10<00:02, 1.09profile/s]2026-08-03 23:00:52,818 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  251. (Worker_TP0 pid=165)
  252. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 90%|█████████ | 19/21 [00:11<00:02, 1.06s/profile]2026-08-03 23:00:54,262 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  253. (Worker_TP0 pid=165)
  254. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 95%|█████████▌| 20/21 [00:13<00:01, 1.17s/profile]2026-08-03 23:00:57,085 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  255. (Worker_TP0 pid=165)
  256. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:15<00:00, 1.67s/profile]
  257. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:15<00:00, 1.32profile/s]
  258. (Worker_TP0 pid=165)
  259. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 0%| | 0/21 [00:00<?, ?profile/s]2026-08-03 23:00:58,880 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  260. (Worker_TP0 pid=165)
  261. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 5%|▍ | 1/21 [00:00<00:10, 1.94profile/s]2026-08-03 23:00:58,933 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  262. (Worker_TP0 pid=165) 2026-08-03 23:00:59,024 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  263. (Worker_TP0 pid=165)
  264. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 14%|█▍ | 3/21 [00:00<00:03, 5.34profile/s]2026-08-03 23:00:59,185 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  265. (Worker_TP0 pid=165)
  266. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 19%|█▉ | 4/21 [00:00<00:03, 5.61profile/s]2026-08-03 23:00:59,453 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  267. (Worker_TP0 pid=165)
  268. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 24%|██▍ | 5/21 [00:01<00:03, 4.81profile/s]2026-08-03 23:00:59,869 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  269. (Worker_TP0 pid=165)
  270. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 29%|██▊ | 6/21 [00:01<00:04, 3.64profile/s]2026-08-03 23:01:00,392 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  271. (Worker_TP0 pid=165)
  272. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 33%|███▎ | 7/21 [00:02<00:04, 2.83profile/s]2026-08-03 23:01:00,971 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  273. (Worker_TP0 pid=165)
  274. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 38%|███▊ | 8/21 [00:02<00:05, 2.37profile/s]2026-08-03 23:01:01,556 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  275. (Worker_TP0 pid=165)
  276. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 43%|████▎ | 9/21 [00:03<00:05, 2.12profile/s]2026-08-03 23:01:02,171 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  277. (Worker_TP0 pid=165)
  278. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 48%|████▊ | 10/21 [00:03<00:05, 1.94profile/s]2026-08-03 23:01:02,799 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  279. (Worker_TP0 pid=165)
  280. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 52%|█████▏ | 11/21 [00:04<00:05, 1.82profile/s]2026-08-03 23:01:03,438 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  281. (Worker_TP0 pid=165)
  282. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 57%|█████▋ | 12/21 [00:05<00:05, 1.73profile/s]2026-08-03 23:01:04,096 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  283. (Worker_TP0 pid=165)
  284. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 62%|██████▏ | 13/21 [00:05<00:04, 1.66profile/s]2026-08-03 23:01:04,811 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  285. (Worker_TP0 pid=165)
  286. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 67%|██████▋ | 14/21 [00:06<00:04, 1.57profile/s]2026-08-03 23:01:05,665 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  287. (Worker_TP0 pid=165)
  288. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 71%|███████▏ | 15/21 [00:07<00:04, 1.43profile/s]2026-08-03 23:01:06,622 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  289. (Worker_TP0 pid=165)
  290. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 76%|███████▌ | 16/21 [00:08<00:03, 1.29profile/s]2026-08-03 23:01:07,602 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  291. (Worker_TP0 pid=165)
  292. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 81%|████████ | 17/21 [00:09<00:03, 1.19profile/s]2026-08-03 23:01:08,608 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  293. (Worker_TP0 pid=165)
  294. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 86%|████████▌ | 18/21 [00:10<00:02, 1.12profile/s]2026-08-03 23:01:09,750 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  295. (Worker_TP0 pid=165)
  296. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 90%|█████████ | 19/21 [00:11<00:01, 1.04profile/s]2026-08-03 23:01:10,957 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  297. (Worker_TP0 pid=165)
  298. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 95%|█████████▌| 20/21 [00:12<00:01, 1.04s/profile]2026-08-03 23:01:13,623 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  299. (Worker_TP0 pid=165)
  300. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:15<00:00, 1.53s/profile]
  301. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:15<00:00, 1.38profile/s]
  302. (Worker_TP0 pid=165)
  303. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 0%| | 0/21 [00:00<?, ?profile/s]
  304. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 14%|█▍ | 3/21 [00:00<00:00, 28.11profile/s]
  305. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 29%|██▊ | 6/21 [00:00<00:01, 8.33profile/s]
  306. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 38%|███▊ | 8/21 [00:01<00:02, 4.88profile/s]
  307. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 43%|████▎ | 9/21 [00:01<00:02, 4.11profile/s]
  308. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 48%|████▊ | 10/21 [00:02<00:03, 3.55profile/s]
  309. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 52%|█████▏ | 11/21 [00:02<00:03, 3.13profile/s]
  310. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 57%|█████▋ | 12/21 [00:03<00:03, 2.77profile/s]
  311. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 62%|██████▏ | 13/21 [00:03<00:03, 2.53profile/s]
  312. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 67%|██████▋ | 14/21 [00:04<00:02, 2.37profile/s]
  313. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 71%|███████▏ | 15/21 [00:04<00:02, 2.18profile/s]
  314. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 76%|███████▌ | 16/21 [00:05<00:02, 2.02profile/s]
  315. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 81%|████████ | 17/21 [00:05<00:02, 1.87profile/s]
  316. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 86%|████████▌ | 18/21 [00:06<00:01, 1.74profile/s]
  317. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 90%|█████████ | 19/21 [00:07<00:01, 1.60profile/s]
  318. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 95%|█████████▌| 20/21 [00:08<00:00, 1.47profile/s]
  319. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:09<00:00, 1.17profile/s]
  320. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:09<00:00, 2.26profile/s]
  321. (Worker_TP0 pid=165) 2026-08-03 23:01:23,003 - INFO - autotuner.py:852 - flashinfer.jit: [Autotuner]: Autotuning process ends
  322. (EngineCore pid=139) INFO 08-03 23:01:24 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
  323. (Worker_TP0 pid=165) INFO 08-03 23:01:24 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested.
  324. (Worker_TP0 pid=165)
  325. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/5 [00:00<?, ?it/s]
  326. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 20%|██ | 1/5 [00:00<00:00, 5.65it/s]
  327. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 40%|████ | 2/5 [00:00<00:00, 6.69it/s]
  328. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 60%|██████ | 3/5 [00:00<00:00, 6.88it/s]
  329. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 80%|████████ | 4/5 [00:00<00:00, 6.28it/s]
  330. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:01<00:00, 3.50it/s]
  331. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 5/5 [00:01<00:00, 4.39it/s]
  332. (Worker_TP0 pid=165)
  333. Capturing CUDA graphs (decode, FULL): 0%| | 0/4 [00:00<?, ?it/s]
  334. Capturing CUDA graphs (decode, FULL): 25%|██▌ | 1/4 [00:00<00:00, 4.67it/s]
  335. Capturing CUDA graphs (decode, FULL): 50%|█████ | 2/4 [00:00<00:00, 5.79it/s]
  336. Capturing CUDA graphs (decode, FULL): 75%|███████▌ | 3/4 [00:00<00:00, 5.56it/s]
  337. Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 5.10it/s]
  338. Capturing CUDA graphs (decode, FULL): 100%|██████████| 4/4 [00:00<00:00, 5.22it/s]
  339. (Worker_TP0 pid=165) INFO 08-03 23:01:36 [gpu_model_runner.py:6707] Graph capturing finished in 12 secs, took 0.31 GiB
  340. (Worker_TP0 pid=165) INFO 08-03 23:01:36 [gpu_worker.py:819] CUDA graph pool memory: 0.31 GiB (actual), 0.9 GiB (estimated), difference: 0.59 GiB (187.7%).
  341. (Worker_TP0 pid=165) INFO 08-03 23:01:36 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
  342. (EngineCore pid=139) INFO 08-03 23:01:36 [core.py:337] init engine (profile, create kv cache, warmup model) took 180.41 s (compilation: 34.42 s)
  343. (EngineCore pid=139) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  344. (EngineCore pid=139) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  345. (EngineCore pid=139) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  346. (EngineCore pid=139) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  347. (EngineCore pid=139) [transformers] The tokenizer you are loading from '/cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
  348. (EngineCore pid=139) INFO 08-03 23:01:44 [vllm.py:1042] Asynchronous scheduling is enabled.
  349. (EngineCore pid=139) INFO 08-03 23:01:44 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
  350. (EngineCore pid=139) WARNING 08-03 23:01:44 [vllm.py:1486] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implemented the `reasoning_start_str` and `reasoning_end_str`.
  351. (EngineCore pid=139) INFO 08-03 23:01:44 [compilation.py:312] Enabled custom fusions: act_quant
  352. (APIServer pid=48) INFO 08-03 23:01:45 [api_server.py:612] Supported tasks: ['generate']
  353. (APIServer pid=48) INFO 08-03 23:01:48 [parser_manager.py:37] "auto" tool choice has been enabled.
  354. (APIServer pid=48) WARNING 08-03 23:01:49 [model.py:1528] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 1.0, 'min_p': 0.0}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
  355. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  356. (APIServer pid=48) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  357. (APIServer pid=48) [transformers] The tokenizer you are loading from '/cache/huggingface/models--poolside--Laguna-S-2.1-NVFP4/snapshots/f8fdfcdc4e7b0c474a0102430a8cae0a3a358669' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
  358. (APIServer pid=48) INFO 08-03 23:01:50 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
  359. (APIServer pid=48) INFO 08-03 23:01:50 [api_server.py:616] Starting vLLM server on http://0.0.0.0:8888
  360. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:37] Available routes are:
  361. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
  362. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /docs, Methods: GET, HEAD
  363. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
  364. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
  365. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /load, Methods: GET
  366. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /version, Methods: GET
  367. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /health, Methods: GET
  368. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /metrics, Methods: GET
  369. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /tokenize, Methods: POST
  370. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /detokenize, Methods: POST
  371. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/models, Methods: GET
  372. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /ping, Methods: GET
  373. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /ping, Methods: POST
  374. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /invocations, Methods: POST
  375. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
  376. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
  377. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/responses, Methods: POST
  378. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
  379. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
  380. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/completions, Methods: POST
  381. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/messages, Methods: POST
  382. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
  383. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /generative_scoring, Methods: POST
  384. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
  385. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
  386. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
  387. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/completions/render, Methods: POST
  388. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
  389. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
  390. (APIServer pid=48) INFO 08-03 23:01:50 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
  391. (APIServer pid=48) INFO: Started server process [48]
  392. (APIServer pid=48) INFO: Waiting for application startup.
  393. (APIServer pid=48) INFO: Application startup complete.
  394. (APIServer pid=48) INFO: 172.19.0.4:34626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  395. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  396. (APIServer pid=48) INFO 08-03 23:05:18 [loggers.py:273] Engine 000: Avg prompt throughput: 1259.3 tokens/s, Avg generation throughput: 10.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 0.0%
  397. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  398. (APIServer pid=48) INFO 08-03 23:05:28 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.1 tokens/s, Avg generation throughput: 18.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 33.0%
  399. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  400. (APIServer pid=48) INFO 08-03 23:05:38 [loggers.py:273] Engine 000: Avg prompt throughput: 586.6 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 53.6%
  401. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  402. (APIServer pid=48) INFO 08-03 23:05:48 [loggers.py:273] Engine 000: Avg prompt throughput: 1172.1 tokens/s, Avg generation throughput: 16.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 60.2%
  403. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  404. (APIServer pid=48) INFO 08-03 23:05:58 [loggers.py:273] Engine 000: Avg prompt throughput: 420.4 tokens/s, Avg generation throughput: 16.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 69.3%
  405. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  406. (APIServer pid=48) INFO 08-03 23:06:08 [loggers.py:273] Engine 000: Avg prompt throughput: 476.4 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 74.6%
  407. (APIServer pid=48) INFO 08-03 23:06:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 74.6%
  408. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  409. (APIServer pid=48) INFO 08-03 23:06:28 [loggers.py:273] Engine 000: Avg prompt throughput: 565.6 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 77.9%
  410. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  411. (APIServer pid=48) INFO 08-03 23:06:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 78.8%
  412. (APIServer pid=48) INFO 08-03 23:06:48 [loggers.py:273] Engine 000: Avg prompt throughput: 1228.5 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 78.8%
  413. (APIServer pid=48) INFO 08-03 23:06:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 78.8%
  414. (APIServer pid=48) INFO 08-03 23:07:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 78.8%
  415. (APIServer pid=48) INFO 08-03 23:07:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  416. (APIServer pid=48) INFO 08-03 23:07:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  417. (APIServer pid=48) INFO 08-03 23:07:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  418. (APIServer pid=48) INFO 08-03 23:07:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  419. (APIServer pid=48) INFO 08-03 23:07:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  420. (APIServer pid=48) INFO 08-03 23:08:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  421. (APIServer pid=48) INFO 08-03 23:08:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  422. (APIServer pid=48) INFO 08-03 23:08:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  423. (APIServer pid=48) INFO 08-03 23:08:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  424. (APIServer pid=48) INFO 08-03 23:08:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 78.8%
  425. (APIServer pid=48) INFO 08-03 23:08:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  426. (APIServer pid=48) INFO 08-03 23:09:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  427. (APIServer pid=48) INFO 08-03 23:09:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  428. (APIServer pid=48) INFO 08-03 23:09:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  429. (APIServer pid=48) INFO 08-03 23:09:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  430. (APIServer pid=48) INFO 08-03 23:09:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  431. (APIServer pid=48) INFO 08-03 23:09:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  432. (APIServer pid=48) INFO 08-03 23:10:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  433. (APIServer pid=48) INFO 08-03 23:10:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  434. (APIServer pid=48) INFO 08-03 23:10:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 78.8%
  435. (APIServer pid=48) INFO 08-03 23:10:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  436. (APIServer pid=48) INFO 08-03 23:10:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  437. (APIServer pid=48) INFO 08-03 23:10:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  438. (APIServer pid=48) INFO 08-03 23:11:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  439. (APIServer pid=48) INFO 08-03 23:11:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  440. (APIServer pid=48) INFO 08-03 23:11:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  441. (APIServer pid=48) INFO 08-03 23:11:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  442. (APIServer pid=48) INFO 08-03 23:11:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  443. (APIServer pid=48) INFO 08-03 23:11:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  444. (APIServer pid=48) INFO 08-03 23:12:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 78.8%
  445. (APIServer pid=48) INFO 08-03 23:12:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 78.8%
  446. (APIServer pid=48) INFO 08-03 23:12:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 78.8%
  447. (APIServer pid=48) INFO 08-03 23:12:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 78.8%
  448. (APIServer pid=48) INFO: 172.19.0.4:34636 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  449. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  450. (APIServer pid=48) INFO 08-03 23:12:48 [loggers.py:273] Engine 000: Avg prompt throughput: 177.8 tokens/s, Avg generation throughput: 16.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 79.0%
  451. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  452. (APIServer pid=48) INFO 08-03 23:12:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.7 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 77.4%
  453. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  454. (APIServer pid=48) INFO 08-03 23:13:08 [loggers.py:273] Engine 000: Avg prompt throughput: 529.7 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 77.6%
  455. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  456. (APIServer pid=48) INFO 08-03 23:13:18 [loggers.py:273] Engine 000: Avg prompt throughput: 295.8 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 78.5%
  457. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  458. (APIServer pid=48) INFO 08-03 23:13:28 [loggers.py:273] Engine 000: Avg prompt throughput: 573.3 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 79.0%
  459. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  460. (APIServer pid=48) INFO 08-03 23:13:38 [loggers.py:273] Engine 000: Avg prompt throughput: 1067.6 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 78.9%
  461. (APIServer pid=48) INFO 08-03 23:13:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 78.9%
  462. (APIServer pid=48) INFO 08-03 23:13:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 78.9%
  463. (APIServer pid=48) INFO 08-03 23:14:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 78.9%
  464. (APIServer pid=48) INFO 08-03 23:14:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 78.9%
  465. (APIServer pid=48) INFO 08-03 23:14:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  466. (APIServer pid=48) INFO 08-03 23:14:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  467. (APIServer pid=48) INFO 08-03 23:14:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  468. (APIServer pid=48) INFO 08-03 23:14:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  469. (APIServer pid=48) INFO 08-03 23:15:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  470. (APIServer pid=48) INFO 08-03 23:15:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  471. (APIServer pid=48) INFO 08-03 23:15:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  472. (APIServer pid=48) INFO 08-03 23:15:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  473. (APIServer pid=48) INFO 08-03 23:15:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  474. (APIServer pid=48) INFO 08-03 23:15:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 78.9%
  475. (APIServer pid=48) INFO 08-03 23:16:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  476. (APIServer pid=48) INFO 08-03 23:16:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  477. (APIServer pid=48) INFO 08-03 23:16:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  478. (APIServer pid=48) INFO 08-03 23:16:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  479. (APIServer pid=48) INFO 08-03 23:16:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  480. (APIServer pid=48) INFO 08-03 23:16:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  481. (APIServer pid=48) INFO 08-03 23:17:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  482. (APIServer pid=48) INFO 08-03 23:17:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  483. (APIServer pid=48) INFO 08-03 23:17:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 78.9%
  484. (APIServer pid=48) INFO 08-03 23:17:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 78.9%
  485. (APIServer pid=48) INFO 08-03 23:17:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 78.9%
  486. (APIServer pid=48) INFO 08-03 23:17:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 78.9%
  487. (APIServer pid=48) INFO 08-03 23:18:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 78.9%
  488. (APIServer pid=48) INFO 08-03 23:18:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 78.9%
  489. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  490. (APIServer pid=48) INFO 08-03 23:18:28 [loggers.py:273] Engine 000: Avg prompt throughput: 433.2 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 80.3%
  491. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  492. (APIServer pid=48) INFO 08-03 23:18:38 [loggers.py:273] Engine 000: Avg prompt throughput: 828.7 tokens/s, Avg generation throughput: 15.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.1%
  493. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  494. (APIServer pid=48) INFO 08-03 23:18:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 81.1%
  495. (APIServer pid=48) INFO 08-03 23:18:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1666.9 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 81.0%
  496. (APIServer pid=48) INFO 08-03 23:19:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 81.0%
  497. (APIServer pid=48) INFO 08-03 23:19:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  498. (APIServer pid=48) INFO 08-03 23:19:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  499. (APIServer pid=48) INFO 08-03 23:19:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  500. (APIServer pid=48) INFO 08-03 23:19:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  501. (APIServer pid=48) INFO 08-03 23:19:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  502. (APIServer pid=48) INFO 08-03 23:20:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  503. (APIServer pid=48) INFO 08-03 23:20:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  504. (APIServer pid=48) INFO 08-03 23:20:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  505. (APIServer pid=48) INFO 08-03 23:20:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  506. (APIServer pid=48) INFO 08-03 23:20:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 81.0%
  507. (APIServer pid=48) INFO 08-03 23:20:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  508. (APIServer pid=48) INFO 08-03 23:21:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  509. (APIServer pid=48) INFO 08-03 23:21:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  510. (APIServer pid=48) INFO 08-03 23:21:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  511. (APIServer pid=48) INFO 08-03 23:21:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  512. (APIServer pid=48) INFO 08-03 23:21:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  513. (APIServer pid=48) INFO 08-03 23:21:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  514. (APIServer pid=48) INFO 08-03 23:22:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  515. (APIServer pid=48) INFO 08-03 23:22:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  516. (APIServer pid=48) INFO 08-03 23:22:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 81.0%
  517. (APIServer pid=48) INFO 08-03 23:22:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  518. (APIServer pid=48) INFO 08-03 23:22:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  519. (APIServer pid=48) INFO 08-03 23:22:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  520. (APIServer pid=48) INFO 08-03 23:23:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  521. (APIServer pid=48) INFO 08-03 23:23:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  522. (APIServer pid=48) INFO 08-03 23:23:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  523. (APIServer pid=48) INFO 08-03 23:23:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  524. (APIServer pid=48) INFO 08-03 23:23:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  525. (APIServer pid=48) INFO 08-03 23:23:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  526. (APIServer pid=48) INFO 08-03 23:24:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 81.0%
  527. (APIServer pid=48) INFO 08-03 23:24:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 81.0%
  528. (APIServer pid=48) INFO 08-03 23:24:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 81.0%
  529. (APIServer pid=48) INFO 08-03 23:24:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 81.0%
  530. (APIServer pid=48) INFO 08-03 23:24:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 81.0%
  531. (APIServer pid=48) INFO 08-03 23:24:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 81.0%
  532. (APIServer pid=48) INFO 08-03 23:25:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 81.0%
  533. (APIServer pid=48) INFO: 172.19.0.4:40398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  534. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  535. (APIServer pid=48) INFO 08-03 23:25:18 [loggers.py:273] Engine 000: Avg prompt throughput: 177.8 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 81.1%
  536. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  537. (APIServer pid=48) INFO 08-03 23:25:28 [loggers.py:273] Engine 000: Avg prompt throughput: 1046.3 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 80.2%
  538. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  539. (APIServer pid=48) INFO 08-03 23:25:38 [loggers.py:273] Engine 000: Avg prompt throughput: 586.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 80.2%
  540. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  541. (APIServer pid=48) INFO 08-03 23:25:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 79.5%
  542. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  543. (APIServer pid=48) INFO 08-03 23:25:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1361.4 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 79.5%
  544. (APIServer pid=48) INFO 08-03 23:26:08 [loggers.py:273] Engine 000: Avg prompt throughput: 785.5 tokens/s, Avg generation throughput: 18.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 79.8%
  545. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  546. (APIServer pid=48) INFO 08-03 23:26:18 [loggers.py:273] Engine 000: Avg prompt throughput: 892.8 tokens/s, Avg generation throughput: 15.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 80.1%
  547. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  548. (APIServer pid=48) INFO 08-03 23:26:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 79.8%
  549. (APIServer pid=48) INFO 08-03 23:26:38 [loggers.py:273] Engine 000: Avg prompt throughput: 1800.2 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 79.8%
  550. (APIServer pid=48) INFO 08-03 23:26:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 79.8%
  551. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  552. (APIServer pid=48) INFO 08-03 23:26:58 [loggers.py:273] Engine 000: Avg prompt throughput: 788.5 tokens/s, Avg generation throughput: 11.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 80.7%
  553. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  554. (APIServer pid=48) INFO 08-03 23:27:08 [loggers.py:273] Engine 000: Avg prompt throughput: 817.0 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.5%
  555. (APIServer pid=48) INFO 08-03 23:27:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.5%
  556. (APIServer pid=48) INFO 08-03 23:27:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.5%
  557. (APIServer pid=48) INFO 08-03 23:27:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.5%
  558. (APIServer pid=48) INFO 08-03 23:27:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.5%
  559. (APIServer pid=48) INFO 08-03 23:27:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.5%
  560. (APIServer pid=48) INFO 08-03 23:28:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 81.5%
  561. (APIServer pid=48) INFO 08-03 23:28:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  562. (APIServer pid=48) INFO 08-03 23:28:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  563. (APIServer pid=48) INFO 08-03 23:28:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  564. (APIServer pid=48) INFO 08-03 23:28:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  565. (APIServer pid=48) INFO 08-03 23:28:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  566. (APIServer pid=48) INFO 08-03 23:29:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  567. (APIServer pid=48) INFO 08-03 23:29:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  568. (APIServer pid=48) INFO 08-03 23:29:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  569. (APIServer pid=48) INFO 08-03 23:29:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  570. (APIServer pid=48) INFO 08-03 23:29:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  571. (APIServer pid=48) INFO 08-03 23:29:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 81.5%
  572. (APIServer pid=48) INFO 08-03 23:30:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  573. (APIServer pid=48) INFO 08-03 23:30:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  574. (APIServer pid=48) INFO 08-03 23:30:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  575. (APIServer pid=48) INFO 08-03 23:30:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  576. (APIServer pid=48) INFO 08-03 23:30:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  577. (APIServer pid=48) INFO 08-03 23:30:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  578. (APIServer pid=48) INFO 08-03 23:31:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  579. (APIServer pid=48) INFO 08-03 23:31:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  580. (APIServer pid=48) INFO 08-03 23:31:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  581. (APIServer pid=48) INFO 08-03 23:31:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 81.5%
  582. (APIServer pid=48) INFO 08-03 23:31:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  583. (APIServer pid=48) INFO 08-03 23:31:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  584. (APIServer pid=48) INFO 08-03 23:32:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  585. (APIServer pid=48) INFO 08-03 23:32:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  586. (APIServer pid=48) INFO 08-03 23:32:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  587. (APIServer pid=48) INFO 08-03 23:32:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  588. (APIServer pid=48) INFO 08-03 23:32:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  589. (APIServer pid=48) INFO 08-03 23:32:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  590. (APIServer pid=48) INFO 08-03 23:33:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  591. (APIServer pid=48) INFO 08-03 23:33:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  592. (APIServer pid=48) INFO 08-03 23:33:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 81.5%
  593. (APIServer pid=48) INFO: 172.19.0.4:55560 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  594. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  595. (APIServer pid=48) INFO 08-03 23:33:38 [loggers.py:273] Engine 000: Avg prompt throughput: 177.8 tokens/s, Avg generation throughput: 15.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 81.6%
  596. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  597. (APIServer pid=48) INFO 08-03 23:33:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 81.1%
  598. (APIServer pid=48) INFO 08-03 23:33:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1046.5 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 81.1%
  599. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  600. (APIServer pid=48) INFO 08-03 23:34:08 [loggers.py:273] Engine 000: Avg prompt throughput: 587.1 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 81.0%
  601. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  602. (APIServer pid=48) INFO 08-03 23:34:18 [loggers.py:273] Engine 000: Avg prompt throughput: 488.7 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 81.1%
  603. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  604. (APIServer pid=48) INFO 08-03 23:34:28 [loggers.py:273] Engine 000: Avg prompt throughput: 956.7 tokens/s, Avg generation throughput: 11.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 81.0%
  605. (APIServer pid=48) INFO 08-03 23:34:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 81.0%
  606. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  607. (APIServer pid=48) INFO 08-03 23:34:48 [loggers.py:273] Engine 000: Avg prompt throughput: 247.1 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 81.5%
  608. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  609. (APIServer pid=48) INFO 08-03 23:34:58 [loggers.py:273] Engine 000: Avg prompt throughput: 476.7 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  610. (APIServer pid=48) INFO 08-03 23:35:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  611. (APIServer pid=48) INFO 08-03 23:35:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  612. (APIServer pid=48) INFO 08-03 23:35:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  613. (APIServer pid=48) INFO 08-03 23:35:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  614. (APIServer pid=48) INFO 08-03 23:35:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  615. (APIServer pid=48) INFO 08-03 23:35:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  616. (APIServer pid=48) INFO 08-03 23:36:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  617. (APIServer pid=48) INFO 08-03 23:36:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  618. (APIServer pid=48) INFO 08-03 23:36:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 81.8%
  619. (APIServer pid=48) INFO 08-03 23:36:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.8%
  620. (APIServer pid=48) INFO 08-03 23:36:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.8%
  621. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  622. (APIServer pid=48) INFO 08-03 23:36:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1227.2 tokens/s, Avg generation throughput: 13.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 81.8%
  623. (APIServer pid=48) INFO 08-03 23:37:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 81.8%
  624. (APIServer pid=48) INFO 08-03 23:37:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 81.8%
  625. (APIServer pid=48) INFO 08-03 23:37:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  626. (APIServer pid=48) INFO 08-03 23:37:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  627. (APIServer pid=48) INFO 08-03 23:37:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  628. (APIServer pid=48) INFO 08-03 23:37:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  629. (APIServer pid=48) INFO 08-03 23:38:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  630. (APIServer pid=48) INFO 08-03 23:38:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  631. (APIServer pid=48) INFO 08-03 23:38:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  632. (APIServer pid=48) INFO 08-03 23:38:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  633. (APIServer pid=48) INFO 08-03 23:38:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.8%
  634. (APIServer pid=48) INFO 08-03 23:38:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  635. (APIServer pid=48) INFO 08-03 23:39:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  636. (APIServer pid=48) INFO 08-03 23:39:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  637. (APIServer pid=48) INFO 08-03 23:39:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  638. (APIServer pid=48) INFO 08-03 23:39:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  639. (APIServer pid=48) INFO 08-03 23:39:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  640. (APIServer pid=48) INFO 08-03 23:39:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  641. (APIServer pid=48) INFO 08-03 23:40:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  642. (APIServer pid=48) INFO 08-03 23:40:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  643. (APIServer pid=48) INFO 08-03 23:40:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 81.8%
  644. (APIServer pid=48) INFO 08-03 23:40:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  645. (APIServer pid=48) INFO 08-03 23:40:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  646. (APIServer pid=48) INFO 08-03 23:40:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  647. (APIServer pid=48) INFO 08-03 23:41:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  648. (APIServer pid=48) INFO 08-03 23:41:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  649. (APIServer pid=48) INFO 08-03 23:41:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  650. (APIServer pid=48) INFO 08-03 23:41:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  651. (APIServer pid=48) INFO 08-03 23:41:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  652. (APIServer pid=48) INFO 08-03 23:41:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  653. (APIServer pid=48) INFO 08-03 23:42:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 81.8%
  654. (APIServer pid=48) INFO 08-03 23:42:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 81.8%
  655. (APIServer pid=48) INFO 08-03 23:42:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 81.8%
  656. (APIServer pid=48) INFO 08-03 23:42:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 81.8%
  657. (APIServer pid=48) INFO 08-03 23:42:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 81.8%
  658. (APIServer pid=48) INFO 08-03 23:42:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 81.8%
  659. (APIServer pid=48) INFO: 172.19.0.4:35372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  660. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  661. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  662. (APIServer pid=48) INFO 08-03 23:43:08 [loggers.py:273] Engine 000: Avg prompt throughput: 177.8 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 81.4%
  663. (APIServer pid=48) INFO 08-03 23:43:18 [loggers.py:273] Engine 000: Avg prompt throughput: 1046.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 81.4%
  664. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  665. (APIServer pid=48) INFO 08-03 23:43:28 [loggers.py:273] Engine 000: Avg prompt throughput: 858.6 tokens/s, Avg generation throughput: 11.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 81.2%
  666. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  667. (APIServer pid=48) INFO 08-03 23:43:38 [loggers.py:273] Engine 000: Avg prompt throughput: 1171.9 tokens/s, Avg generation throughput: 15.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 80.9%
  668. (APIServer pid=48) INFO 08-03 23:43:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 80.9%
  669. (APIServer pid=48) INFO 08-03 23:43:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 80.9%
  670. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  671. (APIServer pid=48) INFO 08-03 23:44:08 [loggers.py:273] Engine 000: Avg prompt throughput: 297.4 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 81.3%
  672. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  673. (APIServer pid=48) INFO 08-03 23:44:18 [loggers.py:273] Engine 000: Avg prompt throughput: 731.6 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.5%
  674. (APIServer pid=48) INFO 08-03 23:44:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.5%
  675. (APIServer pid=48) INFO 08-03 23:44:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.5%
  676. (APIServer pid=48) INFO 08-03 23:44:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.5%
  677. (APIServer pid=48) INFO 08-03 23:44:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.5%
  678. (APIServer pid=48) INFO 08-03 23:45:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 81.5%
  679. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  680. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  681. (APIServer pid=48) INFO 08-03 23:45:18 [loggers.py:273] Engine 000: Avg prompt throughput: 735.6 tokens/s, Avg generation throughput: 15.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  682. (APIServer pid=48) INFO 08-03 23:45:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  683. (APIServer pid=48) INFO 08-03 23:45:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  684. (APIServer pid=48) INFO 08-03 23:45:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  685. (APIServer pid=48) INFO 08-03 23:45:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  686. (APIServer pid=48) INFO 08-03 23:46:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  687. (APIServer pid=48) INFO 08-03 23:46:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  688. (APIServer pid=48) INFO 08-03 23:46:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  689. (APIServer pid=48) INFO 08-03 23:46:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 82.3%
  690. (APIServer pid=48) INFO 08-03 23:46:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 82.3%
  691. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  692. (APIServer pid=48) INFO 08-03 23:46:58 [loggers.py:273] Engine 000: Avg prompt throughput: 756.1 tokens/s, Avg generation throughput: 15.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  693. (APIServer pid=48) INFO 08-03 23:47:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  694. (APIServer pid=48) INFO 08-03 23:47:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  695. (APIServer pid=48) INFO 08-03 23:47:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  696. (APIServer pid=48) INFO 08-03 23:47:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  697. (APIServer pid=48) INFO 08-03 23:47:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  698. (APIServer pid=48) INFO 08-03 23:47:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  699. (APIServer pid=48) INFO 08-03 23:48:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  700. (APIServer pid=48) INFO 08-03 23:48:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  701. (APIServer pid=48) INFO 08-03 23:48:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  702. (APIServer pid=48) INFO 08-03 23:48:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  703. (APIServer pid=48) INFO 08-03 23:48:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  704. (APIServer pid=48) INFO 08-03 23:48:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  705. (APIServer pid=48) INFO 08-03 23:49:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  706. (APIServer pid=48) INFO 08-03 23:49:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  707. (APIServer pid=48) INFO 08-03 23:49:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  708. (APIServer pid=48) INFO 08-03 23:49:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  709. (APIServer pid=48) INFO 08-03 23:49:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  710. (APIServer pid=48) INFO 08-03 23:49:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  711. (APIServer pid=48) INFO 08-03 23:50:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  712. (APIServer pid=48) INFO 08-03 23:50:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  713. (APIServer pid=48) INFO 08-03 23:50:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  714. (APIServer pid=48) INFO 08-03 23:50:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  715. (APIServer pid=48) INFO 08-03 23:50:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  716. (APIServer pid=48) INFO 08-03 23:50:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  717. (APIServer pid=48) INFO 08-03 23:51:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  718. (APIServer pid=48) INFO 08-03 23:51:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  719. (APIServer pid=48) INFO 08-03 23:51:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  720. (APIServer pid=48) INFO 08-03 23:51:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  721. (APIServer pid=48) INFO 08-03 23:51:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  722. (APIServer pid=48) INFO 08-03 23:51:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  723. (APIServer pid=48) INFO 08-03 23:52:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  724. (APIServer pid=48) INFO 08-03 23:52:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  725. (APIServer pid=48) INFO 08-03 23:52:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.6%
  726. (APIServer pid=48) INFO 08-03 23:52:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 82.6%
  727. (APIServer pid=48) INFO 08-03 23:52:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 82.6%
  728. (APIServer pid=48) INFO 08-03 23:52:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 82.6%
  729. (APIServer pid=48) INFO: 172.19.0.4:60434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  730. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  731. (APIServer pid=48) INFO 08-03 23:53:08 [loggers.py:273] Engine 000: Avg prompt throughput: 177.8 tokens/s, Avg generation throughput: 16.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 82.6%
  732. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  733. (APIServer pid=48) INFO 08-03 23:53:18 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.4 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 82.3%
  734. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  735. (APIServer pid=48) INFO 08-03 23:53:28 [loggers.py:273] Engine 000: Avg prompt throughput: 749.4 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 82.2%
  736. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  737. (APIServer pid=48) INFO 08-03 23:53:38 [loggers.py:273] Engine 000: Avg prompt throughput: 610.2 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 82.2%
  738. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  739. (APIServer pid=48) INFO 08-03 23:53:48 [loggers.py:273] Engine 000: Avg prompt throughput: 477.4 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 82.3%
  740. (APIServer pid=48) INFO 08-03 23:53:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 82.3%
  741. (APIServer pid=48) INFO 08-03 23:54:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 82.3%
  742. (APIServer pid=48) INFO 08-03 23:54:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.9%, Prefix cache hit rate: 82.3%
  743. (APIServer pid=48) INFO 08-03 23:54:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 82.3%
  744. (APIServer pid=48) INFO 08-03 23:54:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 82.3%
  745. (APIServer pid=48) INFO 08-03 23:54:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 82.3%
  746. (APIServer pid=48) INFO 08-03 23:54:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 82.3%
  747. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  748. (APIServer pid=48) INFO 08-03 23:55:08 [loggers.py:273] Engine 000: Avg prompt throughput: 902.7 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 82.3%
  749. (APIServer pid=48) INFO 08-03 23:55:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 82.3%
  750. (APIServer pid=48) INFO 08-03 23:55:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 82.3%
  751. (APIServer pid=48) INFO 08-03 23:55:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 82.3%
  752. (APIServer pid=48) INFO 08-03 23:55:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 82.3%
  753. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  754. (APIServer pid=48) INFO 08-03 23:55:58 [loggers.py:273] Engine 000: Avg prompt throughput: 982.3 tokens/s, Avg generation throughput: 15.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.4%
  755. (APIServer pid=48) INFO 08-03 23:56:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.4%
  756. (APIServer pid=48) INFO 08-03 23:56:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.4%
  757. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  758. (APIServer pid=48) INFO 08-03 23:56:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.4%
  759. (APIServer pid=48) INFO 08-03 23:56:38 [loggers.py:273] Engine 000: Avg prompt throughput: 568.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.7%
  760. (APIServer pid=48) INFO 08-03 23:56:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.7%
  761. (APIServer pid=48) INFO 08-03 23:56:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.7%
  762. (APIServer pid=48) INFO 08-03 23:57:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.7%
  763. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  764. (APIServer pid=48) INFO 08-03 23:57:18 [loggers.py:273] Engine 000: Avg prompt throughput: 1559.9 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.6%
  765. (APIServer pid=48) INFO 08-03 23:57:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.6%
  766. (APIServer pid=48) INFO 08-03 23:57:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.6%
  767. (APIServer pid=48) INFO 08-03 23:57:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.6%
  768. (APIServer pid=48) INFO 08-03 23:57:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  769. (APIServer pid=48) INFO 08-03 23:58:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  770. (APIServer pid=48) INFO 08-03 23:58:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  771. (APIServer pid=48) INFO 08-03 23:58:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  772. (APIServer pid=48) INFO 08-03 23:58:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  773. (APIServer pid=48) INFO 08-03 23:58:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  774. (APIServer pid=48) INFO 08-03 23:58:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  775. (APIServer pid=48) INFO 08-03 23:59:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  776. (APIServer pid=48) INFO 08-03 23:59:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  777. (APIServer pid=48) INFO 08-03 23:59:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.6%
  778. (APIServer pid=48) INFO 08-03 23:59:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  779. (APIServer pid=48) INFO 08-03 23:59:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  780. (APIServer pid=48) INFO 08-03 23:59:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  781. (APIServer pid=48) INFO 08-04 00:00:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  782. (APIServer pid=48) INFO 08-04 00:00:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  783. (APIServer pid=48) INFO 08-04 00:00:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  784. (APIServer pid=48) INFO 08-04 00:00:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  785. (APIServer pid=48) INFO 08-04 00:00:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  786. (APIServer pid=48) INFO 08-04 00:00:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  787. (APIServer pid=48) INFO 08-04 00:01:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  788. (APIServer pid=48) INFO 08-04 00:01:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.6%
  789. (APIServer pid=48) INFO 08-04 00:01:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  790. (APIServer pid=48) INFO 08-04 00:01:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  791. (APIServer pid=48) INFO 08-04 00:01:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  792. (APIServer pid=48) INFO 08-04 00:01:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  793. (APIServer pid=48) INFO 08-04 00:02:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  794. (APIServer pid=48) INFO 08-04 00:02:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  795. (APIServer pid=48) INFO 08-04 00:02:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  796. (APIServer pid=48) INFO 08-04 00:02:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  797. (APIServer pid=48) INFO 08-04 00:02:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  798. (APIServer pid=48) INFO 08-04 00:02:58 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 82.6%
  799. (APIServer pid=48) INFO 08-04 00:03:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 82.6%
  800. (APIServer pid=48) INFO 08-04 00:03:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 82.6%
  801. (APIServer pid=48) INFO 08-04 00:03:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 82.6%
  802. (APIServer pid=48) INFO: 172.19.0.4:42242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  803. (APIServer pid=48) INFO 08-04 00:03:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.6%
  804. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  805. (APIServer pid=48) INFO 08-04 00:03:49 [loggers.py:273] Engine 000: Avg prompt throughput: 1014.5 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.3%
  806. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  807. (APIServer pid=48) INFO 08-04 00:03:59 [loggers.py:273] Engine 000: Avg prompt throughput: 1046.3 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 82.0%
  808. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  809. (APIServer pid=48) INFO 08-04 00:04:09 [loggers.py:273] Engine 000: Avg prompt throughput: 1541.9 tokens/s, Avg generation throughput: 13.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 81.6%
  810. (APIServer pid=48) INFO 08-04 00:04:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 81.6%
  811. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  812. (APIServer pid=48) INFO 08-04 00:04:29 [loggers.py:273] Engine 000: Avg prompt throughput: 463.2 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 81.8%
  813. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  814. (APIServer pid=48) INFO 08-04 00:04:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 81.6%
  815. (APIServer pid=48) INFO 08-04 00:04:49 [loggers.py:273] Engine 000: Avg prompt throughput: 1563.5 tokens/s, Avg generation throughput: 18.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 81.6%
  816. (APIServer pid=48) INFO 08-04 00:04:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 81.6%
  817. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  818. (APIServer pid=48) INFO 08-04 00:05:09 [loggers.py:273] Engine 000: Avg prompt throughput: 222.7 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 81.9%
  819. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  820. (APIServer pid=48) INFO 08-04 00:05:19 [loggers.py:273] Engine 000: Avg prompt throughput: 53.4 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 82.3%
  821. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  822. (APIServer pid=48) INFO 08-04 00:05:29 [loggers.py:273] Engine 000: Avg prompt throughput: 586.8 tokens/s, Avg generation throughput: 16.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 82.6%
  823. (APIServer pid=48) INFO 08-04 00:05:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 82.6%
  824. (APIServer pid=48) INFO 08-04 00:05:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 82.6%
  825. (APIServer pid=48) INFO 08-04 00:05:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  826. (APIServer pid=48) INFO 08-04 00:06:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  827. (APIServer pid=48) INFO 08-04 00:06:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  828. (APIServer pid=48) INFO 08-04 00:06:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  829. (APIServer pid=48) INFO 08-04 00:06:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  830. (APIServer pid=48) INFO 08-04 00:06:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  831. (APIServer pid=48) INFO 08-04 00:06:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  832. (APIServer pid=48) INFO 08-04 00:07:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  833. (APIServer pid=48) INFO 08-04 00:07:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  834. (APIServer pid=48) INFO 08-04 00:07:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 82.6%
  835. (APIServer pid=48) INFO 08-04 00:07:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  836. (APIServer pid=48) INFO 08-04 00:07:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  837. (APIServer pid=48) INFO 08-04 00:07:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  838. (APIServer pid=48) INFO 08-04 00:08:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  839. (APIServer pid=48) INFO 08-04 00:08:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  840. (APIServer pid=48) INFO 08-04 00:08:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  841. (APIServer pid=48) INFO 08-04 00:08:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  842. (APIServer pid=48) INFO 08-04 00:08:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  843. (APIServer pid=48) INFO 08-04 00:08:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  844. (APIServer pid=48) INFO 08-04 00:09:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 82.6%
  845. (APIServer pid=48) INFO 08-04 00:09:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  846. (APIServer pid=48) INFO 08-04 00:09:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  847. (APIServer pid=48) INFO 08-04 00:09:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  848. (APIServer pid=48) INFO 08-04 00:09:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  849. (APIServer pid=48) INFO 08-04 00:09:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  850. (APIServer pid=48) INFO 08-04 00:10:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  851. (APIServer pid=48) INFO 08-04 00:10:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  852. (APIServer pid=48) INFO 08-04 00:10:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  853. (APIServer pid=48) INFO 08-04 00:10:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  854. (APIServer pid=48) INFO 08-04 00:10:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 82.6%
  855. (APIServer pid=48) INFO 08-04 00:10:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  856. (APIServer pid=48) INFO 08-04 00:11:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  857. (APIServer pid=48) INFO 08-04 00:11:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 82.6%
  858. (APIServer pid=48) INFO 08-04 00:11:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.6%
  859. (APIServer pid=48) INFO: 172.19.0.4:41928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  860. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  861. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  862. (APIServer pid=48) INFO 08-04 00:11:39 [loggers.py:273] Engine 000: Avg prompt throughput: 177.8 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 82.3%
  863. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  864. (APIServer pid=48) INFO 08-04 00:11:49 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.7 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.3%
  865. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  866. (APIServer pid=48) INFO 08-04 00:11:59 [loggers.py:273] Engine 000: Avg prompt throughput: 587.1 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 82.1%
  867. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  868. (APIServer pid=48) INFO 08-04 00:12:09 [loggers.py:273] Engine 000: Avg prompt throughput: 1251.1 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.1%
  869. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  870. (APIServer pid=48) INFO 08-04 00:12:19 [loggers.py:273] Engine 000: Avg prompt throughput: 545.5 tokens/s, Avg generation throughput: 17.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.2%
  871. (APIServer pid=48) INFO 08-04 00:12:29 [loggers.py:273] Engine 000: Avg prompt throughput: 1134.9 tokens/s, Avg generation throughput: 16.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  872. (APIServer pid=48) INFO 08-04 00:12:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  873. (APIServer pid=48) INFO 08-04 00:12:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  874. (APIServer pid=48) INFO 08-04 00:12:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  875. (APIServer pid=48) INFO 08-04 00:13:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  876. (APIServer pid=48) INFO 08-04 00:13:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  877. (APIServer pid=48) INFO 08-04 00:13:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  878. (APIServer pid=48) INFO 08-04 00:13:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 82.2%
  879. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  880. (APIServer pid=48) INFO 08-04 00:13:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 82.0%
  881. (APIServer pid=48) INFO 08-04 00:13:59 [loggers.py:273] Engine 000: Avg prompt throughput: 1787.8 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 82.0%
  882. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  883. (APIServer pid=48) INFO 08-04 00:14:09 [loggers.py:273] Engine 000: Avg prompt throughput: 673.9 tokens/s, Avg generation throughput: 15.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  884. (APIServer pid=48) INFO 08-04 00:14:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  885. (APIServer pid=48) INFO 08-04 00:14:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  886. (APIServer pid=48) INFO 08-04 00:14:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  887. (APIServer pid=48) INFO 08-04 00:14:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  888. (APIServer pid=48) INFO 08-04 00:14:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  889. (APIServer pid=48) INFO 08-04 00:15:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  890. (APIServer pid=48) INFO 08-04 00:15:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  891. (APIServer pid=48) INFO 08-04 00:15:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 82.3%
  892. (APIServer pid=48) INFO 08-04 00:15:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  893. (APIServer pid=48) INFO 08-04 00:15:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  894. (APIServer pid=48) INFO 08-04 00:15:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  895. (APIServer pid=48) INFO 08-04 00:16:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  896. (APIServer pid=48) INFO 08-04 00:16:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  897. (APIServer pid=48) INFO 08-04 00:16:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  898. (APIServer pid=48) INFO 08-04 00:16:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  899. (APIServer pid=48) INFO 08-04 00:16:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  900. (APIServer pid=48) INFO 08-04 00:16:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  901. (APIServer pid=48) INFO 08-04 00:17:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 82.3%
  902. (APIServer pid=48) INFO 08-04 00:17:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 82.3%
  903. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  904. (APIServer pid=48) INFO 08-04 00:17:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 82.4%
  905. (APIServer pid=48) INFO 08-04 00:17:39 [loggers.py:273] Engine 000: Avg prompt throughput: 1403.3 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 82.4%
  906. (APIServer pid=48) INFO 08-04 00:17:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 82.4%
  907. (APIServer pid=48) INFO 08-04 00:17:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 82.4%
  908. (APIServer pid=48) INFO 08-04 00:18:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 82.4%
  909. (APIServer pid=48) INFO 08-04 00:18:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 82.4%
  910. (APIServer pid=48) INFO 08-04 00:18:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 82.4%
  911. (APIServer pid=48) INFO 08-04 00:18:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  912. (APIServer pid=48) INFO 08-04 00:18:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  913. (APIServer pid=48) INFO 08-04 00:18:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  914. (APIServer pid=48) INFO 08-04 00:19:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  915. (APIServer pid=48) INFO 08-04 00:19:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  916. (APIServer pid=48) INFO 08-04 00:19:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  917. (APIServer pid=48) INFO 08-04 00:19:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  918. (APIServer pid=48) INFO 08-04 00:19:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  919. (APIServer pid=48) INFO 08-04 00:19:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  920. (APIServer pid=48) INFO 08-04 00:20:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  921. (APIServer pid=48) INFO 08-04 00:20:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 82.4%
  922. (APIServer pid=48) INFO 08-04 00:20:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  923. (APIServer pid=48) INFO 08-04 00:20:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  924. (APIServer pid=48) INFO 08-04 00:20:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  925. (APIServer pid=48) INFO 08-04 00:20:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  926. (APIServer pid=48) INFO 08-04 00:21:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  927. (APIServer pid=48) INFO 08-04 00:21:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  928. (APIServer pid=48) INFO 08-04 00:21:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  929. (APIServer pid=48) INFO 08-04 00:21:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  930. (APIServer pid=48) INFO 08-04 00:21:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  931. (APIServer pid=48) INFO 08-04 00:21:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  932. (APIServer pid=48) INFO 08-04 00:22:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 82.4%
  933. (APIServer pid=48) INFO 08-04 00:22:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  934. (APIServer pid=48) INFO 08-04 00:22:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  935. (APIServer pid=48) INFO 08-04 00:22:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  936. (APIServer pid=48) INFO 08-04 00:22:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  937. (APIServer pid=48) INFO 08-04 00:22:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  938. (APIServer pid=48) INFO 08-04 00:23:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  939. (APIServer pid=48) INFO 08-04 00:23:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  940. (APIServer pid=48) INFO 08-04 00:23:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  941. (APIServer pid=48) INFO 08-04 00:23:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  942. (APIServer pid=48) INFO 08-04 00:23:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  943. (APIServer pid=48) INFO 08-04 00:23:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 82.4%
  944. (APIServer pid=48) INFO 08-04 00:24:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 82.4%
  945. (APIServer pid=48) INFO: 172.19.0.4:49810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  946. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  947. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  948. (APIServer pid=48) INFO 08-04 00:24:19 [loggers.py:273] Engine 000: Avg prompt throughput: 177.8 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 82.4%
  949. (APIServer pid=48) INFO 08-04 00:24:29 [loggers.py:273] Engine 000: Avg prompt throughput: 1046.4 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 82.2%
  950. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  951. (APIServer pid=48) INFO 08-04 00:24:39 [loggers.py:273] Engine 000: Avg prompt throughput: 586.9 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 82.2%
  952. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  953. (APIServer pid=48) INFO 08-04 00:24:49 [loggers.py:273] Engine 000: Avg prompt throughput: 253.4 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 82.3%
  954. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  955. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  956. (APIServer pid=48) INFO 08-04 00:24:59 [loggers.py:273] Engine 000: Avg prompt throughput: 632.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 82.5%
  957. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  958. (APIServer pid=48) INFO 08-04 00:25:09 [loggers.py:273] Engine 000: Avg prompt throughput: 508.6 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 82.6%
  959. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  960. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  961. (APIServer pid=48) INFO 08-04 00:25:19 [loggers.py:273] Engine 000: Avg prompt throughput: 282.8 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 82.9%
  962. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  963. (APIServer pid=48) INFO 08-04 00:25:29 [loggers.py:273] Engine 000: Avg prompt throughput: 180.1 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 83.1%
  964. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  965. (APIServer pid=48) INFO 08-04 00:25:39 [loggers.py:273] Engine 000: Avg prompt throughput: 201.6 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 83.3%
  966. (APIServer pid=48) INFO 08-04 00:25:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 83.3%
  967. (APIServer pid=48) INFO 08-04 00:25:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 83.3%
  968. (APIServer pid=48) INFO 08-04 00:26:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 83.3%
  969. (APIServer pid=48) INFO 08-04 00:26:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  970. (APIServer pid=48) INFO 08-04 00:26:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  971. (APIServer pid=48) INFO 08-04 00:26:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  972. (APIServer pid=48) INFO 08-04 00:26:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  973. (APIServer pid=48) INFO 08-04 00:26:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  974. (APIServer pid=48) INFO 08-04 00:27:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  975. (APIServer pid=48) INFO 08-04 00:27:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  976. (APIServer pid=48) INFO 08-04 00:27:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  977. (APIServer pid=48) INFO 08-04 00:27:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  978. (APIServer pid=48) INFO 08-04 00:27:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.4%, Prefix cache hit rate: 83.3%
  979. (APIServer pid=48) INFO 08-04 00:27:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  980. (APIServer pid=48) INFO 08-04 00:28:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  981. (APIServer pid=48) INFO 08-04 00:28:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  982. (APIServer pid=48) INFO 08-04 00:28:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  983. (APIServer pid=48) INFO 08-04 00:28:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  984. (APIServer pid=48) INFO 08-04 00:28:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  985. (APIServer pid=48) INFO 08-04 00:28:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  986. (APIServer pid=48) INFO 08-04 00:29:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  987. (APIServer pid=48) INFO 08-04 00:29:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 83.3%
  988. (APIServer pid=48) INFO 08-04 00:29:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 83.3%
  989. (APIServer pid=48) INFO 08-04 00:29:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 83.3%
  990. (APIServer pid=48) INFO 08-04 00:29:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 83.3%
  991. (APIServer pid=48) INFO 08-04 00:29:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 83.3%
  992. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  993. (APIServer pid=48) INFO 08-04 00:30:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 83.3%
  994. (APIServer pid=48) INFO 08-04 00:30:19 [loggers.py:273] Engine 000: Avg prompt throughput: 1227.3 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  995. (APIServer pid=48) INFO 08-04 00:30:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  996. (APIServer pid=48) INFO 08-04 00:30:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  997. (APIServer pid=48) INFO 08-04 00:30:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  998. (APIServer pid=48) INFO 08-04 00:30:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  999. (APIServer pid=48) INFO 08-04 00:31:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  1000. (APIServer pid=48) INFO 08-04 00:31:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  1001. (APIServer pid=48) INFO 08-04 00:31:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  1002. (APIServer pid=48) INFO 08-04 00:31:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  1003. (APIServer pid=48) INFO 08-04 00:31:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  1004. (APIServer pid=48) INFO 08-04 00:31:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  1005. (APIServer pid=48) INFO 08-04 00:32:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1006. (APIServer pid=48) INFO 08-04 00:32:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1007. (APIServer pid=48) INFO 08-04 00:32:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1008. (APIServer pid=48) INFO 08-04 00:32:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1009. (APIServer pid=48) INFO 08-04 00:32:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1010. (APIServer pid=48) INFO 08-04 00:32:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1011. (APIServer pid=48) INFO 08-04 00:33:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1012. (APIServer pid=48) INFO 08-04 00:33:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1013. (APIServer pid=48) INFO 08-04 00:33:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.3%
  1014. (APIServer pid=48) INFO 08-04 00:33:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1015. (APIServer pid=48) INFO 08-04 00:33:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1016. (APIServer pid=48) INFO 08-04 00:33:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1017. (APIServer pid=48) INFO 08-04 00:34:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1018. (APIServer pid=48) INFO 08-04 00:34:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1019. (APIServer pid=48) INFO 08-04 00:34:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1020. (APIServer pid=48) INFO 08-04 00:34:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1021. (APIServer pid=48) INFO 08-04 00:34:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1022. (APIServer pid=48) INFO 08-04 00:34:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1023. (APIServer pid=48) INFO 08-04 00:35:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.3%
  1024. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1025. (APIServer pid=48) INFO 08-04 00:35:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 83.3%
  1026. (APIServer pid=48) INFO 08-04 00:35:29 [loggers.py:273] Engine 000: Avg prompt throughput: 778.8 tokens/s, Avg generation throughput: 16.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 83.4%
  1027. (APIServer pid=48) INFO 08-04 00:35:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 83.4%
  1028. (APIServer pid=48) INFO 08-04 00:35:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 83.4%
  1029. (APIServer pid=48) INFO 08-04 00:35:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 83.4%
  1030. (APIServer pid=48) INFO 08-04 00:36:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 83.4%
  1031. (APIServer pid=48) INFO 08-04 00:36:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1032. (APIServer pid=48) INFO 08-04 00:36:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1033. (APIServer pid=48) INFO 08-04 00:36:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1034. (APIServer pid=48) INFO 08-04 00:36:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1035. (APIServer pid=48) INFO 08-04 00:36:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1036. (APIServer pid=48) INFO 08-04 00:37:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1037. (APIServer pid=48) INFO 08-04 00:37:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1038. (APIServer pid=48) INFO 08-04 00:37:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1039. (APIServer pid=48) INFO 08-04 00:37:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1040. (APIServer pid=48) INFO 08-04 00:37:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.4%
  1041. (APIServer pid=48) INFO 08-04 00:37:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1042. (APIServer pid=48) INFO 08-04 00:38:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1043. (APIServer pid=48) INFO 08-04 00:38:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1044. (APIServer pid=48) INFO 08-04 00:38:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1045. (APIServer pid=48) INFO 08-04 00:38:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1046. (APIServer pid=48) INFO 08-04 00:38:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1047. (APIServer pid=48) INFO 08-04 00:38:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1048. (APIServer pid=48) INFO 08-04 00:39:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1049. (APIServer pid=48) INFO 08-04 00:39:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1050. (APIServer pid=48) INFO 08-04 00:39:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1051. (APIServer pid=48) INFO 08-04 00:39:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.4%
  1052. (APIServer pid=48) INFO 08-04 00:39:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1053. (APIServer pid=48) INFO 08-04 00:39:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1054. (APIServer pid=48) INFO 08-04 00:40:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1055. (APIServer pid=48) INFO 08-04 00:40:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1056. (APIServer pid=48) INFO 08-04 00:40:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1057. (APIServer pid=48) INFO 08-04 00:40:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1058. (APIServer pid=48) INFO 08-04 00:40:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1059. (APIServer pid=48) INFO 08-04 00:40:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1060. (APIServer pid=48) INFO 08-04 00:41:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1061. (APIServer pid=48) INFO 08-04 00:41:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 83.4%
  1062. (APIServer pid=48) INFO 08-04 00:41:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 83.4%
  1063. (APIServer pid=48) INFO 08-04 00:41:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 83.4%
  1064. (APIServer pid=48) INFO: 172.19.0.4:35536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1065. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1066. (APIServer pid=48) INFO 08-04 00:41:49 [loggers.py:273] Engine 000: Avg prompt throughput: 178.2 tokens/s, Avg generation throughput: 17.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 83.4%
  1067. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1068. (APIServer pid=48) INFO 08-04 00:41:59 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.1 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 83.3%
  1069. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1070. (APIServer pid=48) INFO 08-04 00:42:09 [loggers.py:273] Engine 000: Avg prompt throughput: 586.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 83.2%
  1071. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1072. (APIServer pid=48) INFO 08-04 00:42:19 [loggers.py:273] Engine 000: Avg prompt throughput: 1146.3 tokens/s, Avg generation throughput: 15.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 83.1%
  1073. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1074. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1075. (APIServer pid=48) INFO 08-04 00:42:29 [loggers.py:273] Engine 000: Avg prompt throughput: 426.7 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 83.2%
  1076. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1077. (APIServer pid=48) INFO 08-04 00:42:39 [loggers.py:273] Engine 000: Avg prompt throughput: 1420.9 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 83.3%
  1078. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1079. (APIServer pid=48) INFO 08-04 00:42:49 [loggers.py:273] Engine 000: Avg prompt throughput: 1077.1 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.3%
  1080. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1081. (APIServer pid=48) INFO 08-04 00:42:59 [loggers.py:273] Engine 000: Avg prompt throughput: 1586.8 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.3%
  1082. (APIServer pid=48) INFO 08-04 00:43:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.3%
  1083. (APIServer pid=48) INFO 08-04 00:43:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.3%
  1084. (APIServer pid=48) INFO 08-04 00:43:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.3%
  1085. (APIServer pid=48) INFO 08-04 00:43:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.3%
  1086. (APIServer pid=48) INFO 08-04 00:43:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 83.3%
  1087. (APIServer pid=48) INFO 08-04 00:43:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.3%
  1088. (APIServer pid=48) INFO 08-04 00:44:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.3%
  1089. (APIServer pid=48) INFO 08-04 00:44:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 83.3%
  1090. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1091. (APIServer pid=48) INFO 08-04 00:44:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 83.3%
  1092. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1093. (APIServer pid=48) INFO 08-04 00:44:39 [loggers.py:273] Engine 000: Avg prompt throughput: 807.0 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 83.8%
  1094. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1095. (APIServer pid=48) INFO 08-04 00:44:49 [loggers.py:273] Engine 000: Avg prompt throughput: 221.9 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.1%
  1096. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1097. (APIServer pid=48) INFO 08-04 00:44:59 [loggers.py:273] Engine 000: Avg prompt throughput: 57.5 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.5%
  1098. (APIServer pid=48) INFO 08-04 00:45:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.5%
  1099. (APIServer pid=48) INFO 08-04 00:45:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.5%
  1100. (APIServer pid=48) INFO 08-04 00:45:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.5%
  1101. (APIServer pid=48) INFO 08-04 00:45:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.5%
  1102. (APIServer pid=48) INFO 08-04 00:45:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.5%
  1103. (APIServer pid=48) INFO 08-04 00:45:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 84.5%
  1104. (APIServer pid=48) INFO 08-04 00:46:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1105. (APIServer pid=48) INFO 08-04 00:46:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1106. (APIServer pid=48) INFO 08-04 00:46:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1107. (APIServer pid=48) INFO 08-04 00:46:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1108. (APIServer pid=48) INFO 08-04 00:46:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1109. (APIServer pid=48) INFO 08-04 00:46:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1110. (APIServer pid=48) INFO 08-04 00:47:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1111. (APIServer pid=48) INFO 08-04 00:47:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1112. (APIServer pid=48) INFO 08-04 00:47:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1113. (APIServer pid=48) INFO 08-04 00:47:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1114. (APIServer pid=48) INFO 08-04 00:47:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 84.5%
  1115. (APIServer pid=48) INFO 08-04 00:47:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1116. (APIServer pid=48) INFO 08-04 00:48:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1117. (APIServer pid=48) INFO 08-04 00:48:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1118. (APIServer pid=48) INFO 08-04 00:48:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1119. (APIServer pid=48) INFO 08-04 00:48:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1120. (APIServer pid=48) INFO 08-04 00:48:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1121. (APIServer pid=48) INFO 08-04 00:48:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1122. (APIServer pid=48) INFO 08-04 00:49:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1123. (APIServer pid=48) INFO 08-04 00:49:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1124. (APIServer pid=48) INFO 08-04 00:49:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1125. (APIServer pid=48) INFO 08-04 00:49:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 84.5%
  1126. (APIServer pid=48) INFO 08-04 00:49:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1127. (APIServer pid=48) INFO 08-04 00:49:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1128. (APIServer pid=48) INFO 08-04 00:50:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1129. (APIServer pid=48) INFO 08-04 00:50:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1130. (APIServer pid=48) INFO 08-04 00:50:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1131. (APIServer pid=48) INFO 08-04 00:50:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1132. (APIServer pid=48) INFO 08-04 00:50:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1133. (APIServer pid=48) INFO 08-04 00:50:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1134. (APIServer pid=48) INFO 08-04 00:51:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1135. (APIServer pid=48) INFO 08-04 00:51:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 84.5%
  1136. (APIServer pid=48) INFO 08-04 00:51:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 84.5%
  1137. (APIServer pid=48) INFO: 172.19.0.4:39156 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1138. (APIServer pid=48) INFO: 172.19.0.4:52812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1139. (APIServer pid=48) INFO 08-04 00:51:39 [loggers.py:273] Engine 000: Avg prompt throughput: 178.2 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 84.5%
  1140. (APIServer pid=48) INFO: 172.19.0.4:52812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1141. (APIServer pid=48) INFO 08-04 00:51:49 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.1 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 84.3%
  1142. (APIServer pid=48) INFO: 172.19.0.4:52812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1143. (APIServer pid=48) INFO 08-04 00:51:59 [loggers.py:273] Engine 000: Avg prompt throughput: 801.8 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.4%, Prefix cache hit rate: 84.2%
  1144. (APIServer pid=48) INFO: 172.19.0.4:52812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1145. (APIServer pid=48) INFO 08-04 00:52:09 [loggers.py:273] Engine 000: Avg prompt throughput: 519.7 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 84.2%
  1146. (APIServer pid=48) INFO 08-04 00:52:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 84.2%
  1147. (APIServer pid=48) INFO 08-04 00:52:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 84.2%
  1148. (APIServer pid=48) INFO 08-04 00:52:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 84.2%
  1149. (APIServer pid=48) INFO: 172.19.0.4:52812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1150. (APIServer pid=48) INFO 08-04 00:52:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 84.1%
  1151. (APIServer pid=48) INFO 08-04 00:52:59 [loggers.py:273] Engine 000: Avg prompt throughput: 1163.1 tokens/s, Avg generation throughput: 18.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 84.1%
  1152. (APIServer pid=48) INFO 08-04 00:53:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 84.1%
  1153. (APIServer pid=48) INFO 08-04 00:53:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 84.1%
  1154. (APIServer pid=48) INFO 08-04 00:53:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 84.1%
  1155. (APIServer pid=48) INFO: 172.19.0.4:52812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  1156. (APIServer pid=48) INFO 08-04 00:53:39 [loggers.py:273] Engine 000: Avg prompt throughput: 1058.4 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1157. (APIServer pid=48) INFO 08-04 00:53:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1158. (APIServer pid=48) INFO 08-04 00:53:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1159. (APIServer pid=48) INFO 08-04 00:54:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1160. (APIServer pid=48) INFO 08-04 00:54:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1161. (APIServer pid=48) INFO 08-04 00:54:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1162. (APIServer pid=48) INFO 08-04 00:54:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1163. (APIServer pid=48) INFO 08-04 00:54:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1164. (APIServer pid=48) INFO 08-04 00:54:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1165. (APIServer pid=48) INFO 08-04 00:55:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 84.1%
  1166. (APIServer pid=48) INFO 08-04 00:55:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1167. (APIServer pid=48) INFO 08-04 00:55:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1168. (APIServer pid=48) INFO 08-04 00:55:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1169. (APIServer pid=48) INFO 08-04 00:55:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1170. (APIServer pid=48) INFO 08-04 00:55:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1171. (APIServer pid=48) INFO 08-04 00:56:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1172. (APIServer pid=48) INFO 08-04 00:56:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1173. (APIServer pid=48) INFO 08-04 00:56:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1174. (APIServer pid=48) INFO 08-04 00:56:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 84.1%
  1175. (APIServer pid=48) INFO 08-04 00:56:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1176. (APIServer pid=48) INFO 08-04 00:56:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1177. (APIServer pid=48) INFO 08-04 00:57:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1178. (APIServer pid=48) INFO 08-04 00:57:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1179. (APIServer pid=48) INFO 08-04 00:57:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1180. (APIServer pid=48) INFO 08-04 00:57:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1181. (APIServer pid=48) INFO 08-04 00:57:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1182. (APIServer pid=48) INFO 08-04 00:57:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1183. (APIServer pid=48) INFO 08-04 00:58:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1184. (APIServer pid=48) INFO 08-04 00:58:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.1%
  1185. (APIServer pid=48) INFO 08-04 00:58:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 84.1%
  1186. (APIServer pid=48) INFO 08-04 00:58:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 84.1%
  1187. (APIServer pid=48) INFO 08-04 00:58:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 84.1%
  1188. (APIServer pid=48) INFO 08-04 00:58:59 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 84.1%
  1189. (APIServer pid=48) INFO 08-04 00:59:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 84.1%
  1190. (APIServer pid=48) INFO 08-04 00:59:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 23.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 84.1%
  1191. (APIServer pid=48) INFO 08-04 00:59:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 84.1%
  1192.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!