API tools faq
paste
Guest User

with speculative ON

a guest
Aug 4th, 2026
5
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 241.67 KB | None | 0 0
raw download clone embed print report
  1. (APIServer pid=42) INFO 08-03 21:38:56 [api_utils.py:339]
  2. (APIServer pid=42) INFO 08-03 21:38:56 [api_utils.py:339] █ █ █▄ ▄█
  3. (APIServer pid=42) INFO 08-03 21:38:56 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.2.dev0+g752a3a504.d20260803
  4. (APIServer pid=42) INFO 08-03 21:38:56 [api_utils.py:339] █▄█▀ █ █ █ █ model poolside/Laguna-S-2.1-NVFP4
  5. (APIServer pid=42) INFO 08-03 21:38:56 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
  6. (APIServer pid=42) INFO 08-03 21:38:56 [api_utils.py:339]
  7. (APIServer pid=42) INFO 08-03 21:38:56 [api_utils.py:273] non-default args: {'model_tag': 'poolside/Laguna-S-2.1-NVFP4', 'default_chat_template_kwargs': {'enable_thinking': True}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'poolside_v1', 'host': '0.0.0.0', 'port': 8888, 'model': 'poolside/Laguna-S-2.1-NVFP4', 'max_model_len': 262144, 'reasoning_parser': 'poolside_v1', 'master_addr': '192.168.9.21', 'master_port': 25000, 'nnodes': 2, 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.85, 'max_num_batched_tokens': 4096, 'max_num_seqs': 8, 'speculative_config': {'model': 'poolside/Laguna-S-2.1-DFlash-NVFP4', 'num_speculative_tokens': 15}}
  8. (APIServer pid=42) WARNING 08-03 21:38:56 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_BASE_DIR
  9. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  10. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  11. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  12. (APIServer pid=42) INFO 08-03 21:39:03 [model.py:619] Resolved architecture: LagunaForCausalLM
  13. (APIServer pid=42) INFO 08-03 21:39:03 [model.py:1776] Using max model len 262144
  14. (APIServer pid=42) INFO 08-03 21:39:03 [arg_utils.py:2026] Inferred data_parallel_rank 0 from node_rank 0
  15. (APIServer pid=42) INFO 08-03 21:39:08 [model.py:619] Resolved architecture: DFlashLagunaForCausalLM
  16. (APIServer pid=42) INFO 08-03 21:39:08 [model.py:1776] Using max model len 1048576
  17. (APIServer pid=42) INFO 08-03 21:39:08 [speculative.py:1040] Overriding draft model max model len from 1048576 to 262144
  18. (APIServer pid=42) INFO 08-03 21:39:08 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4096.
  19. (APIServer pid=42) INFO 08-03 21:39:08 [vllm.py:1042] Asynchronous scheduling is enabled.
  20. (APIServer pid=42) INFO 08-03 21:39:08 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
  21. (APIServer pid=42) WARNING 08-03 21:39:08 [vllm.py:1648] max_num_scheduled_tokens is set to 3984 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
  22. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  23. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  24. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  25. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  26. (APIServer pid=42) WARNING 08-03 21:39:09 [vllm.py:1486] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implemented the `reasoning_start_str` and `reasoning_end_str`.
  27. (APIServer pid=42) INFO 08-03 21:39:09 [compilation.py:312] Enabled custom fusions: act_quant
  28. (EngineCore pid=156) INFO 08-03 21:39:13 [core.py:114] Initializing a V1 LLM engine (v0.25.2.dev0+g752a3a504.d20260803) with config: model='poolside/Laguna-S-2.1-NVFP4', speculative_config=SpeculativeConfig(method='dflash', model='poolside/Laguna-S-2.1-DFlash-NVFP4', num_spec_tokens=15), tokenizer='poolside/Laguna-S-2.1-NVFP4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=compressed-tensors, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='poolside_v1', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=poolside/Laguna-S-2.1-NVFP4, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4096], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')
  29. (EngineCore pid=156) INFO 08-03 21:39:13 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=192.168.9.21, mq_connect_ip=192.168.9.21 (local), world_size=2, local_world_size=1
  30. (Worker pid=176) INFO 08-03 21:39:17 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://192.168.9.21:25000 backend=nccl
  31. (Worker pid=176) INFO 08-03 21:39:51 [pynccl.py:113] vLLM is using nccl==2.30.7
  32. (Worker pid=176) WARNING 08-03 21:39:53 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.1 not supported, communicator is not available.
  33. (Worker pid=176) INFO 08-03 21:39:53 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
  34. (Worker pid=176) INFO 08-03 21:39:54 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
  35. (Worker pid=176) INFO 08-03 21:39:54 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
  36. (Worker pid=176) INFO 08-03 21:39:55 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
  37. (Worker pid=176) WARNING 08-03 21:39:55 [__init__.py:204] min_p and logit_bias parameters won't work with speculative decoding.
  38. (Worker_TP0 pid=176) INFO 08-03 21:39:55 [gpu_model_runner.py:5209] Starting to load model poolside/Laguna-S-2.1-NVFP4...
  39. (Worker_TP0 pid=176) INFO 08-03 21:39:55 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
  40. (Worker_TP0 pid=176) INFO 08-03 21:39:55 [flash_attn.py:718] Using FlashAttention version 2
  41. (Worker_TP0 pid=176) INFO 08-03 21:39:55 [nvfp4.py:285] Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION'].
  42. (Worker_TP0 pid=176) INFO 08-03 21:39:56 [unquantized.py:262] Using FlashInfer CUTLASS Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].
  43. (Worker_TP0 pid=176) INFO 08-03 21:39:58 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 92.85 GiB. Available RAM: 58.96 GiB.
  44. (Worker_TP0 pid=176) INFO 08-03 21:39:58 [weight_utils.py:879] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (92.85 GiB) exceeds 90% of available RAM (58.96 GiB).
  45. (Worker_TP0 pid=176)
  46. Loading safetensors checkpoint shards: 0% Completed | 0/49 [00:00<?, ?it/s]
  47. (Worker_TP0 pid=176)
  48. Loading safetensors checkpoint shards: 2% Completed | 1/49 [00:00<00:25, 1.88it/s]
  49. (Worker_TP0 pid=176)
  50. Loading safetensors checkpoint shards: 4% Completed | 2/49 [00:01<00:46, 1.01it/s]
  51. (Worker_TP0 pid=176)
  52. Loading safetensors checkpoint shards: 6% Completed | 3/49 [00:04<01:13, 1.59s/it]
  53. (Worker_TP0 pid=176)
  54. Loading safetensors checkpoint shards: 8% Completed | 4/49 [00:19<05:12, 6.95s/it]
  55. (Worker_TP0 pid=176)
  56. Loading safetensors checkpoint shards: 10% Completed | 5/49 [00:35<07:34, 10.32s/it]
  57. (Worker_TP0 pid=176)
  58. Loading safetensors checkpoint shards: 12% Completed | 6/49 [00:51<08:43, 12.17s/it]
  59. (Worker_TP0 pid=176)
  60. Loading safetensors checkpoint shards: 14% Completed | 7/49 [01:08<09:34, 13.67s/it]
  61. (Worker_TP0 pid=176)
  62. Loading safetensors checkpoint shards: 16% Completed | 8/49 [01:24<09:58, 14.59s/it]
  63. (Worker_TP0 pid=176)
  64. Loading safetensors checkpoint shards: 18% Completed | 9/49 [01:41<10:14, 15.37s/it]
  65. (Worker_TP0 pid=176)
  66. Loading safetensors checkpoint shards: 20% Completed | 10/49 [01:58<10:19, 15.88s/it]
  67. (Worker_TP0 pid=176)
  68. Loading safetensors checkpoint shards: 22% Completed | 11/49 [02:09<09:08, 14.43s/it]
  69. (Worker_TP0 pid=176)
  70. Loading safetensors checkpoint shards: 24% Completed | 12/49 [02:19<08:01, 13.01s/it]
  71. (Worker_TP0 pid=176)
  72. Loading safetensors checkpoint shards: 27% Completed | 13/49 [02:27<06:53, 11.48s/it]
  73. (Worker_TP0 pid=176)
  74. Loading safetensors checkpoint shards: 29% Completed | 14/49 [02:35<05:58, 10.23s/it]
  75. (Worker_TP0 pid=176)
  76. Loading safetensors checkpoint shards: 31% Completed | 15/49 [02:42<05:22, 9.49s/it]
  77. (Worker_TP0 pid=176)
  78. Loading safetensors checkpoint shards: 33% Completed | 16/49 [02:50<04:59, 9.08s/it]
  79. (Worker_TP0 pid=176)
  80. Loading safetensors checkpoint shards: 35% Completed | 17/49 [02:58<04:39, 8.74s/it]
  81. (Worker_TP0 pid=176)
  82. Loading safetensors checkpoint shards: 37% Completed | 18/49 [03:06<04:20, 8.40s/it]
  83. (Worker_TP0 pid=176)
  84. Loading safetensors checkpoint shards: 39% Completed | 19/49 [03:13<04:02, 8.08s/it]
  85. (Worker_TP0 pid=176)
  86. Loading safetensors checkpoint shards: 41% Completed | 20/49 [03:21<03:50, 7.97s/it]
  87. (Worker_TP0 pid=176)
  88. Loading safetensors checkpoint shards: 43% Completed | 21/49 [03:29<03:43, 7.99s/it]
  89. (Worker_TP0 pid=176)
  90. Loading safetensors checkpoint shards: 45% Completed | 22/49 [03:37<03:35, 7.97s/it]
  91. (Worker_TP0 pid=176)
  92. Loading safetensors checkpoint shards: 47% Completed | 23/49 [03:44<03:19, 7.69s/it]
  93. (Worker_TP0 pid=176)
  94. Loading safetensors checkpoint shards: 49% Completed | 24/49 [03:52<03:14, 7.77s/it]
  95. (Worker_TP0 pid=176)
  96. Loading safetensors checkpoint shards: 51% Completed | 25/49 [04:00<03:08, 7.84s/it]
  97. (Worker_TP0 pid=176)
  98. Loading safetensors checkpoint shards: 53% Completed | 26/49 [04:08<03:01, 7.91s/it]
  99. (Worker_TP0 pid=176)
  100. Loading safetensors checkpoint shards: 55% Completed | 27/49 [04:16<02:54, 7.95s/it]
  101. (Worker_TP0 pid=176)
  102. Loading safetensors checkpoint shards: 57% Completed | 28/49 [04:24<02:47, 7.97s/it]
  103. (Worker_TP0 pid=176)
  104. Loading safetensors checkpoint shards: 59% Completed | 29/49 [04:32<02:37, 7.89s/it]
  105. (Worker_TP0 pid=176)
  106. Loading safetensors checkpoint shards: 61% Completed | 30/49 [04:40<02:31, 7.95s/it]
  107. (Worker_TP0 pid=176)
  108. Loading safetensors checkpoint shards: 63% Completed | 31/49 [04:48<02:23, 7.98s/it]
  109. (Worker_TP0 pid=176)
  110. Loading safetensors checkpoint shards: 65% Completed | 32/49 [04:56<02:14, 7.94s/it]
  111. (Worker_TP0 pid=176)
  112. Loading safetensors checkpoint shards: 67% Completed | 33/49 [05:03<02:03, 7.75s/it]
  113. (Worker_TP0 pid=176)
  114. Loading safetensors checkpoint shards: 69% Completed | 34/49 [05:11<01:56, 7.76s/it]
  115. (Worker_TP0 pid=176)
  116. Loading safetensors checkpoint shards: 71% Completed | 35/49 [05:18<01:47, 7.68s/it]
  117. (Worker_TP0 pid=176)
  118. Loading safetensors checkpoint shards: 73% Completed | 36/49 [05:26<01:40, 7.76s/it]
  119. (Worker_TP0 pid=176)
  120. Loading safetensors checkpoint shards: 76% Completed | 37/49 [05:34<01:31, 7.58s/it]
  121. (Worker_TP0 pid=176)
  122. Loading safetensors checkpoint shards: 78% Completed | 38/49 [05:41<01:23, 7.57s/it]
  123. (Worker_TP0 pid=176)
  124. Loading safetensors checkpoint shards: 80% Completed | 39/49 [05:49<01:16, 7.61s/it]
  125. (Worker_TP0 pid=176)
  126. Loading safetensors checkpoint shards: 82% Completed | 40/49 [05:57<01:08, 7.66s/it]
  127. (Worker_TP0 pid=176)
  128. Loading safetensors checkpoint shards: 84% Completed | 41/49 [06:04<01:01, 7.72s/it]
  129. (Worker_TP0 pid=176)
  130. Loading safetensors checkpoint shards: 86% Completed | 42/49 [06:12<00:53, 7.70s/it]
  131. (Worker_TP0 pid=176)
  132. Loading safetensors checkpoint shards: 88% Completed | 43/49 [06:20<00:46, 7.83s/it]
  133. (Worker_TP0 pid=176)
  134. Loading safetensors checkpoint shards: 90% Completed | 44/49 [06:28<00:39, 7.93s/it]
  135. (Worker_TP0 pid=176)
  136. Loading safetensors checkpoint shards: 92% Completed | 45/49 [06:37<00:32, 8.01s/it]
  137. (Worker_TP0 pid=176)
  138. Loading safetensors checkpoint shards: 94% Completed | 46/49 [06:44<00:23, 7.78s/it]
  139. (Worker_TP0 pid=176)
  140. Loading safetensors checkpoint shards: 96% Completed | 47/49 [06:51<00:14, 7.49s/it]
  141. (Worker_TP0 pid=176)
  142. Loading safetensors checkpoint shards: 98% Completed | 48/49 [06:58<00:07, 7.60s/it]
  143. (Worker_TP0 pid=176)
  144. Loading safetensors checkpoint shards: 100% Completed | 49/49 [07:02<00:00, 6.27s/it]
  145. (Worker_TP0 pid=176)
  146. Loading safetensors checkpoint shards: 100% Completed | 49/49 [07:02<00:00, 8.61s/it]
  147. (Worker_TP0 pid=176)
  148. (Worker_TP0 pid=176) INFO 08-03 21:47:00 [default_loader.py:430] Loading weights took 422.16 seconds
  149. (Worker_TP0 pid=176) INFO 08-03 21:47:00 [nvfp4.py:543] Using MoEPrepareAndFinalizeNoDPEPModular
  150. (Worker_TP0 pid=176) INFO 08-03 21:47:01 [unquantized.py:334] Using MoEPrepareAndFinalizeNoDPEPModular
  151. (Worker_TP0 pid=176) INFO 08-03 21:47:01 [gpu_model_runner.py:5233] Loading drafter model...
  152. (Worker_TP0 pid=176) INFO 08-03 21:47:01 [vllm.py:1042] Asynchronous scheduling is enabled.
  153. (Worker_TP0 pid=176) INFO 08-03 21:47:01 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
  154. (Worker_TP0 pid=176) WARNING 08-03 21:47:01 [vllm.py:1648] max_num_scheduled_tokens is set to 3984 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
  155. (Worker_TP0 pid=176) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  156. (Worker_TP0 pid=176) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  157. (Worker_TP0 pid=176) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  158. (Worker_TP0 pid=176) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  159. (Worker_TP0 pid=176) WARNING 08-03 21:47:02 [vllm.py:1486] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implemented the `reasoning_start_str` and `reasoning_end_str`.
  160. (Worker_TP0 pid=176) INFO 08-03 21:47:02 [compilation.py:312] Enabled custom fusions: act_quant
  161. (Worker_TP0 pid=176) INFO 08-03 21:47:02 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
  162. (Worker_TP0 pid=176) WARNING 08-03 21:47:02 [laguna.py:389] Laguna config has sliding_attention layers but neither `swa_rope_parameters` nor a nested per-layer-type `rope_parameters` — SWA layers will reuse the global rope. If the checkpoint was trained with distinct SWA rope (theta / partial_rotary_factor), regenerate its HF config to include either form.
  163. (Worker_TP0 pid=176) INFO 08-03 21:47:03 [weight_utils.py:574] No model.safetensors.index.json found in remote.
  164. (Worker_TP0 pid=176) INFO 08-03 21:47:03 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 2.08 GiB. Available RAM: 55.06 GiB.
  165. (Worker_TP0 pid=176) INFO 08-03 21:47:03 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
  166. (Worker_TP0 pid=176)
  167. Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
  168. (Worker_TP0 pid=176)
  169. Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 76.98it/s]
  170. (Worker_TP0 pid=176)
  171. (Worker_TP0 pid=176) INFO 08-03 21:47:08 [default_loader.py:430] Loading weights took 5.40 seconds
  172. (Worker_TP0 pid=176) INFO 08-03 21:47:08 [llm_base_proposer.py:1437] Detected EAGLE model without its own embed_tokens in the checkpoint. Sharing target model embedding weights with the draft model.
  173. (Worker_TP0 pid=176) INFO 08-03 21:47:08 [llm_base_proposer.py:1512] Detected EAGLE model without its own lm_head in the checkpoint. Sharing target model lm_head weights with the draft model.
  174. (Worker_TP0 pid=176) INFO 08-03 21:47:08 [gpu_model_runner.py:5396] Using auxiliary layers from speculative config: (2, 11, 20, 30, 39, 48)
  175. (Worker_TP0 pid=176) INFO 08-03 21:47:09 [gpu_model_runner.py:5306] Model loading took 48.34 GiB memory and 433.303872 seconds
  176. (Worker_TP0 pid=176) INFO 08-03 21:47:15 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/5f36483c3e/rank_0_0/backbone for vLLM's torch.compile
  177. (Worker_TP0 pid=176) INFO 08-03 21:47:15 [backends.py:1148] Dynamo bytecode transform time: 6.51 s
  178. (Worker_TP0 pid=176) [rank0]:W0803 21:47:17.204000 176 torch/_inductor/utils.py:1731] Not enough SMs to use max_autotune_gemm mode
  179. (Worker_TP0 pid=176) INFO 08-03 21:47:22 [backends.py:378] Cache the graph of compile range (1, 4096) for later use
  180. (Worker_TP0 pid=176) INFO 08-03 21:47:41 [backends.py:393] Compiling a graph for compile range (1, 4096) takes 25.45 s
  181. (Worker_TP0 pid=176) INFO 08-03 21:47:47 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/3f805ea44217d88831964df4d164c8e68ce0e77648b3632252cda2afed5713aa/rank_0_0/model
  182. (Worker_TP0 pid=176) INFO 08-03 21:47:47 [monitor.py:53] torch.compile took 37.93 s in total
  183. (Worker_TP0 pid=176) INFO 08-03 21:47:50 [monitor.py:81] Initial profiling/warmup run took 3.45 s
  184. (Worker_TP0 pid=176) INFO 08-03 21:47:51 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/5f36483c3e/rank_0_0/eagle_head for vLLM's torch.compile
  185. (Worker_TP0 pid=176) INFO 08-03 21:47:51 [backends.py:1148] Dynamo bytecode transform time: 0.61 s
  186. (Worker_TP0 pid=176) INFO 08-03 21:47:57 [backends.py:393] Compiling a graph for compile range (1, 4096) takes 5.69 s
  187. (Worker_TP0 pid=176) INFO 08-03 21:47:58 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/4e23af0dd7f6e150a092f701723ddf15dd953ba984d99775df1913214811181b/rank_0_0/model
  188. (Worker_TP0 pid=176) INFO 08-03 21:47:58 [monitor.py:53] torch.compile took 7.47 s in total
  189. (Worker_TP0 pid=176) INFO 08-03 21:47:59 [monitor.py:81] Initial profiling/warmup run took 0.87 s
  190. (Worker_TP0 pid=176) INFO 08-03 21:48:04 [gpu_model_runner.py:6534] Profiling CUDA graph memory: PIECEWISE=16 (largest=256), FULL=8 (largest=128)
  191. (EngineCore pid=156) INFO 08-03 21:48:10 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
  192. (Worker_TP0 pid=176) INFO 08-03 21:48:56 [gpu_model_runner.py:6639] Estimated CUDA graph memory: 1.20 GiB total
  193. (Worker_TP0 pid=176) INFO 08-03 21:48:57 [gpu_worker.py:569] Available KV cache memory: 48.99 GiB
  194. (Worker_TP0 pid=176) INFO 08-03 21:48:57 [gpu_worker.py:584] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.8500 is equivalent to --gpu-memory-utilization=0.8401 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.8599. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
  195. (EngineCore pid=156) INFO 08-03 21:48:57 [kv_cache_utils.py:2146] GPU KV cache size: 1,372,137 tokens
  196. (EngineCore pid=156) INFO 08-03 21:48:57 [kv_cache_utils.py:2147] Maximum concurrency for 262,144 tokens per request: 5.23x
  197. (Worker_TP0 pid=176) INFO 08-03 21:48:57 [gpu_worker.py:739] Cleared 0.15 GiB of cached CUDA allocator memory before KV cache allocation.
  198. (Worker_TP0 pid=176) INFO 08-03 21:49:18 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
  199. (Worker_TP0 pid=176) INFO 08-03 21:49:18 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
  200. (Worker_TP0 pid=176) 2026-08-03 21:49:18,774 - INFO - autotuner.py:829 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
  201. (Worker_TP0 pid=176)
  202. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 0%| | 0/21 [00:00<?, ?profile/s]2026-08-03 21:49:22,642 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  203. (Worker_TP0 pid=176)
  204. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 5%|▍ | 1/21 [00:03<01:07, 3.38s/profile]2026-08-03 21:49:22,724 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  205. (Worker_TP0 pid=176) 2026-08-03 21:49:22,829 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  206. (Worker_TP0 pid=176)
  207. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 14%|█▍ | 3/21 [00:03<00:16, 1.06profile/s]2026-08-03 21:49:22,989 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  208. (Worker_TP0 pid=176)
  209. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 19%|█▉ | 4/21 [00:03<00:11, 1.49profile/s]2026-08-03 21:49:23,235 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  210. (Worker_TP0 pid=176)
  211. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 24%|██▍ | 5/21 [00:03<00:08, 1.88profile/s]2026-08-03 21:49:23,601 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  212. (Worker_TP0 pid=176)
  213. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 29%|██▊ | 6/21 [00:04<00:07, 2.09profile/s]2026-08-03 21:49:24,057 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  214. (Worker_TP0 pid=176)
  215. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 33%|███▎ | 7/21 [00:04<00:06, 2.12profile/s]2026-08-03 21:49:24,549 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  216. (Worker_TP0 pid=176)
  217. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 38%|███▊ | 8/21 [00:05<00:06, 2.09profile/s]2026-08-03 21:49:25,064 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  218. (Worker_TP0 pid=176)
  219. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 43%|████▎ | 9/21 [00:05<00:05, 2.04profile/s]2026-08-03 21:49:25,610 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  220. (Worker_TP0 pid=176)
  221. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 48%|████▊ | 10/21 [00:06<00:05, 1.97profile/s]2026-08-03 21:49:26,158 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  222. (Worker_TP0 pid=176)
  223. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 52%|█████▏ | 11/21 [00:06<00:05, 1.92profile/s]2026-08-03 21:49:26,726 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  224. (Worker_TP0 pid=176)
  225. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 57%|█████▋ | 12/21 [00:07<00:04, 1.88profile/s]2026-08-03 21:49:27,518 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  226. (Worker_TP0 pid=176)
  227. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 62%|██████▏ | 13/21 [00:08<00:04, 1.64profile/s]2026-08-03 21:49:28,391 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  228. (Worker_TP0 pid=176)
  229. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 67%|██████▋ | 14/21 [00:09<00:04, 1.45profile/s]2026-08-03 21:49:29,300 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  230. (Worker_TP0 pid=176)
  231. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 71%|███████▏ | 15/21 [00:10<00:04, 1.32profile/s]2026-08-03 21:49:29,939 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  232. (Worker_TP0 pid=176)
  233. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 76%|███████▌ | 16/21 [00:10<00:03, 1.39profile/s]2026-08-03 21:49:30,615 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  234. (Worker_TP0 pid=176)
  235. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 81%|████████ | 17/21 [00:11<00:02, 1.41profile/s]2026-08-03 21:49:31,350 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  236. (Worker_TP0 pid=176)
  237. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 86%|████████▌ | 18/21 [00:12<00:02, 1.40profile/s]2026-08-03 21:49:32,272 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  238. (Worker_TP0 pid=176)
  239. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 90%|█████████ | 19/21 [00:13<00:01, 1.29profile/s]2026-08-03 21:49:33,135 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  240. (Worker_TP0 pid=176)
  241. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 95%|█████████▌| 20/21 [00:13<00:00, 1.25profile/s]2026-08-03 21:49:35,432 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 4 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  242. (Worker_TP0 pid=176)
  243. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:16<00:00, 1.25s/profile]
  244. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:16<00:00, 1.30profile/s]
  245. (Worker_TP0 pid=176)
  246. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 0%| | 0/21 [00:00<?, ?profile/s]2026-08-03 21:49:35,645 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  247. (Worker_TP0 pid=176)
  248. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 5%|▍ | 1/21 [00:00<00:04, 4.93profile/s]2026-08-03 21:49:35,805 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  249. (Worker_TP0 pid=176)
  250. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 10%|▉ | 2/21 [00:00<00:03, 5.62profile/s]2026-08-03 21:49:36,013 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  251. (Worker_TP0 pid=176)
  252. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 14%|█▍ | 3/21 [00:00<00:03, 5.22profile/s]2026-08-03 21:49:36,330 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  253. (Worker_TP0 pid=176)
  254. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 19%|█▉ | 4/21 [00:00<00:04, 4.15profile/s]2026-08-03 21:49:36,809 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  255. (Worker_TP0 pid=176)
  256. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 24%|██▍ | 5/21 [00:01<00:05, 3.06profile/s]2026-08-03 21:49:37,487 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  257. (Worker_TP0 pid=176)
  258. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 29%|██▊ | 6/21 [00:02<00:06, 2.24profile/s]2026-08-03 21:49:38,136 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  259. (Worker_TP0 pid=176)
  260. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 33%|███▎ | 7/21 [00:02<00:07, 1.94profile/s]2026-08-03 21:49:38,853 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  261. (Worker_TP0 pid=176)
  262. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 38%|███▊ | 8/21 [00:03<00:07, 1.73profile/s]2026-08-03 21:49:39,751 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  263. (Worker_TP0 pid=176)
  264. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 43%|████▎ | 9/21 [00:04<00:08, 1.48profile/s]2026-08-03 21:49:40,796 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  265. (Worker_TP0 pid=176)
  266. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 48%|████▊ | 10/21 [00:05<00:08, 1.26profile/s]2026-08-03 21:49:41,929 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  267. (Worker_TP0 pid=176)
  268. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 52%|█████▏ | 11/21 [00:06<00:08, 1.12profile/s]2026-08-03 21:49:43,181 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  269. (Worker_TP0 pid=176)
  270. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 57%|█████▋ | 12/21 [00:07<00:09, 1.00s/profile]2026-08-03 21:49:44,099 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  271. (Worker_TP0 pid=176)
  272. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 62%|██████▏ | 13/21 [00:08<00:07, 1.02profile/s]2026-08-03 21:49:44,896 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  273. (Worker_TP0 pid=176)
  274. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 67%|██████▋ | 14/21 [00:09<00:06, 1.08profile/s]2026-08-03 21:49:45,725 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  275. (Worker_TP0 pid=176)
  276. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 71%|███████▏ | 15/21 [00:10<00:05, 1.12profile/s]2026-08-03 21:49:46,623 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  277. (Worker_TP0 pid=176)
  278. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 76%|███████▌ | 16/21 [00:11<00:04, 1.12profile/s]2026-08-03 21:49:47,630 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  279. (Worker_TP0 pid=176)
  280. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 81%|████████ | 17/21 [00:12<00:03, 1.08profile/s]2026-08-03 21:49:48,770 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  281. (Worker_TP0 pid=176)
  282. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 86%|████████▌ | 18/21 [00:13<00:02, 1.01profile/s]2026-08-03 21:49:50,209 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  283. (Worker_TP0 pid=176)
  284. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 90%|█████████ | 19/21 [00:14<00:02, 1.13s/profile]2026-08-03 21:49:51,669 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  285. (Worker_TP0 pid=176)
  286. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 95%|█████████▌| 20/21 [00:16<00:01, 1.23s/profile]2026-08-03 21:49:58,296 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 10 unsupported tactic(s) for trtllm::fused_moe::gemm2 (enable debug logs to see details)
  287. (Worker_TP0 pid=176)
  288. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:22<00:00, 2.85s/profile]
  289. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:22<00:00, 1.09s/profile]
  290. (Worker_TP0 pid=176)
  291. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 0%| | 0/21 [00:00<?, ?profile/s]2026-08-03 21:49:59,554 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  292. (Worker_TP0 pid=176)
  293. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 5%|▍ | 1/21 [00:00<00:06, 3.21profile/s]2026-08-03 21:49:59,624 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  294. (Worker_TP0 pid=176) 2026-08-03 21:49:59,753 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  295. (Worker_TP0 pid=176)
  296. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 14%|█▍ | 3/21 [00:00<00:02, 6.48profile/s]2026-08-03 21:49:59,925 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  297. (Worker_TP0 pid=176)
  298. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 19%|█▉ | 4/21 [00:00<00:02, 6.23profile/s]2026-08-03 21:50:00,205 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  299. (Worker_TP0 pid=176)
  300. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 24%|██▍ | 5/21 [00:00<00:03, 5.00profile/s]2026-08-03 21:50:00,624 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  301. (Worker_TP0 pid=176)
  302. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 29%|██▊ | 6/21 [00:01<00:04, 3.70profile/s]2026-08-03 21:50:01,162 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  303. (Worker_TP0 pid=176)
  304. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 33%|███▎ | 7/21 [00:01<00:04, 2.82profile/s]2026-08-03 21:50:01,745 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  305. (Worker_TP0 pid=176)
  306. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 38%|███▊ | 8/21 [00:02<00:05, 2.35profile/s]2026-08-03 21:50:02,337 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  307. (Worker_TP0 pid=176)
  308. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 43%|████▎ | 9/21 [00:03<00:05, 2.10profile/s]2026-08-03 21:50:02,945 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  309. (Worker_TP0 pid=176)
  310. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 48%|████▊ | 10/21 [00:03<00:05, 1.94profile/s]2026-08-03 21:50:03,579 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  311. (Worker_TP0 pid=176)
  312. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 52%|█████▏ | 11/21 [00:04<00:05, 1.81profile/s]2026-08-03 21:50:04,235 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  313. (Worker_TP0 pid=176)
  314. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 57%|█████▋ | 12/21 [00:04<00:05, 1.71profile/s]2026-08-03 21:50:04,905 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  315. (Worker_TP0 pid=176)
  316. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 62%|██████▏ | 13/21 [00:05<00:04, 1.64profile/s]2026-08-03 21:50:05,617 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  317. (Worker_TP0 pid=176)
  318. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 67%|██████▋ | 14/21 [00:06<00:04, 1.56profile/s]2026-08-03 21:50:06,476 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  319. (Worker_TP0 pid=176)
  320. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 71%|███████▏ | 15/21 [00:07<00:04, 1.42profile/s]2026-08-03 21:50:07,448 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  321. (Worker_TP0 pid=176)
  322. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 76%|███████▌ | 16/21 [00:08<00:03, 1.27profile/s]2026-08-03 21:50:08,456 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  323. (Worker_TP0 pid=176)
  324. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 81%|████████ | 17/21 [00:09<00:03, 1.17profile/s]2026-08-03 21:50:09,474 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  325. (Worker_TP0 pid=176)
  326. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 86%|████████▌ | 18/21 [00:10<00:02, 1.11profile/s]2026-08-03 21:50:10,614 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  327. (Worker_TP0 pid=176)
  328. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 90%|█████████ | 19/21 [00:11<00:01, 1.03profile/s]2026-08-03 21:50:11,797 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  329. (Worker_TP0 pid=176)
  330. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 95%|█████████▌| 20/21 [00:12<00:01, 1.04s/profile]2026-08-03 21:50:14,105 - INFO - autotuner.py:1699 - flashinfer.jit: [Autotuner]: Skipped 2 unsupported tactic(s) for trtllm::fused_moe::gemm1 (enable debug logs to see details)
  331. (Worker_TP0 pid=176)
  332. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:14<00:00, 1.42s/profile]
  333. [AutoTuner]: Tuning trtllm::fused_moe::gemm1: 100%|██████████| 21/21 [00:14<00:00, 1.41profile/s]
  334. (Worker_TP0 pid=176)
  335. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 0%| | 0/21 [00:00<?, ?profile/s]
  336. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 14%|█▍ | 3/21 [00:00<00:00, 26.92profile/s]
  337. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 29%|██▊ | 6/21 [00:00<00:01, 7.94profile/s]
  338. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 38%|███▊ | 8/21 [00:01<00:02, 4.71profile/s]
  339. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 43%|████▎ | 9/21 [00:01<00:03, 4.00profile/s]
  340. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 48%|████▊ | 10/21 [00:02<00:03, 3.47profile/s](EngineCore pid=156) INFO 08-03 21:50:16 [shm_broadcast.py:705] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).
  341.  
  342. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 52%|█████▏ | 11/21 [00:02<00:03, 3.07profile/s]
  343. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 57%|█████▋ | 12/21 [00:03<00:03, 2.78profile/s]
  344. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 62%|██████▏ | 13/21 [00:03<00:03, 2.55profile/s]
  345. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 67%|██████▋ | 14/21 [00:04<00:02, 2.36profile/s]
  346. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 71%|███████▏ | 15/21 [00:04<00:02, 2.19profile/s]
  347. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 76%|███████▌ | 16/21 [00:05<00:02, 2.01profile/s]
  348. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 81%|████████ | 17/21 [00:05<00:02, 1.86profile/s]
  349. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 86%|████████▌ | 18/21 [00:06<00:01, 1.74profile/s]
  350. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 90%|█████████ | 19/21 [00:07<00:01, 1.58profile/s]
  351. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 95%|█████████▌| 20/21 [00:08<00:00, 1.42profile/s]
  352. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:09<00:00, 1.09profile/s]
  353. [AutoTuner]: Tuning trtllm::fused_moe::gemm2: 100%|██████████| 21/21 [00:09<00:00, 2.19profile/s]
  354. (Worker_TP0 pid=176) 2026-08-03 21:50:23,867 - INFO - autotuner.py:852 - flashinfer.jit: [Autotuner]: Autotuning process ends
  355. (Worker_TP0 pid=176) INFO 08-03 21:50:27 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested.
  356. (Worker_TP0 pid=176)
  357. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/16 [00:00<?, ?it/s]
  358. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 6%|▋ | 1/16 [00:00<00:06, 2.38it/s]
  359. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 12%|█▎ | 2/16 [00:00<00:05, 2.43it/s]
  360. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 19%|█▉ | 3/16 [00:01<00:05, 2.59it/s]
  361. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 25%|██▌ | 4/16 [00:01<00:04, 2.64it/s]
  362. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 31%|███▏ | 5/16 [00:01<00:03, 2.83it/s]
  363. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 38%|███▊ | 6/16 [00:02<00:03, 2.83it/s]
  364. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 44%|████▍ | 7/16 [00:02<00:03, 3.00it/s]
  365. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 50%|█████ | 8/16 [00:02<00:02, 3.22it/s]
  366. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 56%|█████▋ | 9/16 [00:03<00:02, 3.41it/s]
  367. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 62%|██████▎ | 10/16 [00:03<00:01, 3.17it/s]
  368. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 69%|██████▉ | 11/16 [00:03<00:01, 3.04it/s]
  369. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 75%|███████▌ | 12/16 [00:04<00:01, 3.18it/s]
  370. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 81%|████████▏ | 13/16 [00:04<00:00, 3.40it/s]
  371. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 88%|████████▊ | 14/16 [00:04<00:00, 3.65it/s]
  372. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 94%|█████████▍| 15/16 [00:04<00:00, 3.69it/s]
  373. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 16/16 [00:05<00:00, 3.34it/s]
  374. Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 16/16 [00:05<00:00, 3.12it/s]
  375. (Worker_TP0 pid=176)
  376. Capturing CUDA graphs (decode, FULL): 0%| | 0/8 [00:00<?, ?it/s]
  377. Capturing CUDA graphs (decode, FULL): 12%|█▎ | 1/8 [00:00<00:02, 3.16it/s]
  378. Capturing CUDA graphs (decode, FULL): 25%|██▌ | 2/8 [00:00<00:01, 3.80it/s]
  379. Capturing CUDA graphs (decode, FULL): 38%|███▊ | 3/8 [00:00<00:01, 3.91it/s]
  380. Capturing CUDA graphs (decode, FULL): 50%|█████ | 4/8 [00:00<00:00, 4.23it/s]
  381. Capturing CUDA graphs (decode, FULL): 62%|██████▎ | 5/8 [00:01<00:00, 4.52it/s]
  382. Capturing CUDA graphs (decode, FULL): 75%|███████▌ | 6/8 [00:01<00:00, 4.67it/s]
  383. Capturing CUDA graphs (decode, FULL): 88%|████████▊ | 7/8 [00:01<00:00, 4.54it/s]
  384. Capturing CUDA graphs (decode, FULL): 100%|██████████| 8/8 [00:01<00:00, 4.82it/s]
  385. Capturing CUDA graphs (decode, FULL): 100%|██████████| 8/8 [00:01<00:00, 4.44it/s]
  386. (Worker_TP0 pid=176) INFO 08-03 21:50:44 [gpu_model_runner.py:6707] Graph capturing finished in 17 secs, took 1.13 GiB
  387. (Worker_TP0 pid=176) INFO 08-03 21:50:44 [gpu_worker.py:819] CUDA graph pool memory: 1.13 GiB (actual), 1.2 GiB (estimated), difference: 0.07 GiB (5.9%).
  388. (Worker_TP0 pid=176) INFO 08-03 21:50:44 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
  389. (EngineCore pid=156) INFO 08-03 21:50:45 [core.py:337] init engine (profile, create kv cache, warmup model) took 215.96 s (compilation: 45.39 s)
  390. (EngineCore pid=156) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  391. (EngineCore pid=156) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  392. (EngineCore pid=156) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  393. (EngineCore pid=156) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'full_attention', 'sliding_attention'}
  394. (EngineCore pid=156) INFO 08-03 21:50:55 [vllm.py:1042] Asynchronous scheduling is enabled.
  395. (EngineCore pid=156) INFO 08-03 21:50:55 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
  396. (EngineCore pid=156) WARNING 08-03 21:50:55 [vllm.py:1648] max_num_scheduled_tokens is set to 3984 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
  397. (EngineCore pid=156) WARNING 08-03 21:50:55 [vllm.py:1486] Auto-initialization of reasoning token IDs failed. Please check whether your reasoning parser has implemented the `reasoning_start_str` and `reasoning_end_str`.
  398. (EngineCore pid=156) INFO 08-03 21:50:55 [compilation.py:312] Enabled custom fusions: act_quant
  399. (APIServer pid=42) INFO 08-03 21:50:55 [api_server.py:612] Supported tasks: ['generate']
  400. (APIServer pid=42) INFO 08-03 21:50:59 [parser_manager.py:37] "auto" tool choice has been enabled.
  401. (APIServer pid=42) WARNING 08-03 21:51:00 [model.py:1528] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 1.0, 'min_p': 0.0}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
  402. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  403. (APIServer pid=42) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'sliding_attention', 'full_attention'}
  404. (APIServer pid=42) INFO 08-03 21:51:03 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
  405. (APIServer pid=42) INFO 08-03 21:51:03 [api_server.py:616] Starting vLLM server on http://0.0.0.0:8888
  406. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:37] Available routes are:
  407. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
  408. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /docs, Methods: GET, HEAD
  409. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
  410. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
  411. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /load, Methods: GET
  412. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /version, Methods: GET
  413. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /health, Methods: GET
  414. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /metrics, Methods: GET
  415. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /tokenize, Methods: POST
  416. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /detokenize, Methods: POST
  417. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/models, Methods: GET
  418. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /ping, Methods: GET
  419. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /ping, Methods: POST
  420. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /invocations, Methods: POST
  421. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
  422. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
  423. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/responses, Methods: POST
  424. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
  425. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
  426. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/completions, Methods: POST
  427. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/messages, Methods: POST
  428. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
  429. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /generative_scoring, Methods: POST
  430. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
  431. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
  432. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
  433. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/completions/render, Methods: POST
  434. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
  435. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
  436. (APIServer pid=42) INFO 08-03 21:51:03 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
  437. (APIServer pid=42) INFO: Started server process [42]
  438. (APIServer pid=42) INFO: Waiting for application startup.
  439. (APIServer pid=42) INFO: Application startup complete.
  440. (APIServer pid=42) INFO: 172.19.0.4:37918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  441. (Worker_TP0 pid=176) WARNING 08-03 21:52:04 [jit_monitor.py:129] Triton kernel JIT compilation during inference: eagle_prepare_next_token_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
  442. (Worker_TP0 pid=176) WARNING 08-03 21:52:04 [jit_monitor.py:129] Triton kernel JIT compilation during inference: copy_and_expand_dflash_inputs_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
  443. (Worker_TP0 pid=176) WARNING 08-03 21:52:05 [jit_monitor.py:129] Triton kernel JIT compilation during inference: expand_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
  444. (Worker_TP0 pid=176) WARNING 08-03 21:52:06 [jit_monitor.py:129] Triton kernel JIT compilation during inference: eagle_prepare_inputs_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
  445. (APIServer pid=42) INFO: 172.19.0.4:37920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  446. (APIServer pid=42) INFO 08-03 21:52:11 [loggers.py:273] Engine 000: Avg prompt throughput: 623.5 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.5%
  447. (APIServer pid=42) INFO 08-03 21:52:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.22, Accepted throughput: 0.07 tokens/s, Drafted throughput: 4.53 tokens/s, Accepted: 5 tokens, Drafted: 345 tokens, Per-position acceptance rate: 0.130, 0.043, 0.043, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.4%
  448. (APIServer pid=42) INFO: 172.19.0.4:37918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  449. (APIServer pid=42) INFO: 172.19.0.4:37918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  450. (APIServer pid=42) INFO 08-03 21:52:21 [loggers.py:273] Engine 000: Avg prompt throughput: 46.3 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 1.9%
  451. (APIServer pid=42) INFO 08-03 21:52:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.86, Accepted throughput: 5.60 tokens/s, Drafted throughput: 97.49 tokens/s, Accepted: 56 tokens, Drafted: 975 tokens, Per-position acceptance rate: 0.308, 0.154, 0.108, 0.092, 0.062, 0.046, 0.046, 0.015, 0.015, 0.015, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 5.7%
  452. (APIServer pid=42) INFO 08-03 21:52:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 1.9%
  453. (APIServer pid=42) INFO: 172.19.0.4:56278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  454. (APIServer pid=42) INFO 08-03 21:53:01 [loggers.py:273] Engine 000: Avg prompt throughput: 598.9 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 1.0%
  455. (APIServer pid=42) INFO 08-03 21:53:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.08, Accepted throughput: 0.12 tokens/s, Drafted throughput: 22.87 tokens/s, Accepted: 5 tokens, Drafted: 915 tokens, Per-position acceptance rate: 0.066, 0.016, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  456. (APIServer pid=42) INFO: 172.19.0.4:56278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  457. (APIServer pid=42) INFO 08-03 21:53:11 [loggers.py:273] Engine 000: Avg prompt throughput: 30.3 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 1.2%
  458. (APIServer pid=42) INFO 08-03 21:53:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.41, Accepted throughput: 7.90 tokens/s, Drafted throughput: 83.99 tokens/s, Accepted: 79 tokens, Drafted: 840 tokens, Per-position acceptance rate: 0.393, 0.268, 0.161, 0.143, 0.107, 0.089, 0.089, 0.054, 0.054, 0.054, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 9.4%
  459. (APIServer pid=42) INFO 08-03 21:53:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 1.2%
  460. (APIServer pid=42) INFO: 172.19.0.4:58342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  461. (APIServer pid=42) INFO 08-03 21:53:31 [loggers.py:273] Engine 000: Avg prompt throughput: 611.1 tokens/s, Avg generation throughput: 5.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.5%, Prefix cache hit rate: 0.8%
  462. (APIServer pid=42) INFO 08-03 21:53:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.20, Accepted throughput: 0.45 tokens/s, Drafted throughput: 33.00 tokens/s, Accepted: 9 tokens, Drafted: 660 tokens, Per-position acceptance rate: 0.205, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.4%
  463. (APIServer pid=42) INFO: 172.19.0.4:58342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  464. (APIServer pid=42) INFO 08-03 21:53:41 [loggers.py:273] Engine 000: Avg prompt throughput: 156.8 tokens/s, Avg generation throughput: 11.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 23.2%
  465. (APIServer pid=42) INFO 08-03 21:53:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.56, Accepted throughput: 4.20 tokens/s, Drafted throughput: 112.50 tokens/s, Accepted: 42 tokens, Drafted: 1125 tokens, Per-position acceptance rate: 0.307, 0.093, 0.040, 0.040, 0.013, 0.013, 0.013, 0.013, 0.013, 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 3.7%
  466. (APIServer pid=42) INFO: 172.19.0.4:58342 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  467. (APIServer pid=42) INFO 08-03 21:53:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 23.2%
  468. (APIServer pid=42) INFO 08-03 21:53:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.14, Accepted throughput: 3.30 tokens/s, Drafted throughput: 43.50 tokens/s, Accepted: 33 tokens, Drafted: 435 tokens, Per-position acceptance rate: 0.552, 0.241, 0.172, 0.103, 0.069, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 7.6%
  469. (APIServer pid=42) INFO 08-03 21:54:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 23.2%
  470. (APIServer pid=42) INFO: 172.19.0.4:50530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  471. (APIServer pid=42) INFO 08-03 21:54:21 [loggers.py:273] Engine 000: Avg prompt throughput: 695.3 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.6%, Prefix cache hit rate: 18.5%
  472. (APIServer pid=42) INFO 08-03 21:54:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.16, Accepted throughput: 0.13 tokens/s, Drafted throughput: 12.50 tokens/s, Accepted: 4 tokens, Drafted: 375 tokens, Per-position acceptance rate: 0.120, 0.040, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.1%
  473. (APIServer pid=42) INFO: 172.19.0.4:50530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  474. (APIServer pid=42) INFO 08-03 21:54:31 [loggers.py:273] Engine 000: Avg prompt throughput: 118.7 tokens/s, Avg generation throughput: 11.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 31.4%
  475. (APIServer pid=42) INFO 08-03 21:54:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.13, Accepted throughput: 1.30 tokens/s, Drafted throughput: 152.99 tokens/s, Accepted: 13 tokens, Drafted: 1530 tokens, Per-position acceptance rate: 0.118, 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.8%
  476. (APIServer pid=42) INFO 08-03 21:54:41 [loggers.py:273] Engine 000: Avg prompt throughput: 32.7 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 31.2%
  477. (APIServer pid=42) INFO 08-03 21:54:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.29, Accepted throughput: 3.30 tokens/s, Drafted throughput: 169.48 tokens/s, Accepted: 33 tokens, Drafted: 1695 tokens, Per-position acceptance rate: 0.150, 0.027, 0.018, 0.018, 0.018, 0.018, 0.018, 0.009, 0.009, 0.009, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.9%
  478. (APIServer pid=42) INFO: 172.19.0.4:50530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  479. (APIServer pid=42) INFO 08-03 21:54:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 31.2%
  480. (APIServer pid=42) INFO 08-03 21:54:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.76, Accepted throughput: 6.00 tokens/s, Drafted throughput: 50.99 tokens/s, Accepted: 60 tokens, Drafted: 510 tokens, Per-position acceptance rate: 0.559, 0.324, 0.235, 0.176, 0.176, 0.118, 0.059, 0.059, 0.059, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 11.8%
  481. (APIServer pid=42) INFO 08-03 21:55:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 31.2%
  482. (APIServer pid=42) INFO: 172.19.0.4:34010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  483. (APIServer pid=42) INFO: 172.19.0.4:34010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  484. (APIServer pid=42) INFO 08-03 21:55:31 [loggers.py:273] Engine 000: Avg prompt throughput: 906.9 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 35.5%
  485. (APIServer pid=42) INFO 08-03 21:55:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.22, Accepted throughput: 0.57 tokens/s, Drafted throughput: 39.00 tokens/s, Accepted: 23 tokens, Drafted: 1560 tokens, Per-position acceptance rate: 0.202, 0.019, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.5%
  486. (APIServer pid=42) INFO 08-03 21:55:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.7%, Prefix cache hit rate: 35.5%
  487. (APIServer pid=42) INFO 08-03 21:55:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.30, Accepted throughput: 3.50 tokens/s, Drafted throughput: 175.48 tokens/s, Accepted: 35 tokens, Drafted: 1755 tokens, Per-position acceptance rate: 0.248, 0.043, 0.009, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 2.0%
  488. (APIServer pid=42) INFO: 172.19.0.4:34010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  489. (APIServer pid=42) INFO 08-03 21:55:51 [loggers.py:273] Engine 000: Avg prompt throughput: 33.5 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 35.4%
  490. (APIServer pid=42) INFO 08-03 21:55:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.76, Accepted throughput: 7.90 tokens/s, Drafted throughput: 67.50 tokens/s, Accepted: 79 tokens, Drafted: 675 tokens, Per-position acceptance rate: 0.600, 0.378, 0.289, 0.178, 0.111, 0.067, 0.044, 0.022, 0.022, 0.022, 0.022, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 11.7%
  491. (APIServer pid=42) INFO: 172.19.0.4:33682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  492. (APIServer pid=42) INFO: 172.19.0.4:33682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  493. (APIServer pid=42) INFO 08-03 21:56:01 [loggers.py:273] Engine 000: Avg prompt throughput: 882.4 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 30.8%
  494. (APIServer pid=42) INFO 08-03 21:56:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.25, Accepted throughput: 1.30 tokens/s, Drafted throughput: 79.49 tokens/s, Accepted: 13 tokens, Drafted: 795 tokens, Per-position acceptance rate: 0.245, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.6%
  495. (APIServer pid=42) INFO 08-03 21:56:11 [loggers.py:273] Engine 000: Avg prompt throughput: 105.8 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 38.2%
  496. (APIServer pid=42) INFO 08-03 21:56:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.17, Accepted throughput: 1.90 tokens/s, Drafted throughput: 172.49 tokens/s, Accepted: 19 tokens, Drafted: 1725 tokens, Per-position acceptance rate: 0.157, 0.009, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.1%
  497. (APIServer pid=42) INFO 08-03 21:56:21 [loggers.py:273] Engine 000: Avg prompt throughput: 33.7 tokens/s, Avg generation throughput: 15.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 38.0%
  498. (APIServer pid=42) INFO 08-03 21:56:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.35, Accepted throughput: 3.90 tokens/s, Drafted throughput: 164.99 tokens/s, Accepted: 39 tokens, Drafted: 1650 tokens, Per-position acceptance rate: 0.264, 0.018, 0.009, 0.009, 0.009, 0.009, 0.009, 0.009, 0.009, 0.009, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 2.4%
  499. (APIServer pid=42) INFO: 172.19.0.4:52164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  500. (APIServer pid=42) INFO: 172.19.0.4:33682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  501. (APIServer pid=42) INFO 08-03 21:56:31 [loggers.py:273] Engine 000: Avg prompt throughput: 968.4 tokens/s, Avg generation throughput: 17.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 33.9%
  502. (APIServer pid=42) INFO 08-03 21:56:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.76, Accepted throughput: 7.40 tokens/s, Drafted throughput: 147.00 tokens/s, Accepted: 74 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.316, 0.143, 0.102, 0.071, 0.051, 0.031, 0.010, 0.010, 0.010, 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 5.0%
  503. (APIServer pid=42) INFO: 172.19.0.4:33682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  504. (APIServer pid=42) INFO 08-03 21:56:41 [loggers.py:273] Engine 000: Avg prompt throughput: 107.9 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 39.9%
  505. (APIServer pid=42) INFO 08-03 21:56:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.21, Accepted throughput: 2.10 tokens/s, Drafted throughput: 151.49 tokens/s, Accepted: 21 tokens, Drafted: 1515 tokens, Per-position acceptance rate: 0.198, 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.4%
  506. (APIServer pid=42) INFO 08-03 21:56:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 39.9%
  507. (APIServer pid=42) INFO 08-03 21:56:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.17, Accepted throughput: 1.90 tokens/s, Drafted throughput: 172.48 tokens/s, Accepted: 19 tokens, Drafted: 1725 tokens, Per-position acceptance rate: 0.165, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.1%
  508. (APIServer pid=42) INFO: 172.19.0.4:33682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  509. (APIServer pid=42) INFO 08-03 21:57:01 [loggers.py:273] Engine 000: Avg prompt throughput: 33.6 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 39.8%
  510. (APIServer pid=42) INFO 08-03 21:57:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.51, Accepted throughput: 8.30 tokens/s, Drafted throughput: 82.49 tokens/s, Accepted: 83 tokens, Drafted: 825 tokens, Per-position acceptance rate: 0.491, 0.218, 0.164, 0.127, 0.091, 0.091, 0.073, 0.055, 0.055, 0.055, 0.055, 0.018, 0.018, 0.000, 0.000, Avg Draft acceptance rate: 10.1%
  511. (APIServer pid=42) INFO 08-03 21:57:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 39.8%
  512. (APIServer pid=42) INFO: 172.19.0.4:51028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  513. (APIServer pid=42) INFO: 172.19.0.4:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  514. (APIServer pid=42) INFO 08-03 21:57:41 [loggers.py:273] Engine 000: Avg prompt throughput: 1259.3 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 35.3%
  515. (APIServer pid=42) INFO 08-03 21:57:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.80, Accepted throughput: 0.30 tokens/s, Drafted throughput: 5.62 tokens/s, Accepted: 12 tokens, Drafted: 225 tokens, Per-position acceptance rate: 0.467, 0.200, 0.133, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 5.3%
  516. (APIServer pid=42) INFO 08-03 21:57:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 35.3%
  517. (APIServer pid=42) INFO 08-03 21:57:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.08, Accepted throughput: 1.00 tokens/s, Drafted throughput: 176.98 tokens/s, Accepted: 10 tokens, Drafted: 1770 tokens, Per-position acceptance rate: 0.085, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.6%
  518. (APIServer pid=42) INFO: 172.19.0.4:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  519. (APIServer pid=42) INFO 08-03 21:58:01 [loggers.py:273] Engine 000: Avg prompt throughput: 1047.9 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 38.0%
  520. (APIServer pid=42) INFO 08-03 21:58:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.50 tokens/s, Drafted throughput: 121.49 tokens/s, Accepted: 5 tokens, Drafted: 1215 tokens, Per-position acceptance rate: 0.049, 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  521. (APIServer pid=42) INFO 08-03 21:58:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 38.0%
  522. (APIServer pid=42) INFO 08-03 21:58:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.07, Accepted throughput: 0.80 tokens/s, Drafted throughput: 169.50 tokens/s, Accepted: 8 tokens, Drafted: 1695 tokens, Per-position acceptance rate: 0.044, 0.027, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  523. (APIServer pid=42) INFO 08-03 21:58:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 38.0%
  524. (APIServer pid=42) INFO 08-03 21:58:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 119.98 tokens/s, Accepted: 1 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  525. (APIServer pid=42) INFO: 172.19.0.4:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  526. (APIServer pid=42) INFO 08-03 21:58:31 [loggers.py:273] Engine 000: Avg prompt throughput: 808.7 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 44.5%
  527. (APIServer pid=42) INFO 08-03 21:58:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 113.99 tokens/s, Accepted: 2 tokens, Drafted: 1140 tokens, Per-position acceptance rate: 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  528. (APIServer pid=42) INFO 08-03 21:58:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 44.5%
  529. (APIServer pid=42) INFO 08-03 21:58:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 156.00 tokens/s, Accepted: 0 tokens, Drafted: 1560 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  530. (APIServer pid=42) INFO: 172.19.0.4:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  531. (APIServer pid=42) INFO 08-03 21:58:51 [loggers.py:273] Engine 000: Avg prompt throughput: 395.9 tokens/s, Avg generation throughput: 9.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 52.1%
  532. (APIServer pid=42) INFO 08-03 21:58:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.50 tokens/s, Drafted throughput: 128.99 tokens/s, Accepted: 5 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.058, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  533. (APIServer pid=42) INFO 08-03 21:59:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 52.1%
  534. (APIServer pid=42) INFO 08-03 21:59:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.60 tokens/s, Drafted throughput: 145.48 tokens/s, Accepted: 6 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.062, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  535. (APIServer pid=42) INFO 08-03 21:59:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 52.1%
  536. (APIServer pid=42) INFO 08-03 21:59:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 151.49 tokens/s, Accepted: 1 tokens, Drafted: 1515 tokens, Per-position acceptance rate: 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  537. (APIServer pid=42) INFO 08-03 21:59:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 52.1%
  538. (APIServer pid=42) INFO 08-03 21:59:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 158.99 tokens/s, Accepted: 0 tokens, Drafted: 1590 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  539. (APIServer pid=42) INFO: 172.19.0.4:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  540. (APIServer pid=42) INFO 08-03 21:59:31 [loggers.py:273] Engine 000: Avg prompt throughput: 407.9 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 58.2%
  541. (APIServer pid=42) INFO 08-03 21:59:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 124.49 tokens/s, Accepted: 1 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  542. (APIServer pid=42) INFO 08-03 21:59:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 58.2%
  543. (APIServer pid=42) INFO 08-03 21:59:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 148.48 tokens/s, Accepted: 3 tokens, Drafted: 1485 tokens, Per-position acceptance rate: 0.030, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  544. (APIServer pid=42) INFO 08-03 21:59:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 58.2%
  545. (APIServer pid=42) INFO 08-03 21:59:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 142.49 tokens/s, Accepted: 3 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  546. (APIServer pid=42) INFO 08-03 22:00:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 58.2%
  547. (APIServer pid=42) INFO 08-03 22:00:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.50 tokens/s, Drafted throughput: 144.00 tokens/s, Accepted: 5 tokens, Drafted: 1440 tokens, Per-position acceptance rate: 0.052, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  548. (APIServer pid=42) INFO 08-03 22:00:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 58.2%
  549. (APIServer pid=42) INFO 08-03 22:00:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 142.50 tokens/s, Accepted: 4 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.042, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  550. (APIServer pid=42) INFO 08-03 22:00:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 58.2%
  551. (APIServer pid=42) INFO 08-03 22:00:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 148.49 tokens/s, Accepted: 0 tokens, Drafted: 1485 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  552. (APIServer pid=42) INFO 08-03 22:00:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  553. (APIServer pid=42) INFO 08-03 22:00:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 2 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  554. (APIServer pid=42) INFO 08-03 22:00:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  555. (APIServer pid=42) INFO 08-03 22:00:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 151.48 tokens/s, Accepted: 4 tokens, Drafted: 1515 tokens, Per-position acceptance rate: 0.040, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  556. (APIServer pid=42) INFO 08-03 22:00:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  557. (APIServer pid=42) INFO 08-03 22:00:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 149.99 tokens/s, Accepted: 2 tokens, Drafted: 1500 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  558. (APIServer pid=42) INFO 08-03 22:01:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  559. (APIServer pid=42) INFO 08-03 22:01:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 149.98 tokens/s, Accepted: 0 tokens, Drafted: 1500 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  560. (APIServer pid=42) INFO 08-03 22:01:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  561. (APIServer pid=42) INFO 08-03 22:01:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.60 tokens/s, Drafted throughput: 149.99 tokens/s, Accepted: 6 tokens, Drafted: 1500 tokens, Per-position acceptance rate: 0.060, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  562. (APIServer pid=42) INFO 08-03 22:01:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  563. (APIServer pid=42) INFO 08-03 22:01:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 155.98 tokens/s, Accepted: 3 tokens, Drafted: 1560 tokens, Per-position acceptance rate: 0.029, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  564. (APIServer pid=42) INFO 08-03 22:01:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  565. (APIServer pid=42) INFO 08-03 22:01:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 155.99 tokens/s, Accepted: 2 tokens, Drafted: 1560 tokens, Per-position acceptance rate: 0.019, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  566. (APIServer pid=42) INFO 08-03 22:01:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  567. (APIServer pid=42) INFO 08-03 22:01:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.50 tokens/s, Drafted throughput: 155.99 tokens/s, Accepted: 5 tokens, Drafted: 1560 tokens, Per-position acceptance rate: 0.048, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  568. (APIServer pid=42) INFO 08-03 22:01:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  569. (APIServer pid=42) INFO 08-03 22:01:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.07, Accepted throughput: 0.70 tokens/s, Drafted throughput: 152.99 tokens/s, Accepted: 7 tokens, Drafted: 1530 tokens, Per-position acceptance rate: 0.069, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  570. (APIServer pid=42) INFO 08-03 22:02:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  571. (APIServer pid=42) INFO 08-03 22:02:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 152.99 tokens/s, Accepted: 4 tokens, Drafted: 1530 tokens, Per-position acceptance rate: 0.039, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  572. (APIServer pid=42) INFO 08-03 22:02:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 58.2%
  573. (APIServer pid=42) INFO 08-03 22:02:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.50 tokens/s, Drafted throughput: 154.49 tokens/s, Accepted: 5 tokens, Drafted: 1545 tokens, Per-position acceptance rate: 0.049, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  574. (APIServer pid=42) INFO: 172.19.0.4:51040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  575. (APIServer pid=42) INFO 08-03 22:02:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 58.2%
  576. (APIServer pid=42) INFO 08-03 22:02:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 0 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  577. (APIServer pid=42) INFO 08-03 22:02:31 [loggers.py:273] Engine 000: Avg prompt throughput: 990.8 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 62.1%
  578. (APIServer pid=42) INFO 08-03 22:02:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 103.49 tokens/s, Accepted: 0 tokens, Drafted: 1035 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  579. (APIServer pid=42) INFO 08-03 22:02:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 62.1%
  580. (APIServer pid=42) INFO 08-03 22:02:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 3 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  581. (APIServer pid=42) INFO 08-03 22:02:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 62.1%
  582. (APIServer pid=42) INFO 08-03 22:02:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.50 tokens/s, Drafted throughput: 142.50 tokens/s, Accepted: 5 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.053, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  583. (APIServer pid=42) INFO 08-03 22:03:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 62.1%
  584. (APIServer pid=42) INFO 08-03 22:03:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 145.48 tokens/s, Accepted: 4 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.041, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  585. (APIServer pid=42) INFO 08-03 22:03:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 62.1%
  586. (APIServer pid=42) INFO 08-03 22:03:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 140.99 tokens/s, Accepted: 3 tokens, Drafted: 1410 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  587. (APIServer pid=42) INFO: 172.19.0.4:60762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  588. (APIServer pid=42) INFO 08-03 22:03:21 [loggers.py:273] Engine 000: Avg prompt throughput: 1059.5 tokens/s, Avg generation throughput: 10.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 59.9%
  589. (APIServer pid=42) INFO 08-03 22:03:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.08, Accepted throughput: 0.80 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 8 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.071, 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  590. (APIServer pid=42) INFO 08-03 22:03:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 59.9%
  591. (APIServer pid=42) INFO 08-03 22:03:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.11, Accepted throughput: 1.60 tokens/s, Drafted throughput: 215.98 tokens/s, Accepted: 16 tokens, Drafted: 2160 tokens, Per-position acceptance rate: 0.111, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.7%
  592. (APIServer pid=42) INFO 08-03 22:03:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 59.9%
  593. (APIServer pid=42) INFO 08-03 22:03:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.13, Accepted throughput: 1.90 tokens/s, Drafted throughput: 221.99 tokens/s, Accepted: 19 tokens, Drafted: 2220 tokens, Per-position acceptance rate: 0.128, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.9%
  594. (APIServer pid=42) INFO 08-03 22:03:51 [loggers.py:273] Engine 000: Avg prompt throughput: 32.1 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  595. (APIServer pid=42) INFO 08-03 22:03:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.29, Accepted throughput: 4.10 tokens/s, Drafted throughput: 208.48 tokens/s, Accepted: 41 tokens, Drafted: 2085 tokens, Per-position acceptance rate: 0.144, 0.022, 0.022, 0.014, 0.014, 0.014, 0.014, 0.014, 0.014, 0.014, 0.007, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 2.0%
  596. (APIServer pid=42) INFO: 172.19.0.4:60762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  597. (APIServer pid=42) INFO 08-03 22:04:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  598. (APIServer pid=42) INFO 08-03 22:04:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.43, Accepted throughput: 4.80 tokens/s, Drafted throughput: 166.47 tokens/s, Accepted: 48 tokens, Drafted: 1665 tokens, Per-position acceptance rate: 0.162, 0.108, 0.072, 0.054, 0.027, 0.009, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 2.9%
  599. (APIServer pid=42) INFO 08-03 22:04:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  600. (APIServer pid=42) INFO 08-03 22:04:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.50 tokens/s, Drafted throughput: 145.48 tokens/s, Accepted: 5 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.052, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  601. (APIServer pid=42) INFO 08-03 22:04:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  602. (APIServer pid=42) INFO 08-03 22:04:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.50 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 5 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.051, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  603. (APIServer pid=42) INFO 08-03 22:04:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  604. (APIServer pid=42) INFO 08-03 22:04:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 0 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  605. (APIServer pid=42) INFO 08-03 22:04:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  606. (APIServer pid=42) INFO 08-03 22:04:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 2 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  607. (APIServer pid=42) INFO 08-03 22:04:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  608. (APIServer pid=42) INFO 08-03 22:04:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 145.48 tokens/s, Accepted: 2 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.021, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  609. (APIServer pid=42) INFO 08-03 22:05:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  610. (APIServer pid=42) INFO 08-03 22:05:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 145.48 tokens/s, Accepted: 0 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  611. (APIServer pid=42) INFO 08-03 22:05:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  612. (APIServer pid=42) INFO 08-03 22:05:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 2 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.021, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  613. (APIServer pid=42) INFO 08-03 22:05:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 59.8%
  614. (APIServer pid=42) INFO 08-03 22:05:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 2 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.021, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  615. (APIServer pid=42) INFO 08-03 22:05:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  616. (APIServer pid=42) INFO 08-03 22:05:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 4 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.041, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  617. (APIServer pid=42) INFO 08-03 22:05:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  618. (APIServer pid=42) INFO 08-03 22:05:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 148.49 tokens/s, Accepted: 0 tokens, Drafted: 1485 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  619. (APIServer pid=42) INFO 08-03 22:05:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  620. (APIServer pid=42) INFO 08-03 22:05:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 145.50 tokens/s, Accepted: 1 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  621. (APIServer pid=42) INFO 08-03 22:06:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  622. (APIServer pid=42) INFO 08-03 22:06:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 142.49 tokens/s, Accepted: 4 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.042, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  623. (APIServer pid=42) INFO 08-03 22:06:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  624. (APIServer pid=42) INFO 08-03 22:06:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 143.98 tokens/s, Accepted: 4 tokens, Drafted: 1440 tokens, Per-position acceptance rate: 0.042, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  625. (APIServer pid=42) INFO 08-03 22:06:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  626. (APIServer pid=42) INFO 08-03 22:06:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 143.99 tokens/s, Accepted: 1 tokens, Drafted: 1440 tokens, Per-position acceptance rate: 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  627. (APIServer pid=42) INFO 08-03 22:06:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  628. (APIServer pid=42) INFO 08-03 22:06:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 143.99 tokens/s, Accepted: 1 tokens, Drafted: 1440 tokens, Per-position acceptance rate: 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  629. (APIServer pid=42) INFO 08-03 22:06:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  630. (APIServer pid=42) INFO 08-03 22:06:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 1 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  631. (APIServer pid=42) INFO 08-03 22:06:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  632. (APIServer pid=42) INFO 08-03 22:06:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 145.48 tokens/s, Accepted: 3 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  633. (APIServer pid=42) INFO 08-03 22:07:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  634. (APIServer pid=42) INFO 08-03 22:07:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 3 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  635. (APIServer pid=42) INFO 08-03 22:07:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  636. (APIServer pid=42) INFO 08-03 22:07:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 3 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  637. (APIServer pid=42) INFO 08-03 22:07:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 59.8%
  638. (APIServer pid=42) INFO 08-03 22:07:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 3 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  639. (APIServer pid=42) INFO: 172.19.0.4:38216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  640. (APIServer pid=42) INFO: 172.19.0.4:38228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  641. (APIServer pid=42) INFO 08-03 22:07:31 [loggers.py:273] Engine 000: Avg prompt throughput: 181.0 tokens/s, Avg generation throughput: 9.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 60.8%
  642. (APIServer pid=42) INFO 08-03 22:07:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.20, Accepted throughput: 1.50 tokens/s, Drafted throughput: 112.49 tokens/s, Accepted: 15 tokens, Drafted: 1125 tokens, Per-position acceptance rate: 0.120, 0.053, 0.027, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.3%
  643. (APIServer pid=42) INFO 08-03 22:07:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 60.8%
  644. (APIServer pid=42) INFO 08-03 22:07:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.70 tokens/s, Drafted throughput: 173.99 tokens/s, Accepted: 7 tokens, Drafted: 1740 tokens, Per-position acceptance rate: 0.043, 0.017, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  645. (APIServer pid=42) INFO: 172.19.0.4:38228 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  646. (APIServer pid=42) INFO 08-03 22:07:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 60.3%
  647. (APIServer pid=42) INFO 08-03 22:07:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 24.00 tokens/s, Accepted: 0 tokens, Drafted: 240 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  648. (APIServer pid=42) INFO 08-03 22:08:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 60.3%
  649. (APIServer pid=42) INFO: 172.19.0.4:45466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  650. (APIServer pid=42) INFO: 172.19.0.4:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  651. (APIServer pid=42) INFO 08-03 22:09:21 [loggers.py:273] Engine 000: Avg prompt throughput: 14.6 tokens/s, Avg generation throughput: 12.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 61.7%
  652. (APIServer pid=42) INFO 08-03 22:09:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.11, Accepted throughput: 0.13 tokens/s, Drafted throughput: 18.00 tokens/s, Accepted: 12 tokens, Drafted: 1620 tokens, Per-position acceptance rate: 0.093, 0.019, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.7%
  653. (APIServer pid=42) INFO: 172.19.0.4:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  654. (APIServer pid=42) INFO 08-03 22:09:31 [loggers.py:273] Engine 000: Avg prompt throughput: 1047.6 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 61.1%
  655. (APIServer pid=42) INFO 08-03 22:09:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 124.48 tokens/s, Accepted: 2 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.024, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  656. (APIServer pid=42) INFO 08-03 22:09:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 61.1%
  657. (APIServer pid=42) INFO 08-03 22:09:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 170.99 tokens/s, Accepted: 2 tokens, Drafted: 1710 tokens, Per-position acceptance rate: 0.018, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  658. (APIServer pid=42) INFO: 172.19.0.4:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  659. (APIServer pid=42) INFO 08-03 22:09:51 [loggers.py:273] Engine 000: Avg prompt throughput: 802.8 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 62.0%
  660. (APIServer pid=42) INFO 08-03 22:09:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 119.99 tokens/s, Accepted: 3 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.037, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  661. (APIServer pid=42) INFO 08-03 22:10:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.2%, Prefix cache hit rate: 62.0%
  662. (APIServer pid=42) INFO 08-03 22:10:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 167.98 tokens/s, Accepted: 3 tokens, Drafted: 1680 tokens, Per-position acceptance rate: 0.027, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  663. (APIServer pid=42) INFO: 172.19.0.4:45468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  664. (APIServer pid=42) INFO 08-03 22:10:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.2%, Prefix cache hit rate: 61.4%
  665. (APIServer pid=42) INFO 08-03 22:10:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 57.00 tokens/s, Accepted: 0 tokens, Drafted: 570 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  666. (APIServer pid=42) INFO 08-03 22:10:21 [loggers.py:273] Engine 000: Avg prompt throughput: 2249.4 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 61.4%
  667. (APIServer pid=42) INFO 08-03 22:10:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 0 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  668. (APIServer pid=42) INFO 08-03 22:10:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 61.4%
  669. (APIServer pid=42) INFO 08-03 22:10:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 148.49 tokens/s, Accepted: 0 tokens, Drafted: 1485 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  670. (APIServer pid=42) INFO 08-03 22:10:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 61.4%
  671. (APIServer pid=42) INFO 08-03 22:10:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 3.00 tokens/s, Accepted: 0 tokens, Drafted: 30 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  672. (APIServer pid=42) INFO 08-03 22:10:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 61.4%
  673. (APIServer pid=42) INFO: 172.19.0.4:56620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  674. (APIServer pid=42) INFO: 172.19.0.4:56630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  675. (APIServer pid=42) INFO 08-03 22:11:11 [loggers.py:273] Engine 000: Avg prompt throughput: 5.0 tokens/s, Avg generation throughput: 13.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 62.5%
  676. (APIServer pid=42) INFO 08-03 22:11:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.16, Accepted throughput: 0.60 tokens/s, Drafted throughput: 54.99 tokens/s, Accepted: 18 tokens, Drafted: 1650 tokens, Per-position acceptance rate: 0.091, 0.027, 0.018, 0.009, 0.009, 0.009, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.1%
  677. (APIServer pid=42) INFO: 172.19.0.4:56630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  678. (APIServer pid=42) INFO 08-03 22:11:21 [loggers.py:273] Engine 000: Avg prompt throughput: 1047.6 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 62.0%
  679. (APIServer pid=42) INFO 08-03 22:11:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.07, Accepted throughput: 0.60 tokens/s, Drafted throughput: 119.98 tokens/s, Accepted: 6 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.062, 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  680. (APIServer pid=42) INFO 08-03 22:11:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 62.0%
  681. (APIServer pid=42) INFO 08-03 22:11:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 167.97 tokens/s, Accepted: 2 tokens, Drafted: 1680 tokens, Per-position acceptance rate: 0.009, 0.009, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  682. (APIServer pid=42) INFO: 172.19.0.4:56630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  683. (APIServer pid=42) INFO 08-03 22:11:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 62.9%
  684. (APIServer pid=42) INFO 08-03 22:11:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 0 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  685. (APIServer pid=42) INFO 08-03 22:11:51 [loggers.py:273] Engine 000: Avg prompt throughput: 589.6 tokens/s, Avg generation throughput: 10.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 62.9%
  686. (APIServer pid=42) INFO 08-03 22:11:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.13, Accepted throughput: 1.20 tokens/s, Drafted throughput: 142.49 tokens/s, Accepted: 12 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.116, 0.011, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.8%
  687. (APIServer pid=42) INFO 08-03 22:12:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 62.9%
  688. (APIServer pid=42) INFO 08-03 22:12:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 167.99 tokens/s, Accepted: 0 tokens, Drafted: 1680 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  689. (APIServer pid=42) INFO: 172.19.0.4:56630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  690. (APIServer pid=42) INFO 08-03 22:12:11 [loggers.py:273] Engine 000: Avg prompt throughput: 570.8 tokens/s, Avg generation throughput: 9.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 64.2%
  691. (APIServer pid=42) INFO 08-03 22:12:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.08, Accepted throughput: 0.70 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 7 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.084, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.6%
  692. (APIServer pid=42) INFO 08-03 22:12:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 64.2%
  693. (APIServer pid=42) INFO 08-03 22:12:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 152.98 tokens/s, Accepted: 4 tokens, Drafted: 1530 tokens, Per-position acceptance rate: 0.039, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  694. (APIServer pid=42) INFO 08-03 22:12:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 64.2%
  695. (APIServer pid=42) INFO 08-03 22:12:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.60 tokens/s, Drafted throughput: 154.49 tokens/s, Accepted: 6 tokens, Drafted: 1545 tokens, Per-position acceptance rate: 0.049, 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  696. (APIServer pid=42) INFO 08-03 22:12:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 64.2%
  697. (APIServer pid=42) INFO 08-03 22:12:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 154.48 tokens/s, Accepted: 3 tokens, Drafted: 1545 tokens, Per-position acceptance rate: 0.029, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  698. (APIServer pid=42) INFO 08-03 22:12:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 64.2%
  699. (APIServer pid=42) INFO 08-03 22:12:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 122.87 tokens/s, Accepted: 4 tokens, Drafted: 1230 tokens, Per-position acceptance rate: 0.049, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  700. (APIServer pid=42) INFO: 172.19.0.4:47172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  701. (APIServer pid=42) INFO: 172.19.0.4:47184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  702. (APIServer pid=42) INFO 08-03 22:13:01 [loggers.py:273] Engine 000: Avg prompt throughput: 6.8 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 64.9%
  703. (APIServer pid=42) INFO 08-03 22:13:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 4 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.041, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  704. (APIServer pid=42) INFO 08-03 22:13:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.5%, Prefix cache hit rate: 64.9%
  705. (APIServer pid=42) INFO 08-03 22:13:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 153.00 tokens/s, Accepted: 0 tokens, Drafted: 1530 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  706. (APIServer pid=42) INFO: 172.19.0.4:56630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  707. (APIServer pid=42) INFO 08-03 22:13:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 65.8%
  708. (APIServer pid=42) INFO 08-03 22:13:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 126.00 tokens/s, Accepted: 0 tokens, Drafted: 1260 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  709. (APIServer pid=42) INFO 08-03 22:13:31 [loggers.py:273] Engine 000: Avg prompt throughput: 1019.7 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 65.8%
  710. (APIServer pid=42) INFO 08-03 22:13:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 94.49 tokens/s, Accepted: 1 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.016, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  711. (APIServer pid=42) INFO 08-03 22:13:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 65.8%
  712. (APIServer pid=42) INFO 08-03 22:13:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 136.49 tokens/s, Accepted: 3 tokens, Drafted: 1365 tokens, Per-position acceptance rate: 0.033, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  713. (APIServer pid=42) INFO 08-03 22:13:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  714. (APIServer pid=42) INFO 08-03 22:13:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 149.99 tokens/s, Accepted: 2 tokens, Drafted: 1500 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  715. (APIServer pid=42) INFO 08-03 22:14:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  716. (APIServer pid=42) INFO 08-03 22:14:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 151.49 tokens/s, Accepted: 1 tokens, Drafted: 1515 tokens, Per-position acceptance rate: 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  717. (APIServer pid=42) INFO 08-03 22:14:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  718. (APIServer pid=42) INFO 08-03 22:14:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 125.99 tokens/s, Accepted: 3 tokens, Drafted: 1260 tokens, Per-position acceptance rate: 0.036, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  719. (APIServer pid=42) INFO 08-03 22:14:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  720. (APIServer pid=42) INFO 08-03 22:14:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 136.48 tokens/s, Accepted: 4 tokens, Drafted: 1365 tokens, Per-position acceptance rate: 0.044, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  721. (APIServer pid=42) INFO 08-03 22:14:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  722. (APIServer pid=42) INFO 08-03 22:14:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 150.00 tokens/s, Accepted: 2 tokens, Drafted: 1500 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  723. (APIServer pid=42) INFO 08-03 22:14:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  724. (APIServer pid=42) INFO 08-03 22:14:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 151.49 tokens/s, Accepted: 1 tokens, Drafted: 1515 tokens, Per-position acceptance rate: 0.010, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  725. (APIServer pid=42) INFO 08-03 22:14:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  726. (APIServer pid=42) INFO 08-03 22:14:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 128.99 tokens/s, Accepted: 4 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.047, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  727. (APIServer pid=42) INFO 08-03 22:15:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  728. (APIServer pid=42) INFO 08-03 22:15:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 128.99 tokens/s, Accepted: 3 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.035, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  729. (APIServer pid=42) INFO 08-03 22:15:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  730. (APIServer pid=42) INFO 08-03 22:15:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 152.98 tokens/s, Accepted: 2 tokens, Drafted: 1530 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  731. (APIServer pid=42) INFO 08-03 22:15:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  732. (APIServer pid=42) INFO 08-03 22:15:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.50 tokens/s, Drafted throughput: 148.50 tokens/s, Accepted: 5 tokens, Drafted: 1485 tokens, Per-position acceptance rate: 0.051, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  733. (APIServer pid=42) INFO 08-03 22:15:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  734. (APIServer pid=42) INFO 08-03 22:15:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 128.99 tokens/s, Accepted: 3 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.035, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  735. (APIServer pid=42) INFO 08-03 22:15:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  736. (APIServer pid=42) INFO 08-03 22:15:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 129.00 tokens/s, Accepted: 2 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.023, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  737. (APIServer pid=42) INFO 08-03 22:15:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  738. (APIServer pid=42) INFO 08-03 22:15:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 148.49 tokens/s, Accepted: 2 tokens, Drafted: 1485 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  739. (APIServer pid=42) INFO 08-03 22:16:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  740. (APIServer pid=42) INFO 08-03 22:16:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 146.99 tokens/s, Accepted: 3 tokens, Drafted: 1470 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  741. (APIServer pid=42) INFO 08-03 22:16:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 65.8%
  742. (APIServer pid=42) INFO 08-03 22:16:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 4 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.047, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  743. (APIServer pid=42) INFO 08-03 22:16:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 65.8%
  744. (APIServer pid=42) INFO 08-03 22:16:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.12, Accepted throughput: 1.00 tokens/s, Drafted throughput: 126.00 tokens/s, Accepted: 10 tokens, Drafted: 1260 tokens, Per-position acceptance rate: 0.107, 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.8%
  745. (APIServer pid=42) INFO 08-03 22:16:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 65.8%
  746. (APIServer pid=42) INFO 08-03 22:16:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.33, Accepted throughput: 0.10 tokens/s, Drafted throughput: 4.50 tokens/s, Accepted: 1 tokens, Drafted: 45 tokens, Per-position acceptance rate: 0.333, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 2.2%
  747. (APIServer pid=42) INFO 08-03 22:16:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 65.8%
  748. (APIServer pid=42) INFO: 172.19.0.4:34688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  749. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  750. (APIServer pid=42) INFO 08-03 22:16:51 [loggers.py:273] Engine 000: Avg prompt throughput: 2.3 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.0%, Prefix cache hit rate: 66.5%
  751. (APIServer pid=42) INFO 08-03 22:16:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.20, Accepted throughput: 0.05 tokens/s, Drafted throughput: 3.75 tokens/s, Accepted: 1 tokens, Drafted: 75 tokens, Per-position acceptance rate: 0.200, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.3%
  752. (APIServer pid=42) INFO 08-03 22:17:01 [loggers.py:273] Engine 000: Avg prompt throughput: 2.7 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.9%, Prefix cache hit rate: 66.5%
  753. (APIServer pid=42) INFO 08-03 22:17:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.16, Accepted throughput: 1.90 tokens/s, Drafted throughput: 175.49 tokens/s, Accepted: 19 tokens, Drafted: 1755 tokens, Per-position acceptance rate: 0.137, 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 1.1%
  754. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  755. (APIServer pid=42) INFO 08-03 22:17:11 [loggers.py:273] Engine 000: Avg prompt throughput: 1047.5 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 66.0%
  756. (APIServer pid=42) INFO 08-03 22:17:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 122.99 tokens/s, Accepted: 3 tokens, Drafted: 1230 tokens, Per-position acceptance rate: 0.037, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  757. (APIServer pid=42) INFO 08-03 22:17:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 66.0%
  758. (APIServer pid=42) INFO 08-03 22:17:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 172.48 tokens/s, Accepted: 3 tokens, Drafted: 1725 tokens, Per-position acceptance rate: 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  759. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  760. (APIServer pid=42) INFO 08-03 22:17:31 [loggers.py:273] Engine 000: Avg prompt throughput: 588.3 tokens/s, Avg generation throughput: 5.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.0%, Prefix cache hit rate: 66.5%
  761. (APIServer pid=42) INFO 08-03 22:17:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.08, Accepted throughput: 0.40 tokens/s, Drafted throughput: 71.99 tokens/s, Accepted: 4 tokens, Drafted: 720 tokens, Per-position acceptance rate: 0.083, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.6%
  762. (APIServer pid=42) INFO 08-03 22:17:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.1%, Prefix cache hit rate: 66.5%
  763. (APIServer pid=42) INFO 08-03 22:17:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 152.99 tokens/s, Accepted: 2 tokens, Drafted: 1530 tokens, Per-position acceptance rate: 0.020, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  764. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  765. (APIServer pid=42) INFO 08-03 22:17:51 [loggers.py:273] Engine 000: Avg prompt throughput: 1363.3 tokens/s, Avg generation throughput: 5.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 66.6%
  766. (APIServer pid=42) INFO 08-03 22:17:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.20 tokens/s, Drafted throughput: 82.49 tokens/s, Accepted: 2 tokens, Drafted: 825 tokens, Per-position acceptance rate: 0.036, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  767. (APIServer pid=42) INFO 08-03 22:18:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 66.6%
  768. (APIServer pid=42) INFO 08-03 22:18:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.60 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 6 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.062, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  769. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  770. (APIServer pid=42) INFO 08-03 22:18:11 [loggers.py:273] Engine 000: Avg prompt throughput: 275.4 tokens/s, Avg generation throughput: 3.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 68.2%
  771. (APIServer pid=42) INFO 08-03 22:18:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.10 tokens/s, Drafted throughput: 43.50 tokens/s, Accepted: 1 tokens, Drafted: 435 tokens, Per-position acceptance rate: 0.034, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  772. (APIServer pid=42) INFO 08-03 22:18:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.2%
  773. (APIServer pid=42) INFO 08-03 22:18:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 100.49 tokens/s, Accepted: 1 tokens, Drafted: 1005 tokens, Per-position acceptance rate: 0.015, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  774. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  775. (APIServer pid=42) INFO 08-03 22:18:31 [loggers.py:273] Engine 000: Avg prompt throughput: 83.0 tokens/s, Avg generation throughput: 9.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  776. (APIServer pid=42) INFO 08-03 22:18:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 133.48 tokens/s, Accepted: 2 tokens, Drafted: 1335 tokens, Per-position acceptance rate: 0.022, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  777. (APIServer pid=42) INFO 08-03 22:18:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  778. (APIServer pid=42) INFO 08-03 22:18:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 145.48 tokens/s, Accepted: 2 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.021, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  779. (APIServer pid=42) INFO 08-03 22:18:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  780. (APIServer pid=42) INFO 08-03 22:18:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 118.50 tokens/s, Accepted: 2 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.025, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  781. (APIServer pid=42) INFO 08-03 22:19:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  782. (APIServer pid=42) INFO 08-03 22:19:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.40 tokens/s, Drafted throughput: 133.49 tokens/s, Accepted: 4 tokens, Drafted: 1335 tokens, Per-position acceptance rate: 0.045, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  783. (APIServer pid=42) INFO 08-03 22:19:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  784. (APIServer pid=42) INFO 08-03 22:19:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 145.49 tokens/s, Accepted: 2 tokens, Drafted: 1455 tokens, Per-position acceptance rate: 0.021, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  785. (APIServer pid=42) INFO 08-03 22:19:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  786. (APIServer pid=42) INFO 08-03 22:19:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 140.99 tokens/s, Accepted: 3 tokens, Drafted: 1410 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  787. (APIServer pid=42) INFO 08-03 22:19:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  788. (APIServer pid=42) INFO 08-03 22:19:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 116.99 tokens/s, Accepted: 2 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  789. (APIServer pid=42) INFO 08-03 22:19:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  790. (APIServer pid=42) INFO 08-03 22:19:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 1 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  791. (APIServer pid=42) INFO 08-03 22:19:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  792. (APIServer pid=42) INFO 08-03 22:19:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 142.49 tokens/s, Accepted: 3 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  793. (APIServer pid=42) INFO 08-03 22:20:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  794. (APIServer pid=42) INFO 08-03 22:20:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 140.99 tokens/s, Accepted: 2 tokens, Drafted: 1410 tokens, Per-position acceptance rate: 0.021, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  795. (APIServer pid=42) INFO 08-03 22:20:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.9%
  796. (APIServer pid=42) INFO 08-03 22:20:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 119.99 tokens/s, Accepted: 4 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.050, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  797. (APIServer pid=42) INFO 08-03 22:20:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  798. (APIServer pid=42) INFO 08-03 22:20:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 121.49 tokens/s, Accepted: 2 tokens, Drafted: 1215 tokens, Per-position acceptance rate: 0.025, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  799. (APIServer pid=42) INFO 08-03 22:20:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  800. (APIServer pid=42) INFO 08-03 22:20:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 142.49 tokens/s, Accepted: 3 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  801. (APIServer pid=42) INFO 08-03 22:20:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  802. (APIServer pid=42) INFO 08-03 22:20:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 142.48 tokens/s, Accepted: 3 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  803. (APIServer pid=42) INFO 08-03 22:20:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  804. (APIServer pid=42) INFO 08-03 22:20:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 124.49 tokens/s, Accepted: 2 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.024, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  805. (APIServer pid=42) INFO 08-03 22:21:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  806. (APIServer pid=42) INFO 08-03 22:21:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 118.48 tokens/s, Accepted: 2 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.025, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  807. (APIServer pid=42) INFO 08-03 22:21:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  808. (APIServer pid=42) INFO 08-03 22:21:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 139.48 tokens/s, Accepted: 3 tokens, Drafted: 1395 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  809. (APIServer pid=42) INFO 08-03 22:21:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  810. (APIServer pid=42) INFO 08-03 22:21:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 140.99 tokens/s, Accepted: 1 tokens, Drafted: 1410 tokens, Per-position acceptance rate: 0.011, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  811. (APIServer pid=42) INFO 08-03 22:21:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  812. (APIServer pid=42) INFO 08-03 22:21:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 2 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.024, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  813. (APIServer pid=42) INFO 08-03 22:21:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  814. (APIServer pid=42) INFO 08-03 22:21:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 113.99 tokens/s, Accepted: 2 tokens, Drafted: 1140 tokens, Per-position acceptance rate: 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  815. (APIServer pid=42) INFO 08-03 22:21:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  816. (APIServer pid=42) INFO 08-03 22:21:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.07, Accepted throughput: 0.70 tokens/s, Drafted throughput: 140.99 tokens/s, Accepted: 7 tokens, Drafted: 1410 tokens, Per-position acceptance rate: 0.074, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  817. (APIServer pid=42) INFO 08-03 22:22:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 69.9%
  818. (APIServer pid=42) INFO 08-03 22:22:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 143.99 tokens/s, Accepted: 2 tokens, Drafted: 1440 tokens, Per-position acceptance rate: 0.021, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  819. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  820. (APIServer pid=42) INFO 08-03 22:22:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.9%
  821. (APIServer pid=42) INFO 08-03 22:22:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 107.99 tokens/s, Accepted: 0 tokens, Drafted: 1080 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  822. (APIServer pid=42) INFO 08-03 22:22:21 [loggers.py:273] Engine 000: Avg prompt throughput: 343.4 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 71.3%
  823. (APIServer pid=42) INFO 08-03 22:22:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 95.99 tokens/s, Accepted: 2 tokens, Drafted: 960 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  824. (APIServer pid=42) INFO 08-03 22:22:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 71.3%
  825. (APIServer pid=42) INFO 08-03 22:22:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.07, Accepted throughput: 0.70 tokens/s, Drafted throughput: 142.48 tokens/s, Accepted: 7 tokens, Drafted: 1425 tokens, Per-position acceptance rate: 0.074, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  826. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  827. (APIServer pid=42) INFO 08-03 22:22:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 72.3%
  828. (APIServer pid=42) INFO 08-03 22:22:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 122.99 tokens/s, Accepted: 0 tokens, Drafted: 1230 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  829. (APIServer pid=42) INFO 08-03 22:22:51 [loggers.py:273] Engine 000: Avg prompt throughput: 829.4 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 72.3%
  830. (APIServer pid=42) INFO 08-03 22:22:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 110.99 tokens/s, Accepted: 0 tokens, Drafted: 1110 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  831. (APIServer pid=42) INFO 08-03 22:23:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 72.3%
  832. (APIServer pid=42) INFO 08-03 22:23:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 100.49 tokens/s, Accepted: 0 tokens, Drafted: 1005 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  833. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  834. (APIServer pid=42) INFO 08-03 22:23:11 [loggers.py:273] Engine 000: Avg prompt throughput: 762.8 tokens/s, Avg generation throughput: 6.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 73.4%
  835. (APIServer pid=42) INFO 08-03 22:23:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 88.49 tokens/s, Accepted: 0 tokens, Drafted: 885 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  836. (APIServer pid=42) INFO 08-03 22:23:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 73.4%
  837. (APIServer pid=42) INFO 08-03 22:23:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 0 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  838. (APIServer pid=42) INFO 08-03 22:23:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 73.4%
  839. (APIServer pid=42) INFO 08-03 22:23:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 128.98 tokens/s, Accepted: 0 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  840. (APIServer pid=42) INFO 08-03 22:23:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 73.4%
  841. (APIServer pid=42) INFO 08-03 22:23:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 94.49 tokens/s, Accepted: 0 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  842. (APIServer pid=42) INFO 08-03 22:23:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 73.4%
  843. (APIServer pid=42) INFO 08-03 22:23:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 132.00 tokens/s, Accepted: 2 tokens, Drafted: 1320 tokens, Per-position acceptance rate: 0.023, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  844. (APIServer pid=42) INFO 08-03 22:24:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 73.4%
  845. (APIServer pid=42) INFO 08-03 22:24:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 135.00 tokens/s, Accepted: 0 tokens, Drafted: 1350 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  846. (APIServer pid=42) INFO 08-03 22:24:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.8%, Prefix cache hit rate: 73.4%
  847. (APIServer pid=42) INFO 08-03 22:24:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 1 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  848. (APIServer pid=42) INFO 08-03 22:24:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  849. (APIServer pid=42) INFO 08-03 22:24:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 92.99 tokens/s, Accepted: 0 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  850. (APIServer pid=42) INFO 08-03 22:24:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  851. (APIServer pid=42) INFO 08-03 22:24:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 125.99 tokens/s, Accepted: 1 tokens, Drafted: 1260 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  852. (APIServer pid=42) INFO 08-03 22:24:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  853. (APIServer pid=42) INFO 08-03 22:24:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 1 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  854. (APIServer pid=42) INFO 08-03 22:24:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  855. (APIServer pid=42) INFO 08-03 22:24:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.30 tokens/s, Drafted throughput: 130.49 tokens/s, Accepted: 3 tokens, Drafted: 1305 tokens, Per-position acceptance rate: 0.023, 0.011, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  856. (APIServer pid=42) INFO 08-03 22:25:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  857. (APIServer pid=42) INFO 08-03 22:25:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 96.00 tokens/s, Accepted: 0 tokens, Drafted: 960 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  858. (APIServer pid=42) INFO 08-03 22:25:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  859. (APIServer pid=42) INFO 08-03 22:25:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 1 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  860. (APIServer pid=42) INFO 08-03 22:25:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  861. (APIServer pid=42) INFO 08-03 22:25:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 0 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  862. (APIServer pid=42) INFO 08-03 22:25:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  863. (APIServer pid=42) INFO 08-03 22:25:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 129.00 tokens/s, Accepted: 0 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  864. (APIServer pid=42) INFO 08-03 22:25:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  865. (APIServer pid=42) INFO 08-03 22:25:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 94.49 tokens/s, Accepted: 0 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  866. (APIServer pid=42) INFO 08-03 22:25:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  867. (APIServer pid=42) INFO 08-03 22:25:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 131.99 tokens/s, Accepted: 0 tokens, Drafted: 1320 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  868. (APIServer pid=42) INFO 08-03 22:26:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  869. (APIServer pid=42) INFO 08-03 22:26:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 130.50 tokens/s, Accepted: 0 tokens, Drafted: 1305 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  870. (APIServer pid=42) INFO 08-03 22:26:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  871. (APIServer pid=42) INFO 08-03 22:26:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 1 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  872. (APIServer pid=42) INFO 08-03 22:26:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  873. (APIServer pid=42) INFO 08-03 22:26:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 0 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  874. (APIServer pid=42) INFO 08-03 22:26:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  875. (APIServer pid=42) INFO 08-03 22:26:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 0 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  876. (APIServer pid=42) INFO 08-03 22:26:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  877. (APIServer pid=42) INFO 08-03 22:26:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 128.98 tokens/s, Accepted: 1 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  878. (APIServer pid=42) INFO 08-03 22:26:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  879. (APIServer pid=42) INFO 08-03 22:26:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 128.99 tokens/s, Accepted: 0 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  880. (APIServer pid=42) INFO 08-03 22:27:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  881. (APIServer pid=42) INFO 08-03 22:27:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 94.49 tokens/s, Accepted: 2 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.032, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  882. (APIServer pid=42) INFO 08-03 22:27:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 73.4%
  883. (APIServer pid=42) INFO 08-03 22:27:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 124.49 tokens/s, Accepted: 3 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.036, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  884. (APIServer pid=42) INFO 08-03 22:27:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  885. (APIServer pid=42) INFO 08-03 22:27:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 128.99 tokens/s, Accepted: 0 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  886. (APIServer pid=42) INFO 08-03 22:27:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  887. (APIServer pid=42) INFO 08-03 22:27:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 129.00 tokens/s, Accepted: 4 tokens, Drafted: 1290 tokens, Per-position acceptance rate: 0.047, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  888. (APIServer pid=42) INFO 08-03 22:27:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  889. (APIServer pid=42) INFO 08-03 22:27:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 97.49 tokens/s, Accepted: 2 tokens, Drafted: 975 tokens, Per-position acceptance rate: 0.031, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  890. (APIServer pid=42) INFO 08-03 22:27:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  891. (APIServer pid=42) INFO 08-03 22:27:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 124.48 tokens/s, Accepted: 3 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.036, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  892. (APIServer pid=42) INFO 08-03 22:28:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  893. (APIServer pid=42) INFO 08-03 22:28:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 1 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  894. (APIServer pid=42) INFO 08-03 22:28:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  895. (APIServer pid=42) INFO 08-03 22:28:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 125.99 tokens/s, Accepted: 1 tokens, Drafted: 1260 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  896. (APIServer pid=42) INFO 08-03 22:28:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  897. (APIServer pid=42) INFO 08-03 22:28:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 95.99 tokens/s, Accepted: 1 tokens, Drafted: 960 tokens, Per-position acceptance rate: 0.016, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  898. (APIServer pid=42) INFO 08-03 22:28:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  899. (APIServer pid=42) INFO 08-03 22:28:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 1 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  900. (APIServer pid=42) INFO 08-03 22:28:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  901. (APIServer pid=42) INFO 08-03 22:28:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 126.00 tokens/s, Accepted: 0 tokens, Drafted: 1260 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  902. (APIServer pid=42) INFO 08-03 22:28:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  903. (APIServer pid=42) INFO 08-03 22:28:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 127.50 tokens/s, Accepted: 0 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  904. (APIServer pid=42) INFO 08-03 22:29:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  905. (APIServer pid=42) INFO 08-03 22:29:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 100.49 tokens/s, Accepted: 1 tokens, Drafted: 1005 tokens, Per-position acceptance rate: 0.015, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  906. (APIServer pid=42) INFO 08-03 22:29:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  907. (APIServer pid=42) INFO 08-03 22:29:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 121.49 tokens/s, Accepted: 3 tokens, Drafted: 1215 tokens, Per-position acceptance rate: 0.037, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  908. (APIServer pid=42) INFO 08-03 22:29:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  909. (APIServer pid=42) INFO 08-03 22:29:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 121.50 tokens/s, Accepted: 1 tokens, Drafted: 1215 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  910. (APIServer pid=42) INFO 08-03 22:29:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  911. (APIServer pid=42) INFO 08-03 22:29:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.20 tokens/s, Drafted throughput: 124.50 tokens/s, Accepted: 2 tokens, Drafted: 1245 tokens, Per-position acceptance rate: 0.024, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  912. (APIServer pid=42) INFO 08-03 22:29:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  913. (APIServer pid=42) INFO 08-03 22:29:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 98.99 tokens/s, Accepted: 1 tokens, Drafted: 990 tokens, Per-position acceptance rate: 0.015, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  914. (APIServer pid=42) INFO 08-03 22:29:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  915. (APIServer pid=42) INFO 08-03 22:29:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 2 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.025, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  916. (APIServer pid=42) INFO 08-03 22:30:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 73.4%
  917. (APIServer pid=42) INFO 08-03 22:30:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 127.50 tokens/s, Accepted: 1 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  918. (APIServer pid=42) INFO 08-03 22:30:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  919. (APIServer pid=42) INFO 08-03 22:30:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 125.99 tokens/s, Accepted: 3 tokens, Drafted: 1260 tokens, Per-position acceptance rate: 0.036, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  920. (APIServer pid=42) INFO 08-03 22:30:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  921. (APIServer pid=42) INFO 08-03 22:30:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 100.49 tokens/s, Accepted: 0 tokens, Drafted: 1005 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  922. (APIServer pid=42) INFO 08-03 22:30:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  923. (APIServer pid=42) INFO 08-03 22:30:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.50 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  924. (APIServer pid=42) INFO 08-03 22:30:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  925. (APIServer pid=42) INFO 08-03 22:30:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 127.49 tokens/s, Accepted: 1 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.012, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  926. (APIServer pid=42) INFO 08-03 22:30:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  927. (APIServer pid=42) INFO 08-03 22:30:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 127.48 tokens/s, Accepted: 0 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  928. (APIServer pid=42) INFO 08-03 22:31:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  929. (APIServer pid=42) INFO 08-03 22:31:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 102.00 tokens/s, Accepted: 1 tokens, Drafted: 1020 tokens, Per-position acceptance rate: 0.015, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  930. (APIServer pid=42) INFO 08-03 22:31:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  931. (APIServer pid=42) INFO 08-03 22:31:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.07, Accepted throughput: 0.60 tokens/s, Drafted throughput: 122.99 tokens/s, Accepted: 6 tokens, Drafted: 1230 tokens, Per-position acceptance rate: 0.073, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  932. (APIServer pid=42) INFO 08-03 22:31:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 73.4%
  933. (APIServer pid=42) INFO 08-03 22:31:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 127.50 tokens/s, Accepted: 0 tokens, Drafted: 1275 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  934. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  935. (APIServer pid=42) INFO 08-03 22:31:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.4%
  936. (APIServer pid=42) INFO 08-03 22:31:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 104.99 tokens/s, Accepted: 0 tokens, Drafted: 1050 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  937. (APIServer pid=42) INFO 08-03 22:31:41 [loggers.py:273] Engine 000: Avg prompt throughput: 1037.7 tokens/s, Avg generation throughput: 4.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 74.5%
  938. (APIServer pid=42) INFO 08-03 22:31:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 61.50 tokens/s, Accepted: 0 tokens, Drafted: 615 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  939. (APIServer pid=42) INFO 08-03 22:31:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 74.5%
  940. (APIServer pid=42) INFO 08-03 22:31:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  941. (APIServer pid=42) INFO: 172.19.0.4:34694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  942. (APIServer pid=42) INFO 08-03 22:32:01 [loggers.py:273] Engine 000: Avg prompt throughput: 659.7 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  943. (APIServer pid=42) INFO 08-03 22:32:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 82.49 tokens/s, Accepted: 0 tokens, Drafted: 825 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  944. (APIServer pid=42) INFO 08-03 22:32:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  945. (APIServer pid=42) INFO 08-03 22:32:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 120.00 tokens/s, Accepted: 0 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  946. (APIServer pid=42) INFO 08-03 22:32:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  947. (APIServer pid=42) INFO 08-03 22:32:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 92.99 tokens/s, Accepted: 0 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  948. (APIServer pid=42) INFO 08-03 22:32:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  949. (APIServer pid=42) INFO 08-03 22:32:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 117.00 tokens/s, Accepted: 0 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  950. (APIServer pid=42) INFO 08-03 22:32:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  951. (APIServer pid=42) INFO 08-03 22:32:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.50 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  952. (APIServer pid=42) INFO 08-03 22:32:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  953. (APIServer pid=42) INFO 08-03 22:32:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 120.00 tokens/s, Accepted: 0 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  954. (APIServer pid=42) INFO 08-03 22:33:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  955. (APIServer pid=42) INFO 08-03 22:33:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 91.49 tokens/s, Accepted: 1 tokens, Drafted: 915 tokens, Per-position acceptance rate: 0.016, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  956. (APIServer pid=42) INFO 08-03 22:33:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  957. (APIServer pid=42) INFO 08-03 22:33:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 113.99 tokens/s, Accepted: 0 tokens, Drafted: 1140 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  958. (APIServer pid=42) INFO 08-03 22:33:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  959. (APIServer pid=42) INFO 08-03 22:33:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  960. (APIServer pid=42) INFO 08-03 22:33:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 75.9%
  961. (APIServer pid=42) INFO 08-03 22:33:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.48 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  962. (APIServer pid=42) INFO 08-03 22:33:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  963. (APIServer pid=42) INFO 08-03 22:33:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 89.99 tokens/s, Accepted: 0 tokens, Drafted: 900 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  964. (APIServer pid=42) INFO 08-03 22:33:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  965. (APIServer pid=42) INFO 08-03 22:33:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 106.49 tokens/s, Accepted: 0 tokens, Drafted: 1065 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  966. (APIServer pid=42) INFO 08-03 22:34:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  967. (APIServer pid=42) INFO 08-03 22:34:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 3 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.038, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  968. (APIServer pid=42) INFO 08-03 22:34:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  969. (APIServer pid=42) INFO 08-03 22:34:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 2 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.025, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  970. (APIServer pid=42) INFO 08-03 22:34:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  971. (APIServer pid=42) INFO 08-03 22:34:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 87.00 tokens/s, Accepted: 1 tokens, Drafted: 870 tokens, Per-position acceptance rate: 0.017, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  972. (APIServer pid=42) INFO 08-03 22:34:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  973. (APIServer pid=42) INFO 08-03 22:34:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 120.00 tokens/s, Accepted: 0 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  974. (APIServer pid=42) INFO 08-03 22:34:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  975. (APIServer pid=42) INFO 08-03 22:34:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  976. (APIServer pid=42) INFO 08-03 22:34:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  977. (APIServer pid=42) INFO 08-03 22:34:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 118.51 tokens/s, Accepted: 1 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  978. (APIServer pid=42) INFO 08-03 22:35:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  979. (APIServer pid=42) INFO 08-03 22:35:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 89.99 tokens/s, Accepted: 0 tokens, Drafted: 900 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  980. (APIServer pid=42) INFO 08-03 22:35:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  981. (APIServer pid=42) INFO 08-03 22:35:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  982. (APIServer pid=42) INFO 08-03 22:35:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  983. (APIServer pid=42) INFO 08-03 22:35:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 118.50 tokens/s, Accepted: 4 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.051, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  984. (APIServer pid=42) INFO 08-03 22:35:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  985. (APIServer pid=42) INFO 08-03 22:35:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 113.99 tokens/s, Accepted: 3 tokens, Drafted: 1140 tokens, Per-position acceptance rate: 0.039, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  986. (APIServer pid=42) INFO 08-03 22:35:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  987. (APIServer pid=42) INFO 08-03 22:35:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 89.99 tokens/s, Accepted: 2 tokens, Drafted: 900 tokens, Per-position acceptance rate: 0.033, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  988. (APIServer pid=42) INFO 08-03 22:35:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  989. (APIServer pid=42) INFO 08-03 22:35:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 117.00 tokens/s, Accepted: 2 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  990. (APIServer pid=42) INFO 08-03 22:36:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  991. (APIServer pid=42) INFO 08-03 22:36:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 115.49 tokens/s, Accepted: 0 tokens, Drafted: 1155 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  992. (APIServer pid=42) INFO 08-03 22:36:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  993. (APIServer pid=42) INFO 08-03 22:36:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 115.50 tokens/s, Accepted: 0 tokens, Drafted: 1155 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  994. (APIServer pid=42) INFO 08-03 22:36:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  995. (APIServer pid=42) INFO 08-03 22:36:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 88.50 tokens/s, Accepted: 0 tokens, Drafted: 885 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  996. (APIServer pid=42) INFO 08-03 22:36:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  997. (APIServer pid=42) INFO 08-03 22:36:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 118.48 tokens/s, Accepted: 1 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  998. (APIServer pid=42) INFO 08-03 22:36:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 75.9%
  999. (APIServer pid=42) INFO 08-03 22:36:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 113.99 tokens/s, Accepted: 0 tokens, Drafted: 1140 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1000. (APIServer pid=42) INFO 08-03 22:36:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1001. (APIServer pid=42) INFO 08-03 22:36:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 119.99 tokens/s, Accepted: 0 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1002. (APIServer pid=42) INFO 08-03 22:37:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1003. (APIServer pid=42) INFO 08-03 22:37:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 90.00 tokens/s, Accepted: 1 tokens, Drafted: 900 tokens, Per-position acceptance rate: 0.017, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1004. (APIServer pid=42) INFO 08-03 22:37:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1005. (APIServer pid=42) INFO 08-03 22:37:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 114.00 tokens/s, Accepted: 0 tokens, Drafted: 1140 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1006. (APIServer pid=42) INFO 08-03 22:37:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1007. (APIServer pid=42) INFO 08-03 22:37:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 116.99 tokens/s, Accepted: 0 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1008. (APIServer pid=42) INFO 08-03 22:37:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1009. (APIServer pid=42) INFO 08-03 22:37:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 115.49 tokens/s, Accepted: 0 tokens, Drafted: 1155 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1010. (APIServer pid=42) INFO 08-03 22:37:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1011. (APIServer pid=42) INFO 08-03 22:37:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 87.00 tokens/s, Accepted: 0 tokens, Drafted: 870 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1012. (APIServer pid=42) INFO 08-03 22:37:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1013. (APIServer pid=42) INFO 08-03 22:37:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 118.48 tokens/s, Accepted: 1 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1014. (APIServer pid=42) INFO 08-03 22:38:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1015. (APIServer pid=42) INFO 08-03 22:38:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.08, Accepted throughput: 0.60 tokens/s, Drafted throughput: 118.50 tokens/s, Accepted: 6 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.076, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  1016. (APIServer pid=42) INFO 08-03 22:38:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1017. (APIServer pid=42) INFO 08-03 22:38:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.08, Accepted throughput: 0.60 tokens/s, Drafted throughput: 114.00 tokens/s, Accepted: 6 tokens, Drafted: 1140 tokens, Per-position acceptance rate: 0.079, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.5%
  1018. (APIServer pid=42) INFO 08-03 22:38:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1019. (APIServer pid=42) INFO 08-03 22:38:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 90.00 tokens/s, Accepted: 0 tokens, Drafted: 900 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1020. (APIServer pid=42) INFO 08-03 22:38:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1021. (APIServer pid=42) INFO 08-03 22:38:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 121.49 tokens/s, Accepted: 3 tokens, Drafted: 1215 tokens, Per-position acceptance rate: 0.037, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  1022. (APIServer pid=42) INFO 08-03 22:38:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1023. (APIServer pid=42) INFO 08-03 22:38:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 119.99 tokens/s, Accepted: 4 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.050, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  1024. (APIServer pid=42) INFO 08-03 22:38:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1025. (APIServer pid=42) INFO 08-03 22:38:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 110.99 tokens/s, Accepted: 2 tokens, Drafted: 1110 tokens, Per-position acceptance rate: 0.027, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  1026. (APIServer pid=42) INFO 08-03 22:39:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1027. (APIServer pid=42) INFO 08-03 22:39:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 94.48 tokens/s, Accepted: 1 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.016, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1028. (APIServer pid=42) INFO 08-03 22:39:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1029. (APIServer pid=42) INFO 08-03 22:39:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 115.49 tokens/s, Accepted: 0 tokens, Drafted: 1155 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1030. (APIServer pid=42) INFO 08-03 22:39:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1031. (APIServer pid=42) INFO 08-03 22:39:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 116.99 tokens/s, Accepted: 1 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1032. (APIServer pid=42) INFO 08-03 22:39:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1033. (APIServer pid=42) INFO 08-03 22:39:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 105.00 tokens/s, Accepted: 0 tokens, Drafted: 1050 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1034. (APIServer pid=42) INFO 08-03 22:39:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1035. (APIServer pid=42) INFO 08-03 22:39:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 96.00 tokens/s, Accepted: 0 tokens, Drafted: 960 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1036. (APIServer pid=42) INFO 08-03 22:39:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1037. (APIServer pid=42) INFO 08-03 22:39:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 119.98 tokens/s, Accepted: 1 tokens, Drafted: 1200 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1038. (APIServer pid=42) INFO 08-03 22:40:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 75.9%
  1039. (APIServer pid=42) INFO 08-03 22:40:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 112.49 tokens/s, Accepted: 1 tokens, Drafted: 1125 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1040. (APIServer pid=42) INFO 08-03 22:40:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1041. (APIServer pid=42) INFO 08-03 22:40:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 106.49 tokens/s, Accepted: 0 tokens, Drafted: 1065 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1042. (APIServer pid=42) INFO 08-03 22:40:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1043. (APIServer pid=42) INFO 08-03 22:40:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 95.99 tokens/s, Accepted: 0 tokens, Drafted: 960 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1044. (APIServer pid=42) INFO 08-03 22:40:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1045. (APIServer pid=42) INFO 08-03 22:40:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 112.49 tokens/s, Accepted: 0 tokens, Drafted: 1125 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1046. (APIServer pid=42) INFO 08-03 22:40:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1047. (APIServer pid=42) INFO 08-03 22:40:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 115.49 tokens/s, Accepted: 0 tokens, Drafted: 1155 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1048. (APIServer pid=42) INFO 08-03 22:40:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1049. (APIServer pid=42) INFO 08-03 22:40:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 105.00 tokens/s, Accepted: 2 tokens, Drafted: 1050 tokens, Per-position acceptance rate: 0.029, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  1050. (APIServer pid=42) INFO 08-03 22:41:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1051. (APIServer pid=42) INFO 08-03 22:41:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 100.49 tokens/s, Accepted: 3 tokens, Drafted: 1005 tokens, Per-position acceptance rate: 0.045, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  1052. (APIServer pid=42) INFO 08-03 22:41:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1053. (APIServer pid=42) INFO 08-03 22:41:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 112.50 tokens/s, Accepted: 0 tokens, Drafted: 1125 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1054. (APIServer pid=42) INFO 08-03 22:41:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1055. (APIServer pid=42) INFO 08-03 22:41:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 118.48 tokens/s, Accepted: 0 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1056. (APIServer pid=42) INFO 08-03 22:41:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1057. (APIServer pid=42) INFO 08-03 22:41:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 102.00 tokens/s, Accepted: 0 tokens, Drafted: 1020 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1058. (APIServer pid=42) INFO 08-03 22:41:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1059. (APIServer pid=42) INFO 08-03 22:41:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 98.99 tokens/s, Accepted: 2 tokens, Drafted: 990 tokens, Per-position acceptance rate: 0.030, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  1060. (APIServer pid=42) INFO 08-03 22:41:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1061. (APIServer pid=42) INFO 08-03 22:41:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.06, Accepted throughput: 0.50 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 5 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.063, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.4%
  1062. (APIServer pid=42) INFO 08-03 22:42:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1063. (APIServer pid=42) INFO 08-03 22:42:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.04, Accepted throughput: 0.30 tokens/s, Drafted throughput: 117.00 tokens/s, Accepted: 3 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.038, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  1064. (APIServer pid=42) INFO 08-03 22:42:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1065. (APIServer pid=42) INFO 08-03 22:42:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 99.00 tokens/s, Accepted: 1 tokens, Drafted: 990 tokens, Per-position acceptance rate: 0.015, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1066. (APIServer pid=42) INFO 08-03 22:42:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1067. (APIServer pid=42) INFO 08-03 22:42:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 103.49 tokens/s, Accepted: 0 tokens, Drafted: 1035 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1068. (APIServer pid=42) INFO 08-03 22:42:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1069. (APIServer pid=42) INFO 08-03 22:42:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 116.98 tokens/s, Accepted: 2 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  1070. (APIServer pid=42) INFO 08-03 22:42:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1071. (APIServer pid=42) INFO 08-03 22:42:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.03, Accepted throughput: 0.20 tokens/s, Drafted throughput: 117.00 tokens/s, Accepted: 2 tokens, Drafted: 1170 tokens, Per-position acceptance rate: 0.026, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.2%
  1072. (APIServer pid=42) INFO 08-03 22:42:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1073. (APIServer pid=42) INFO 08-03 22:42:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 94.49 tokens/s, Accepted: 0 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1074. (APIServer pid=42) INFO 08-03 22:43:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1075. (APIServer pid=42) INFO 08-03 22:43:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 106.50 tokens/s, Accepted: 0 tokens, Drafted: 1065 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1076. (APIServer pid=42) INFO 08-03 22:43:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1077. (APIServer pid=42) INFO 08-03 22:43:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 115.50 tokens/s, Accepted: 0 tokens, Drafted: 1155 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1078. (APIServer pid=42) INFO 08-03 22:43:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 75.9%
  1079. (APIServer pid=42) INFO 08-03 22:43:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 111.00 tokens/s, Accepted: 0 tokens, Drafted: 1110 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1080. (APIServer pid=42) INFO 08-03 22:43:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 75.9%
  1081. (APIServer pid=42) INFO 08-03 22:43:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 95.99 tokens/s, Accepted: 0 tokens, Drafted: 960 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1082. (APIServer pid=42) INFO 08-03 22:43:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 75.9%
  1083. (APIServer pid=42) INFO 08-03 22:43:41 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 106.49 tokens/s, Accepted: 0 tokens, Drafted: 1065 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1084. (APIServer pid=42) INFO 08-03 22:43:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 75.9%
  1085. (APIServer pid=42) INFO 08-03 22:43:51 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 111.00 tokens/s, Accepted: 0 tokens, Drafted: 1110 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1086. (APIServer pid=42) INFO 08-03 22:44:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 75.9%
  1087. (APIServer pid=42) INFO 08-03 22:44:01 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.05, Accepted throughput: 0.40 tokens/s, Drafted throughput: 118.49 tokens/s, Accepted: 4 tokens, Drafted: 1185 tokens, Per-position acceptance rate: 0.051, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.3%
  1088. (APIServer pid=42) INFO 08-03 22:44:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 6.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 75.9%
  1089. (APIServer pid=42) INFO 08-03 22:44:11 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.02, Accepted throughput: 0.10 tokens/s, Drafted throughput: 94.50 tokens/s, Accepted: 1 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.016, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1090. (APIServer pid=42) INFO 08-03 22:44:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 75.9%
  1091. (APIServer pid=42) INFO 08-03 22:44:21 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 109.49 tokens/s, Accepted: 0 tokens, Drafted: 1095 tokens, Per-position acceptance rate: 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.0%
  1092. (APIServer pid=42) INFO 08-03 22:44:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.7%, Prefix cache hit rate: 75.9%
  1093. (APIServer pid=42) INFO 08-03 22:44:31 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 1.01, Accepted throughput: 0.10 tokens/s, Drafted throughput: 112.49 tokens/s, Accepted: 1 tokens, Drafted: 1125 tokens, Per-position acceptance rate: 0.013, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 0.1%
  1094.  
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!