API tools faq
paste
Guest User

vllm

a guest
Jul 29th, 2025
20
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 20.31 KB | None | 0 0
raw download clone embed print report
  1. (venv) peter@gpu-server:~$ vllm serve /srv/models/awq/GLM-4.5-Air-AWQ --host 192.168.1.12 --port 8000 --api-key token-abc123 --tensor-parallel-size 8 --tool-call-parser glm45 --reasoning-parser glm45 --enable-auto-tool-choice --dtype half --enable-expert-parallel --tensor-parallel-size 2 --pipeline-parallel-size 1 --max-model-len 16384 --max-num-seqs 16 --kv-cache-dtype fp8 --calculate-kv-scales --gpu-memory-utilization 0.98
  2. INFO 07-29 15:58:06 [__init__.py:235] Automatically detected platform cuda.
  3. (APIServer pid=16464) INFO 07-29 15:58:09 [api_server.py:1774] vLLM API server version 0.10.1.dev164+g755fa8b65
  4. (APIServer pid=16464) INFO 07-29 15:58:09 [utils.py:326] non-default args: {'model_tag': '/srv/models/awq/GLM-4.5-Air-AWQ', 'host': '192.168.1.12', 'api_key': 'token-abc123', 'enable_auto_tool_choice': True, 'tool_call_parser': 'glm45', 'model': '/srv/models/awq/GLM-4.5-Air-AWQ', 'dtype': 'half', 'max_model_len': 16384, 'reasoning_parser': 'glm45', 'tensor_parallel_size': 2, 'enable_expert_parallel': True, 'gpu_memory_utilization': 0.98, 'kv_cache_dtype': 'fp8', 'calculate_kv_scales': True, 'max_num_seqs': 16}
  5. (APIServer pid=16464) INFO 07-29 15:58:15 [config.py:713] Resolved architecture: Glm4MoeForCausalLM
  6. (APIServer pid=16464) WARNING 07-29 15:58:15 [config.py:3544] Casting torch.bfloat16 to torch.float16.
  7. (APIServer pid=16464) INFO 07-29 15:58:15 [config.py:1724] Using max model len 16384
  8. (APIServer pid=16464) WARNING 07-29 15:58:15 [arg_utils.py:1694] --kv-cache-dtype is not supported by the V1 Engine. Falling back to V0.
  9. (APIServer pid=16464) INFO 07-29 15:58:15 [config.py:1853] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.
  10. (APIServer pid=16464) INFO 07-29 15:58:16 [api_server.py:280] Started engine process with PID 16536
  11. INFO 07-29 15:58:20 [__init__.py:235] Automatically detected platform cuda.
  12. INFO 07-29 15:58:23 [llm_engine.py:225] Initializing a V0 LLM engine (v0.10.1.dev164+g755fa8b65) with config: model='/srv/models/awq/GLM-4.5-Air-AWQ', speculative_config=None, tokenizer='/srv/models/awq/GLM-4.5-Air-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=fp8, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend='glm45'), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/srv/models/awq/GLM-4.5-Air-AWQ, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":16,"local_cache_dir":null}, use_cached_outputs=True,
  13. WARNING 07-29 15:58:23 [multiproc_worker_utils.py:307] Reducing Torch parallelism from 8 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
  14. INFO 07-29 15:58:23 [cuda.py:331] Using FlashInfer backend.
  15. INFO 07-29 15:58:27 [__init__.py:235] Automatically detected platform cuda.
  16. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:30 [multiproc_worker_utils.py:226] Worker ready; awaiting tasks
  17. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:30 [cuda.py:331] Using FlashInfer backend.
  18. INFO 07-29 15:58:31 [__init__.py:1376] Found nccl from library libnccl.so.2
  19. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:31 [__init__.py:1376] Found nccl from library libnccl.so.2
  20. INFO 07-29 15:58:31 [pynccl.py:70] vLLM is using nccl==2.26.2
  21. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:31 [pynccl.py:70] vLLM is using nccl==2.26.2
  22. INFO 07-29 15:58:31 [custom_all_reduce_utils.py:246] reading GPU P2P access cache from /home/peter/.cache/vllm/gpu_p2p_access_cache_for_0,1,2.json
  23. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:31 [custom_all_reduce_utils.py:246] reading GPU P2P access cache from /home/peter/.cache/vllm/gpu_p2p_access_cache_for_0,1,2.json
  24. (VllmWorkerProcess pid=16574) WARNING 07-29 15:58:31 [custom_all_reduce.py:147] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  25. WARNING 07-29 15:58:31 [custom_all_reduce.py:147] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  26. INFO 07-29 15:58:31 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_35c37c5c'), local_subscribe_addr='ipc:///tmp/222a790a-c5d4-42d1-a650-6f8d42ef0f55', remote_subscribe_addr=None, remote_addr_ipv6=False)
  27. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:31 [parallel_state.py:1102] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
  28. INFO 07-29 15:58:31 [parallel_state.py:1102] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
  29. INFO 07-29 15:58:31 [model_runner.py:1083] Starting to load model /srv/models/awq/GLM-4.5-Air-AWQ...
  30. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:32 [model_runner.py:1083] Starting to load model /srv/models/awq/GLM-4.5-Air-AWQ...
  31. INFO 07-29 15:58:32 [compressed_tensors_wNa16.py:95] Using MarlinLinearKernel for CompressedTensorsWNA16
  32. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:32 [compressed_tensors_wNa16.py:95] Using MarlinLinearKernel for CompressedTensorsWNA16
  33. INFO 07-29 15:58:32 [compressed_tensors_wNa16.py:95] Using ExllamaLinearKernel for CompressedTensorsWNA16
  34. (VllmWorkerProcess pid=16574) INFO 07-29 15:58:33 [compressed_tensors_wNa16.py:95] Using ExllamaLinearKernel for CompressedTensorsWNA16
  35. ERROR 07-29 15:58:33 [engine.py:456]
  36. Traceback (most recent call last):
  37. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 444, in run_mp_engine
  38. engine = MQLLMEngine.from_vllm_config(
  39. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  40. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 133, in from_vllm_config
  41. return cls(
  42. ^^^^
  43. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 87, in __init__
  44. self.engine = LLMEngine(*args, **kwargs)
  45. ^^^^^^^^^^^^^^^^^^^^^^^^^^
  46. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 260, in __init__
  47. self.model_executor = executor_class(vllm_config=vllm_config)
  48. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  49. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 263, in __init__
  50. super().__init__(*args, **kwargs)
  51. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 53, in __init__
  52. self._init_executor()
  53. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/mp_distributed_executor.py", line 126, in _init_executor
  54. self._run_workers("load_model",
  55. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/mp_distributed_executor.py", line 186, in _run_workers
  56. driver_worker_output = run_method(self.driver_worker, sent_method,
  57. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  58. File "/home/peter/venv/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2987, in run_method
  59. return func(*args, **kwargs)
  60. ^^^^^^^^^^^^^^^^^^^^^
  61. File "/home/peter/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 211, in load_model
  62. self.model_runner.load_model()
  63. File "/home/peter/venv/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1086, in load_model
  64. self.model = get_model(vllm_config=self.vllm_config)
  65. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  66. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 118, in get_model
  67. return loader.load_model(vllm_config=vllm_config,
  68. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  69. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 44, in load_model
  70. model = initialize_model(vllm_config=vllm_config,
  71. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  72. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 63, in initialize_model
  73. return model_class(vllm_config=vllm_config, prefix=prefix)
  74. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  75. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 596, in __init__
  76. self.model = Glm4MoeModel(vllm_config=vllm_config,
  77. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  78. File "/home/peter/venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 183, in __init__
  79. old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
  80. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 397, in __init__
  81. self.start_layer, self.end_layer, self.layers = make_layers(
  82. ^^^^^^^^^^^^
  83. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 640, in make_layers
  84. maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
  85. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  86. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 399, in <lambda>
  87. lambda prefix: Glm4MoeDecoderLayer(
  88. ^^^^^^^^^^^^^^^^^^^^
  89. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 343, in __init__
  90. self.mlp = Glm4MoeMLP(hidden_size=config.hidden_size,
  91. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  92. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 81, in __init__
  93. self.down_proj = RowParallelLinear(intermediate_size,
  94. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  95. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 1297, in __init__
  96. self.quant_method.create_weights(
  97. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 659, in create_weights
  98. layer.scheme.create_weights(
  99. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py", line 108, in create_weights
  100. assert input_size_per_partition % group_size == 0
  101. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  102. AssertionError
  103. Process SpawnProcess-1:
  104. ERROR 07-29 15:58:33 [multiproc_worker_utils.py:121] Worker VllmWorkerProcess pid 16574 died, exit code: -15
  105. INFO 07-29 15:58:33 [multiproc_worker_utils.py:125] Killing local vLLM worker processes
  106. Traceback (most recent call last):
  107. File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
  108. self.run()
  109. File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
  110. self._target(*self._args, **self._kwargs)
  111. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 458, in run_mp_engine
  112. raise e from None
  113. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 444, in run_mp_engine
  114. engine = MQLLMEngine.from_vllm_config(
  115. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  116. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 133, in from_vllm_config
  117. return cls(
  118. ^^^^
  119. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 87, in __init__
  120. self.engine = LLMEngine(*args, **kwargs)
  121. ^^^^^^^^^^^^^^^^^^^^^^^^^^
  122. File "/home/peter/venv/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 260, in __init__
  123. self.model_executor = executor_class(vllm_config=vllm_config)
  124. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  125. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 263, in __init__
  126. super().__init__(*args, **kwargs)
  127. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 53, in __init__
  128. self._init_executor()
  129. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/mp_distributed_executor.py", line 126, in _init_executor
  130. self._run_workers("load_model",
  131. File "/home/peter/venv/lib/python3.12/site-packages/vllm/executor/mp_distributed_executor.py", line 186, in _run_workers
  132. driver_worker_output = run_method(self.driver_worker, sent_method,
  133. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  134. File "/home/peter/venv/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2987, in run_method
  135. return func(*args, **kwargs)
  136. ^^^^^^^^^^^^^^^^^^^^^
  137. File "/home/peter/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 211, in load_model
  138. self.model_runner.load_model()
  139. File "/home/peter/venv/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1086, in load_model
  140. self.model = get_model(vllm_config=self.vllm_config)
  141. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  142. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 118, in get_model
  143. return loader.load_model(vllm_config=vllm_config,
  144. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  145. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 44, in load_model
  146. model = initialize_model(vllm_config=vllm_config,
  147. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  148. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/utils.py", line 63, in initialize_model
  149. return model_class(vllm_config=vllm_config, prefix=prefix)
  150. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  151. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 596, in __init__
  152. self.model = Glm4MoeModel(vllm_config=vllm_config,
  153. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  154. File "/home/peter/venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 183, in __init__
  155. old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs)
  156. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 397, in __init__
  157. self.start_layer, self.end_layer, self.layers = make_layers(
  158. ^^^^^^^^^^^^
  159. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 640, in make_layers
  160. maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}"))
  161. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  162. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 399, in <lambda>
  163. lambda prefix: Glm4MoeDecoderLayer(
  164. ^^^^^^^^^^^^^^^^^^^^
  165. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 343, in __init__
  166. self.mlp = Glm4MoeMLP(hidden_size=config.hidden_size,
  167. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  168. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/models/glm4_moe.py", line 81, in __init__
  169. self.down_proj = RowParallelLinear(intermediate_size,
  170. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  171. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/layers/linear.py", line 1297, in __init__
  172. self.quant_method.create_weights(
  173. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 659, in create_weights
  174. layer.scheme.create_weights(
  175. File "/home/peter/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py", line 108, in create_weights
  176. assert input_size_per_partition % group_size == 0
  177. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  178. AssertionError
  179. [rank0]:[W729 15:58:34.680629485 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
  180. (APIServer pid=16464) Traceback (most recent call last):
  181. (APIServer pid=16464) File "/home/peter/venv/bin/vllm", line 8, in <module>
  182. (APIServer pid=16464) sys.exit(main())
  183. (APIServer pid=16464) ^^^^^^
  184. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 54, in main
  185. (APIServer pid=16464) args.dispatch_function(args)
  186. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 52, in cmd
  187. (APIServer pid=16464) uvloop.run(run_server(args))
  188. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
  189. (APIServer pid=16464) return __asyncio.run(
  190. (APIServer pid=16464) ^^^^^^^^^^^^^^
  191. (APIServer pid=16464) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
  192. (APIServer pid=16464) return runner.run(main)
  193. (APIServer pid=16464) ^^^^^^^^^^^^^^^^
  194. (APIServer pid=16464) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
  195. (APIServer pid=16464) return self._loop.run_until_complete(task)
  196. (APIServer pid=16464) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  197. (APIServer pid=16464) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  198. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
  199. (APIServer pid=16464) return await main
  200. (APIServer pid=16464) ^^^^^^^^^^
  201. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1817, in run_server
  202. (APIServer pid=16464) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
  203. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1837, in run_server_worker
  204. (APIServer pid=16464) async with build_async_engine_client(
  205. (APIServer pid=16464) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
  206. (APIServer pid=16464) return await anext(self.gen)
  207. (APIServer pid=16464) ^^^^^^^^^^^^^^^^^^^^^
  208. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 166, in build_async_engine_client
  209. (APIServer pid=16464) async with build_async_engine_client_from_engine_args(
  210. (APIServer pid=16464) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
  211. (APIServer pid=16464) return await anext(self.gen)
  212. (APIServer pid=16464) ^^^^^^^^^^^^^^^^^^^^^
  213. (APIServer pid=16464) File "/home/peter/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 303, in build_async_engine_client_from_engine_args
  214. (APIServer pid=16464) raise RuntimeError(
  215. (APIServer pid=16464) RuntimeError: Engine process failed to start. See stack trace for the root cause.
  216. /usr/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  217. warnings.warn('resource_tracker: There appear to be %d '
  218.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!