API tools faq
paste
bullerwins

Untitled

bullerwins
Oct 28th, 2025
68
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 66.04 KB | None | 0 0
raw download clone embed print report
  1. CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6 VLLM_PP_LAYER_PARTITION=8,6,23,6,6,6,7 vllm serve \
  2. /mnt/llms/models/ModelCloud/MiniMax-M2-GPTQMODEL-W4A16/ \
  3. --served-model-name MiniMax-M2-AWQ \
  4. --enable-auto-tool-choice \
  5. --tool-call-parser minimax_m2 \
  6. --reasoning-parser minimax_m2_append_think \
  7. --swap-space 16 \
  8. --max-num-seqs 32 \
  9. --max-model-len 32000 \
  10. --gpu-memory-utilization 0.9 \
  11. --tensor-parallel-size 1 -pp 7 \
  12. --enable-expert-parallel \
  13. --trust-remote-code \
  14. --disable-log-requests \
  15. --host 0.0.0.0 \
  16. --port 5000
  17. WARNING 10-28 20:00:43 [argparse_utils.py:79] argument '--disable-log-requests' is deprecated and replaced with '--enable-log-requests'. This will be removed in v0.12.0.
  18. (APIServer pid=90425) INFO 10-28 20:00:43 [api_server.py:1869] vLLM API server version 0.11.1rc4.dev66+g130aa8cbc
  19. (APIServer pid=90425) INFO 10-28 20:00:43 [utils.py:253] non-default args: {'model_tag': '/mnt/llms/models/ModelCloud/MiniMax-M2-GPTQMODEL-W4A16/', 'host': '0.0.0.0', 'port': 5000, 'enable_auto_tool_choice': True, 'tool_call_parser': 'minimax_m2', 'model': '/mnt/llms/models/ModelCloud/MiniMax-M2-GPTQMODEL-W4A16/', 'trust_remote_code': True, 'max_model_len': 32000, 'served_model_name': ['MiniMax-M2-AWQ'], 'reasoning_parser': 'minimax_m2_append_think', 'pipeline_parallel_size': 7, 'enable_expert_parallel': True, 'swap_space': 16.0, 'max_num_seqs': 32}
  20. (APIServer pid=90425) The module name (originally ) is not a valid Python identifier. Please rename the original module to avoid import issues.
  21. (APIServer pid=90425) The module name (originally ) is not a valid Python identifier. Please rename the original module to avoid import issues.
  22. (APIServer pid=90425) INFO 10-28 20:00:43 [model.py:668] Resolved architecture: MiniMaxM2ForCausalLM
  23. (APIServer pid=90425) INFO 10-28 20:00:43 [model.py:1773] Using max model len 32000
  24. (APIServer pid=90425) INFO 10-28 20:00:43 [gptq_marlin.py:228] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
  25. (APIServer pid=90425) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
  26. (APIServer pid=90425) INFO 10-28 20:00:43 [scheduler.py:211] Chunked prefill is enabled with max_num_batched_tokens=2048.
  27. (EngineCore_DP0 pid=90613) INFO 10-28 20:01:18 [core.py:93] Initializing a V1 LLM engine (v0.11.1rc4.dev66+g130aa8cbc) with config: model='/mnt/llms/models/ModelCloud/MiniMax-M2-GPTQMODEL-W4A16/', speculative_config=None, tokenizer='/mnt/llms/models/ModelCloud/MiniMax-M2-GPTQMODEL-W4A16/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=7, data_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='minimax_m2_append_think', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=MiniMax-M2-AWQ, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': None, 'mode': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention', 'vllm::sparse_attn_indexer'], 'use_inductor': None, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_cudagraph_capture_size': 64, 'local_cache_dir': None}
  28. (EngineCore_DP0 pid=90613) WARNING 10-28 20:01:18 [multiproc_executor.py:753] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
  29. [W1028 20:01:26.322056532 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [localhost]:38321 (errno: 97 - Address family not supported by protocol).
  30. [W1028 20:01:31.402715065 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [localhost]:38321 (errno: 97 - Address family not supported by protocol).
  31. [W1028 20:01:37.333521526 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [localhost]:38321 (errno: 97 - Address family not supported by protocol).
  32. [W1028 20:01:43.214583062 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [localhost]:38321 (errno: 97 - Address family not supported by protocol).
  33. [W1028 20:01:49.211086288 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [localhost]:38321 (errno: 97 - Address family not supported by protocol).
  34. [W1028 20:01:55.123105856 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [localhost]:38321 (errno: 97 - Address family not supported by protocol).
  35. [W1028 20:02:01.042950399 socket.cpp:767] [c10d] The client socket cannot be initialized to connect to [localhost]:38321 (errno: 97 - Address family not supported by protocol).
  36. [Gloo] Rank 0 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  37. [Gloo] Rank 1 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  38. [Gloo] Rank 2 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  39. [Gloo] Rank 4 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  40. [Gloo] Rank 5 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  41. [Gloo] Rank 3 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  42. [Gloo] Rank 6 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  43. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  44. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  45. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  46. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  47. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  48. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  49. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  50. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  51. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  52. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  53. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  54. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  55. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  56. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  57. [Gloo] Rank 0 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  58. [Gloo] Rank 3 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  59. [Gloo] Rank 1 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  60. [Gloo] Rank 5 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  61. [Gloo] Rank 2 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  62. [Gloo] Rank 4 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  63. [Gloo] Rank 6 is connected to 6 peer ranks. Expected number of connected peer ranks is : 6
  64. INFO 10-28 20:02:01 [pynccl.py:111] vLLM is using nccl==2.27.5
  65. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  66. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  67. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  68. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  69. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  70. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  71. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  72. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  73. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  74. INFO 10-28 20:02:01 [parallel_state.py:1325] rank 6 in world size 7 is assigned as DP rank 0, PP rank 6, TP rank 0, EP rank 0
  75. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  76. INFO 10-28 20:02:01 [parallel_state.py:1325] rank 0 in world size 7 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
  77. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  78. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  79. INFO 10-28 20:02:01 [parallel_state.py:1325] rank 3 in world size 7 is assigned as DP rank 0, PP rank 3, TP rank 0, EP rank 0
  80. INFO 10-28 20:02:01 [parallel_state.py:1325] rank 2 in world size 7 is assigned as DP rank 0, PP rank 2, TP rank 0, EP rank 0
  81. INFO 10-28 20:02:01 [parallel_state.py:1325] rank 1 in world size 7 is assigned as DP rank 0, PP rank 1, TP rank 0, EP rank 0
  82. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  83. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  84. INFO 10-28 20:02:01 [parallel_state.py:1325] rank 5 in world size 7 is assigned as DP rank 0, PP rank 5, TP rank 0, EP rank 0
  85. INFO 10-28 20:02:01 [parallel_state.py:1325] rank 4 in world size 7 is assigned as DP rank 0, PP rank 4, TP rank 0, EP rank 0
  86. (Worker_PP0_EP0 pid=90725) INFO 10-28 20:02:02 [gpu_model_runner.py:2849] Starting to load model /mnt/llms/models/ModelCloud/MiniMax-M2-GPTQMODEL-W4A16/...
  87. (Worker_PP0_EP0 pid=90725) INFO 10-28 20:02:02 [gptq_marlin.py:359] Using MarlinLinearKernel for GPTQMarlinLinearMethod
  88. (Worker_PP6_EP0 pid=90895) INFO 10-28 20:02:02 [gptq_marlin.py:359] Using MarlinLinearKernel for GPTQMarlinLinearMethod
  89. (Worker_PP2_EP0 pid=90779) INFO 10-28 20:02:02 [gptq_marlin.py:359] Using MarlinLinearKernel for GPTQMarlinLinearMethod
  90. (Worker_PP1_EP0 pid=90737) INFO 10-28 20:02:02 [gptq_marlin.py:359] Using MarlinLinearKernel for GPTQMarlinLinearMethod
  91. (Worker_PP5_EP0 pid=90871) INFO 10-28 20:02:02 [gptq_marlin.py:359] Using MarlinLinearKernel for GPTQMarlinLinearMethod
  92. (Worker_PP4_EP0 pid=90847) INFO 10-28 20:02:02 [gptq_marlin.py:359] Using MarlinLinearKernel for GPTQMarlinLinearMethod
  93. (Worker_PP3_EP0 pid=90804) INFO 10-28 20:02:02 [gptq_marlin.py:359] Using MarlinLinearKernel for GPTQMarlinLinearMethod
  94. (Worker_PP0_EP0 pid=90725) INFO 10-28 20:02:02 [cuda.py:405] Using Flash Attention backend on V1 engine.
  95. (Worker_PP6_EP0 pid=90895) INFO 10-28 20:02:02 [cuda.py:405] Using Flash Attention backend on V1 engine.
  96. (Worker_PP2_EP0 pid=90779) INFO 10-28 20:02:02 [cuda.py:405] Using Flash Attention backend on V1 engine.
  97. (Worker_PP1_EP0 pid=90737) INFO 10-28 20:02:02 [cuda.py:405] Using Flash Attention backend on V1 engine.
  98. (Worker_PP5_EP0 pid=90871) INFO 10-28 20:02:02 [cuda.py:405] Using Flash Attention backend on V1 engine.
  99. (Worker_PP4_EP0 pid=90847) INFO 10-28 20:02:02 [cuda.py:405] Using Flash Attention backend on V1 engine.
  100. (Worker_PP3_EP0 pid=90804) INFO 10-28 20:02:02 [cuda.py:405] Using Flash Attention backend on V1 engine.
  101. Loading safetensors checkpoint shards: 0% Completed | 0/32 [00:00<?, ?it/s]
  102. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] WorkerProc failed to start.
  103. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] Traceback (most recent call last):
  104. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in worker_main
  105. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] worker = WorkerProc(*args, **kwargs)
  106. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  107. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 460, in __init__
  108. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.worker.load_model()
  109. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 233, in load_model
  110. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model_runner.load_model(eep_scale_up=eep_scale_up)
  111. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2883, in load_model
  112. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model = model_loader.load_model(
  113. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^
  114. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
  115. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.load_weights(model, model_config)
  116. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 300, in load_weights
  117. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_weights = model.load_weights(
  118. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^
  119. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 556, in load_weights
  120. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return loader.load_weights(weights)
  121. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  122. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in load_weights
  123. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] autoloaded_weights = set(self._load_module("", self.module, weights))
  124. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  125. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 282, in _load_module
  126. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._load_module(
  127. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 255, in _load_module
  128. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_params = module_load_weights(weights)
  129. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  130. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 462, in load_weights
  131. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] param = params_dict[name]
  132. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ~~~~~~~~~~~^^^^^^
  133. (Worker_PP0_EP0 pid=90725) ERROR 10-28 20:02:18 [multiproc_executor.py:631] KeyError: 'layers.7.self_attn.qkv_proj.g_idx'
  134. Loading safetensors checkpoint shards: 0% Completed | 0/32 [00:14<?, ?it/s]
  135. (Worker_PP0_EP0 pid=90725)
  136. (Worker_PP1_EP0 pid=90737) INFO 10-28 20:02:18 [multiproc_executor.py:592] Parent process exited, terminating worker
  137. (Worker_PP0_EP0 pid=90725) INFO 10-28 20:02:18 [multiproc_executor.py:592] Parent process exited, terminating worker
  138. (Worker_PP5_EP0 pid=90871) INFO 10-28 20:02:18 [multiproc_executor.py:592] Parent process exited, terminating worker
  139. (Worker_PP3_EP0 pid=90804) INFO 10-28 20:02:18 [multiproc_executor.py:592] Parent process exited, terminating worker
  140. (Worker_PP4_EP0 pid=90847) INFO 10-28 20:02:18 [multiproc_executor.py:592] Parent process exited, terminating worker
  141. (Worker_PP6_EP0 pid=90895) INFO 10-28 20:02:18 [multiproc_executor.py:592] Parent process exited, terminating worker
  142. (Worker_PP2_EP0 pid=90779) INFO 10-28 20:02:18 [multiproc_executor.py:592] Parent process exited, terminating worker
  143. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] WorkerProc failed to start.
  144. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] Traceback (most recent call last):
  145. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in worker_main
  146. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] worker = WorkerProc(*args, **kwargs)
  147. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  148. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 460, in __init__
  149. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.worker.load_model()
  150. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 233, in load_model
  151. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model_runner.load_model(eep_scale_up=eep_scale_up)
  152. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2883, in load_model
  153. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model = model_loader.load_model(
  154. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^
  155. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
  156. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.load_weights(model, model_config)
  157. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 300, in load_weights
  158. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_weights = model.load_weights(
  159. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^
  160. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 556, in load_weights
  161. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return loader.load_weights(weights)
  162. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  163. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in load_weights
  164. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] autoloaded_weights = set(self._load_module("", self.module, weights))
  165. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  166. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 282, in _load_module
  167. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._load_module(
  168. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 255, in _load_module
  169. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_params = module_load_weights(weights)
  170. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  171. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 434, in load_weights
  172. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for name, loaded_weight in weights:
  173. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  174. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 164, in <genexpr>
  175. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for parts, weights_data in group
  176. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^
  177. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 154, in <genexpr>
  178. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for weight_name, weight_data in weights
  179. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  180. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 325, in <genexpr>
  181. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] (name, weight) for name, weight in weights if not self._can_skip(name)
  182. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  183. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 258, in get_all_weights
  184. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._get_weights_iterator(primary_weights)
  185. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 244, in <genexpr>
  186. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return ((source.prefix + name, tensor) for (name, tensor) in weights_iterator)
  187. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^
  188. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 627, in safetensors_weights_iterator
  189. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] param = f.get_tensor(name)
  190. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^
  191. (Worker_PP5_EP0 pid=90871) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ValueError: could not determine the shape of object type 'torch.storage.UntypedStorage'
  192. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] WorkerProc failed to start.
  193. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] Traceback (most recent call last):
  194. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in worker_main
  195. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] worker = WorkerProc(*args, **kwargs)
  196. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  197. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 460, in __init__
  198. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.worker.load_model()
  199. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 233, in load_model
  200. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model_runner.load_model(eep_scale_up=eep_scale_up)
  201. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2883, in load_model
  202. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model = model_loader.load_model(
  203. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^
  204. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
  205. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.load_weights(model, model_config)
  206. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 300, in load_weights
  207. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_weights = model.load_weights(
  208. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^
  209. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 556, in load_weights
  210. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return loader.load_weights(weights)
  211. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  212. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in load_weights
  213. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] autoloaded_weights = set(self._load_module("", self.module, weights))
  214. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  215. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 282, in _load_module
  216. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._load_module(
  217. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 255, in _load_module
  218. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_params = module_load_weights(weights)
  219. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  220. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 434, in load_weights
  221. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for name, loaded_weight in weights:
  222. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  223. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 164, in <genexpr>
  224. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for parts, weights_data in group
  225. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^
  226. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 154, in <genexpr>
  227. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for weight_name, weight_data in weights
  228. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  229. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 325, in <genexpr>
  230. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] (name, weight) for name, weight in weights if not self._can_skip(name)
  231. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  232. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 258, in get_all_weights
  233. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._get_weights_iterator(primary_weights)
  234. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 244, in <genexpr>
  235. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return ((source.prefix + name, tensor) for (name, tensor) in weights_iterator)
  236. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^
  237. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 627, in safetensors_weights_iterator
  238. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] param = f.get_tensor(name)
  239. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^
  240. (Worker_PP3_EP0 pid=90804) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ValueError: could not determine the shape of object type 'torch.storage.UntypedStorage'
  241. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] WorkerProc failed to start.
  242. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] Traceback (most recent call last):
  243. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in worker_main
  244. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] worker = WorkerProc(*args, **kwargs)
  245. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  246. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 460, in __init__
  247. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.worker.load_model()
  248. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 233, in load_model
  249. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model_runner.load_model(eep_scale_up=eep_scale_up)
  250. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2883, in load_model
  251. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model = model_loader.load_model(
  252. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] WorkerProc failed to start.
  253. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^
  254. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] Traceback (most recent call last):
  255. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
  256. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in worker_main
  257. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.load_weights(model, model_config)
  258. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] worker = WorkerProc(*args, **kwargs)
  259. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 300, in load_weights
  260. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  261. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_weights = model.load_weights(
  262. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 460, in __init__
  263. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^
  264. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.worker.load_model()
  265. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 556, in load_weights
  266. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 233, in load_model
  267. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return loader.load_weights(weights)
  268. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model_runner.load_model(eep_scale_up=eep_scale_up)
  269. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  270. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2883, in load_model
  271. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in load_weights
  272. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model = model_loader.load_model(
  273. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] autoloaded_weights = set(self._load_module("", self.module, weights))
  274. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^
  275. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  276. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
  277. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 282, in _load_module
  278. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.load_weights(model, model_config)
  279. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._load_module(
  280. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 300, in load_weights
  281. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 255, in _load_module
  282. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_weights = model.load_weights(
  283. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_params = module_load_weights(weights)
  284. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^
  285. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  286. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 556, in load_weights
  287. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 434, in load_weights
  288. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return loader.load_weights(weights)
  289. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for name, loaded_weight in weights:
  290. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  291. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  292. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in load_weights
  293. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 164, in <genexpr>
  294. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] autoloaded_weights = set(self._load_module("", self.module, weights))
  295. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for parts, weights_data in group
  296. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  297. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^
  298. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 282, in _load_module
  299. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 154, in <genexpr>
  300. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._load_module(
  301. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for weight_name, weight_data in weights
  302. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 255, in _load_module
  303. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  304. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_params = module_load_weights(weights)
  305. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 325, in <genexpr>
  306. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  307. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] (name, weight) for name, weight in weights if not self._can_skip(name)
  308. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 434, in load_weights
  309. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  310. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for name, loaded_weight in weights:
  311. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 258, in get_all_weights
  312. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  313. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._get_weights_iterator(primary_weights)
  314. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 164, in <genexpr>
  315. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 244, in <genexpr>
  316. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for parts, weights_data in group
  317. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return ((source.prefix + name, tensor) for (name, tensor) in weights_iterator)
  318. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^
  319. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^
  320. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 154, in <genexpr>
  321. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 627, in safetensors_weights_iterator
  322. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for weight_name, weight_data in weights
  323. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] param = f.get_tensor(name)
  324. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  325. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^
  326. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 325, in <genexpr>
  327. (Worker_PP4_EP0 pid=90847) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ValueError: could not determine the shape of object type 'torch.storage.UntypedStorage'
  328. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] (name, weight) for name, weight in weights if not self._can_skip(name)
  329. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  330. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 258, in get_all_weights
  331. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._get_weights_iterator(primary_weights)
  332. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 244, in <genexpr>
  333. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return ((source.prefix + name, tensor) for (name, tensor) in weights_iterator)
  334. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^
  335. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 627, in safetensors_weights_iterator
  336. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] param = f.get_tensor(name)
  337. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^
  338. (Worker_PP6_EP0 pid=90895) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ValueError: could not determine the shape of object type 'torch.storage.UntypedStorage'
  339. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] WorkerProc failed to start.
  340. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] Traceback (most recent call last):
  341. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 605, in worker_main
  342. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] worker = WorkerProc(*args, **kwargs)
  343. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  344. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 460, in __init__
  345. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.worker.load_model()
  346. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 233, in load_model
  347. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model_runner.load_model(eep_scale_up=eep_scale_up)
  348. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2883, in load_model
  349. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.model = model_loader.load_model(
  350. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^
  351. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
  352. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] self.load_weights(model, model_config)
  353. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 300, in load_weights
  354. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_weights = model.load_weights(
  355. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^
  356. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 556, in load_weights
  357. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return loader.load_weights(weights)
  358. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  359. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 328, in load_weights
  360. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] autoloaded_weights = set(self._load_module("", self.module, weights))
  361. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  362. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 282, in _load_module
  363. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._load_module(
  364. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 255, in _load_module
  365. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] loaded_params = module_load_weights(weights)
  366. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  367. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/minimax_m2.py", line 434, in load_weights
  368. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for name, loaded_weight in weights:
  369. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  370. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 164, in <genexpr>
  371. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for parts, weights_data in group
  372. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^
  373. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 154, in <genexpr>
  374. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] for weight_name, weight_data in weights
  375. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  376. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 325, in <genexpr>
  377. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] (name, weight) for name, weight in weights if not self._can_skip(name)
  378. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^
  379. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 258, in get_all_weights
  380. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] yield from self._get_weights_iterator(primary_weights)
  381. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/default_loader.py", line 244, in <genexpr>
  382. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] return ((source.prefix + name, tensor) for (name, tensor) in weights_iterator)
  383. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^
  384. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py", line 627, in safetensors_weights_iterator
  385. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] param = f.get_tensor(name)
  386. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ^^^^^^^^^^^^^^^^^^
  387. (Worker_PP2_EP0 pid=90779) ERROR 10-28 20:02:18 [multiproc_executor.py:631] ValueError: could not determine the shape of object type 'torch.storage.UntypedStorage'
  388. [rank0]:[W1028 20:02:18.689988840 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
  389. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] EngineCore failed to start.
  390. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] Traceback (most recent call last):
  391. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 770, in run_engine_core
  392. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] engine_core = EngineCoreProc(*args, **kwargs)
  393. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  394. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 538, in __init__
  395. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] super().__init__(
  396. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 102, in __init__
  397. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] self.model_executor = executor_class(vllm_config)
  398. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  399. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 98, in __init__
  400. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] self._init_executor()
  401. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 116, in _init_executor
  402. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] self.workers = WorkerProc.wait_for_ready(unready_workers)
  403. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  404. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 543, in wait_for_ready
  405. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] raise e from None
  406. (EngineCore_DP0 pid=90613) ERROR 10-28 20:02:22 [core.py:779] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
  407. (EngineCore_DP0 pid=90613) Process EngineCore_DP0:
  408. (EngineCore_DP0 pid=90613) Traceback (most recent call last):
  409. (EngineCore_DP0 pid=90613) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
  410. (EngineCore_DP0 pid=90613) self.run()
  411. (EngineCore_DP0 pid=90613) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
  412. (EngineCore_DP0 pid=90613) self._target(*self._args, **self._kwargs)
  413. (EngineCore_DP0 pid=90613) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 783, in run_engine_core
  414. (EngineCore_DP0 pid=90613) raise e
  415. (EngineCore_DP0 pid=90613) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 770, in run_engine_core
  416. (EngineCore_DP0 pid=90613) engine_core = EngineCoreProc(*args, **kwargs)
  417. (EngineCore_DP0 pid=90613) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  418. (EngineCore_DP0 pid=90613) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 538, in __init__
  419. (EngineCore_DP0 pid=90613) super().__init__(
  420. (EngineCore_DP0 pid=90613) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 102, in __init__
  421. (EngineCore_DP0 pid=90613) self.model_executor = executor_class(vllm_config)
  422. (EngineCore_DP0 pid=90613) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  423. (EngineCore_DP0 pid=90613) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 98, in __init__
  424. (EngineCore_DP0 pid=90613) self._init_executor()
  425. (EngineCore_DP0 pid=90613) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 116, in _init_executor
  426. (EngineCore_DP0 pid=90613) self.workers = WorkerProc.wait_for_ready(unready_workers)
  427. (EngineCore_DP0 pid=90613) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  428. (EngineCore_DP0 pid=90613) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 543, in wait_for_ready
  429. (EngineCore_DP0 pid=90613) raise e from None
  430. (EngineCore_DP0 pid=90613) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
  431. (APIServer pid=90425) Traceback (most recent call last):
  432. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/bin/vllm", line 10, in <module>
  433. (APIServer pid=90425) sys.exit(main())
  434. (APIServer pid=90425) ^^^^^^
  435. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
  436. (APIServer pid=90425) args.dispatch_function(args)
  437. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 59, in cmd
  438. (APIServer pid=90425) uvloop.run(run_server(args))
  439. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
  440. (APIServer pid=90425) return __asyncio.run(
  441. (APIServer pid=90425) ^^^^^^^^^^^^^^
  442. (APIServer pid=90425) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
  443. (APIServer pid=90425) return runner.run(main)
  444. (APIServer pid=90425) ^^^^^^^^^^^^^^^^
  445. (APIServer pid=90425) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
  446. (APIServer pid=90425) return self._loop.run_until_complete(task)
  447. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  448. (APIServer pid=90425) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  449. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
  450. (APIServer pid=90425) return await main
  451. (APIServer pid=90425) ^^^^^^^^^^
  452. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1913, in run_server
  453. (APIServer pid=90425) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
  454. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 1929, in run_server_worker
  455. (APIServer pid=90425) async with build_async_engine_client(
  456. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^^^^^^
  457. (APIServer pid=90425) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
  458. (APIServer pid=90425) return await anext(self.gen)
  459. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^
  460. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 184, in build_async_engine_client
  461. (APIServer pid=90425) async with build_async_engine_client_from_engine_args(
  462. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  463. (APIServer pid=90425) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
  464. (APIServer pid=90425) return await anext(self.gen)
  465. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^
  466. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 231, in build_async_engine_client_from_engine_args
  467. (APIServer pid=90425) async_llm = AsyncLLM.from_vllm_config(
  468. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^^^^^^
  469. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/utils/func_utils.py", line 116, in inner
  470. (APIServer pid=90425) return fn(*args, **kwargs)
  471. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^
  472. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 219, in from_vllm_config
  473. (APIServer pid=90425) return cls(
  474. (APIServer pid=90425) ^^^^
  475. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 141, in __init__
  476. (APIServer pid=90425) self.engine_core = EngineCoreClient.make_async_mp_client(
  477. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  478. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
  479. (APIServer pid=90425) return AsyncMPClient(*client_args)
  480. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  481. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 807, in __init__
  482. (APIServer pid=90425) super().__init__(
  483. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 468, in __init__
  484. (APIServer pid=90425) with launch_core_engines(vllm_config, executor_class, log_stats) as (
  485. (APIServer pid=90425) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  486. (APIServer pid=90425) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
  487. (APIServer pid=90425) next(self.gen)
  488. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 889, in launch_core_engines
  489. (APIServer pid=90425) wait_for_engine_startup(
  490. (APIServer pid=90425) File "/home/ubuntuai/vllm/.venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 946, in wait_for_engine_startup
  491. (APIServer pid=90425) raise RuntimeError(
  492. (APIServer pid=90425) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!