API tools faq
paste
Guest User

glm-4_5-air-awq-test-rtx-pro-6000

a guest
Jul 30th, 2025
12
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 10.40 KB | None | 0 0
raw download clone embed print report
  1. (vllm) [email protected]:/workspace$ vllm serve cpatonn/GLM-4.5-Air-AWQ --dtype float16 --quantization compressed-tensors
  2. INFO 07-30 12:17:53 [__init__.py:235] Automatically detected platform cuda.
  3. (APIServer pid=4148) INFO 07-30 12:17:55 [api_server.py:1774] vLLM API server version 0.10.1.dev202+g5bbaf492a
  4. (APIServer pid=4148) INFO 07-30 12:17:55 [utils.py:326] non-default args: {'model_tag': 'cpatonn/GLM-4.5-Air-AWQ', 'model': 'cpatonn/GLM-4.5-Air-AWQ', 'dtype': 'float16', 'quantization': 'compressed-tensors'}
  5. (APIServer pid=4148) INFO 07-30 12:18:00 [config.py:713] Resolved architecture: Glm4MoeForCausalLM
  6. (APIServer pid=4148) WARNING 07-30 12:18:00 [config.py:3549] Casting torch.bfloat16 to torch.float16.
  7. (APIServer pid=4148) INFO 07-30 12:18:00 [config.py:1729] Using max model len 131072
  8. (APIServer pid=4148) INFO 07-30 12:18:01 [config.py:2540] Chunked prefill is enabled with max_num_batched_tokens=8192.
  9. INFO 07-30 12:18:05 [__init__.py:235] Automatically detected platform cuda.
  10. INFO 07-30 12:18:06 [core.py:586] Waiting for init message from front-end.
  11. INFO 07-30 12:18:06 [core.py:73] Initializing a V1 LLM engine (v0.10.1.dev202+g5bbaf492a) with config: model='cpatonn/GLM-4.5-Air-AWQ', speculative_config=None, tokenizer='cpatonn/GLM-4.5-Air-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=cpatonn/GLM-4.5-Air-AWQ, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
  12. INFO 07-30 12:18:07 [parallel_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
  13. INFO 07-30 12:18:07 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling.
  14. INFO 07-30 12:18:07 [gpu_model_runner.py:1866] Starting to load model cpatonn/GLM-4.5-Air-AWQ...
  15. INFO 07-30 12:18:07 [gpu_model_runner.py:1898] Loading model from scratch...
  16. INFO 07-30 12:18:07 [compressed_tensors_wNa16.py:95] Using MarlinLinearKernel for CompressedTensorsWNA16
  17. INFO 07-30 12:18:07 [cuda.py:305] Using Flash Attention backend on V1 engine.
  18. INFO 07-30 12:18:07 [compressed_tensors_wNa16.py:95] Using ExllamaLinearKernel for CompressedTensorsWNA16
  19. INFO 07-30 12:18:07 [compressed_tensors_moe.py:81] Using CompressedTensorsWNA16MarlinMoEMethod
  20. INFO 07-30 12:18:08 [weight_utils.py:296] Using model weights format ['*.safetensors']
  21. Loading safetensors checkpoint shards: 0% Completed | 0/13 [00:00<?, ?it/s]
  22. Loading safetensors checkpoint shards: 8% Completed | 1/13 [00:02<00:27, 2.27s/it]
  23. Loading safetensors checkpoint shards: 15% Completed | 2/13 [00:04<00:26, 2.42s/it]
  24. Loading safetensors checkpoint shards: 23% Completed | 3/13 [00:07<00:24, 2.47s/it]
  25. Loading safetensors checkpoint shards: 31% Completed | 4/13 [00:09<00:20, 2.32s/it]
  26. Loading safetensors checkpoint shards: 38% Completed | 5/13 [00:11<00:19, 2.41s/it]
  27. Loading safetensors checkpoint shards: 46% Completed | 6/13 [00:14<00:17, 2.45s/it]
  28. Loading safetensors checkpoint shards: 54% Completed | 7/13 [00:17<00:14, 2.47s/it]
  29. Loading safetensors checkpoint shards: 62% Completed | 8/13 [00:19<00:12, 2.49s/it]
  30. Loading safetensors checkpoint shards: 69% Completed | 9/13 [00:20<00:07, 1.97s/it]
  31. Loading safetensors checkpoint shards: 77% Completed | 10/13 [00:22<00:06, 2.09s/it]
  32. Loading safetensors checkpoint shards: 85% Completed | 11/13 [00:25<00:04, 2.22s/it]
  33. Loading safetensors checkpoint shards: 92% Completed | 12/13 [00:27<00:02, 2.32s/it]
  34. Loading safetensors checkpoint shards: 100% Completed | 13/13 [00:30<00:00, 2.39s/it]
  35. Loading safetensors checkpoint shards: 100% Completed | 13/13 [00:30<00:00, 2.33s/it]
  36.  
  37. INFO 07-30 12:18:39 [default_loader.py:262] Loading weights took 30.70 seconds
  38. INFO 07-30 12:18:40 [gpu_model_runner.py:1915] Model loading took 57.7051 GiB and 32.535371 seconds
  39. INFO 07-30 12:18:49 [backends.py:530] Using cache directory: /root/.cache/vllm/torch_compile_cache/e8629d07d0/rank_0_0/backbone for vLLM's torch.compile
  40. INFO 07-30 12:18:49 [backends.py:541] Dynamo bytecode transform time: 8.88 s
  41. INFO 07-30 12:18:57 [backends.py:161] Directly load the compiled graph(s) for dynamic shape from the cache, took 6.287 s
  42. WARNING 07-30 12:18:59 [fused_moe.py:696] Using default MoE config. Performance might be sub-optimal! Config file not found at /venv/vllm/lib/python3.12/site-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=8192,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json
  43. INFO 07-30 12:18:59 [monitor.py:34] torch.compile takes 8.88 s in total
  44. /venv/vllm/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  45. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  46. warnings.warn(
  47. INFO 07-30 12:19:00 [gpu_worker.py:276] Available KV cache memory: 25.65 GiB
  48. INFO 07-30 12:19:01 [kv_cache_utils.py:839] GPU KV cache size: 146,176 tokens
  49. INFO 07-30 12:19:01 [kv_cache_utils.py:843] Maximum concurrency for 131,072 tokens per request: 1.12x
  50. Capturing CUDA graph shapes: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:05<00:00, 11.61it/s]
  51. INFO 07-30 12:19:07 [gpu_model_runner.py:2520] Graph capturing finished in 6 secs, took 1.55 GiB
  52. INFO 07-30 12:19:07 [core.py:201] init engine (profile, create kv cache, warmup model) took 26.40 seconds
  53. (APIServer pid=4148) INFO 07-30 12:19:08 [loggers.py:142] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 9136
  54. (APIServer pid=4148) INFO 07-30 12:19:08 [api_server.py:1595] Supported_tasks: ['generate']
  55. (APIServer pid=4148) INFO 07-30 12:19:09 [api_server.py:1847] Starting vLLM API server 0 on http://0.0.0.0:8000
  56. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:29] Available routes are:
  57. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /openapi.json, Methods: HEAD, GET
  58. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /docs, Methods: HEAD, GET
  59. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: HEAD, GET
  60. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /redoc, Methods: HEAD, GET
  61. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /health, Methods: GET
  62. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /load, Methods: GET
  63. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /ping, Methods: POST
  64. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /ping, Methods: GET
  65. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /tokenize, Methods: POST
  66. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /detokenize, Methods: POST
  67. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/models, Methods: GET
  68. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /version, Methods: GET
  69. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/responses, Methods: POST
  70. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/responses/{response_id}, Methods: GET
  71. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/responses/{response_id}/cancel, Methods: POST
  72. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
  73. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/completions, Methods: POST
  74. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/embeddings, Methods: POST
  75. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /pooling, Methods: POST
  76. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /classify, Methods: POST
  77. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /score, Methods: POST
  78. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/score, Methods: POST
  79. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
  80. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/audio/translations, Methods: POST
  81. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /rerank, Methods: POST
  82. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v1/rerank, Methods: POST
  83. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /v2/rerank, Methods: POST
  84. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /scale_elastic_ep, Methods: POST
  85. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /is_scaling_elastic_ep, Methods: POST
  86. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /invocations, Methods: POST
  87. (APIServer pid=4148) INFO 07-30 12:19:09 [launcher.py:37] Route: /metrics, Methods: GET
  88. (APIServer pid=4148) INFO: Started server process [4148]
  89. (APIServer pid=4148) INFO: Waiting for application startup.
  90. (APIServer pid=4148) INFO: Application startup complete.
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!