API tools faq
paste
MrCsabaToth

Error during model RLVR GRPO

MrCsabaToth
Mar 8th, 2026 (edited)
47
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 36.01 KB | Software | 0 0
raw download clone embed print report
  1. from unsloth import FastLanguageModel
  2.  
  3. model, tokenizer = FastLanguageModel.from_pretrained(
  4.     model_name=MODEL_NAME,
  5.     max_seq_length=MAX_SEQ_LENGTH,
  6.     load_in_4bit=False,   # False = 16-bit LoRA (not 4-bit)
  7.     fast_inference=True,  # enable vLLM as the GRPO rollout engine
  8.     max_lora_rank=LORA_RANK,  # pre-allocate vLLM memory for LoRA
  9.     load_in_fp8=True,     # Float8 training — the key FP8 flag
  10. )
  11. print('Base model loaded.')
  12.  
  13. 🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
  14. 🦥 Unsloth Zoo will now patch everything to make training faster!
  15. INFO 03-08 07:38:45 [vllm_utils.py:723] Unsloth: Patching vLLM v1 graph capture
  16. ==((====))==  Unsloth 2026.3.3: Fast Qwen3 patching. Transformers: 4.56.2. vLLM: 0.15.1.
  17.    \\   /|    NVIDIA H100 80GB HBM3. Num GPUs = 1. Max memory: 79.179 GB. Platform: Linux.
  18. O^O/ \_/ \    Torch: 2.9.1+cu128. CUDA: 9.0. CUDA Toolkit: 12.8. Triton: 3.5.1
  19. \        /    Bfloat16 = TRUE. FA [Xformers = 0.0.33.post2. FA2 = False]
  20.  "-____-"     Free license: http://github.com/unslothai/unsloth
  21. Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
  22. Unsloth: Standby mode is enabled. Changing `gpu_memory_utilization` to 0.87875.
  23. Unsloth: vLLM loading unsloth/Qwen3-8B-FP8 with actual GPU utilization = 87.21%
  24. Unsloth: Your GPU has CUDA compute capability 9.0 with VRAM = 79.18 GB.
  25. Unsloth: Using conservativeness = 1.0. Chunked prefill tokens = 2048. Num Sequences = 128.
  26. Unsloth: vLLM's KV Cache can use up to 58.33 GB. Also swap space = 6 GB.
  27. Unsloth: Not an error, but `use_cudagraph` is not supported in vLLM.config.CompilationConfig. Skipping.
  28. Unsloth: Not an error, but `use_inductor` is not supported in vLLM.config.CompilationConfig. Skipping.
  29. WARNING 03-08 07:38:51 [compilation.py:762] Level is deprecated and will be removed in the next release,either 0.12.0 or 0.11.2 whichever is soonest.Use mode instead.If both level and mode are given,only mode will be used.
  30. Unsloth: Not an error, but `device` is not supported in vLLM. Skipping.
  31. INFO 03-08 07:38:51 [utils.py:261] non-default args: {'dtype': torch.bfloat16, 'max_model_len': 2048, 'enable_prefix_caching': True, 'swap_space': 6, 'gpu_memory_utilization': 0.8720648659409341, 'max_num_batched_tokens': 8192, 'max_num_seqs': 128, 'max_logprobs': 0, 'disable_log_stats': True, 'enable_lora': True, 'max_lora_rank': 32, 'enable_chunked_prefill': True, 'compilation_config': {'level': 3, 'mode': 3, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'splitting_ops': None, 'compile_mm_encoder': False, 'compile_sizes': None, 'compile_ranges_split_points': None, 'inductor_compile_config': {'epilogue_fusion': True, 'max_autotune': False, 'shape_padding': True, 'trace.enabled': False, 'triton.cudagraphs': False, 'debug': False, 'dce': True, 'memory_planning': True, 'coordinate_descent_tuning': False, 'trace.graph_diagram': False, 'compile_threads': 32, 'group_fusion': True, 'disable_progress': False, 'verbose_progress': True, 'triton.multi_kernel': 0, 'triton.use_block_ptr': True, 'triton.enable_persistent_tma_matmul': True, 'triton.autotune_at_compile_time': False, 'triton.cooperative_reductions': False, 'cuda.compile_opt_level': '-O2', 'cuda.enable_cuda_lto': True, 'combo_kernels': False, 'benchmark_combo_kernel': True, 'combo_kernel_foreach_dynamic_shapes': True, 'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}, 'enable_sleep_mode': True, 'model': 'unsloth/Qwen3-8B-FP8'}
  32. WARNING 03-08 07:38:51 [arg_utils.py:1220] The global random seed is set to 0. Since VLLM_ENABLE_V1_MULTIPROCESSING is set to False, this may affect the random state of the Python process that launched vLLM.
  33. /opt/conda/lib/python3.13/site-packages/pydantic/type_adapter.py:605: UserWarning: Pydantic serializer warnings:
  34.  PydanticSerializationUnexpectedValue(Expected `enum` - serialized value may not be as expected [field_name='mode', input_value=3, input_type=int])
  35.  return self.serializer.to_python(
  36. INFO 03-08 07:38:59 [model.py:541] Resolved architecture: Qwen3ForCausalLM
  37. INFO 03-08 07:38:59 [model.py:1561] Using max model len 2048
  38. INFO 03-08 07:38:59 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=8192.
  39. INFO 03-08 07:38:59 [vllm.py:624] Asynchronous scheduling is enabled.
  40. INFO 03-08 07:39:00 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='unsloth/Qwen3-8B-FP8', speculative_config=None, tokenizer='unsloth/Qwen3-8B-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=unsloth/Qwen3-8B-FP8, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': 3, 'mode': 3, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'epilogue_fusion': True, 'max_autotune': False, 'shape_padding': True, 'trace.enabled': False, 'triton.cudagraphs': False, 'debug': False, 'dce': True, 'memory_planning': True, 'coordinate_descent_tuning': False, 'trace.graph_diagram': False, 'compile_threads': 32, 'group_fusion': True, 'disable_progress': False, 'verbose_progress': True, 'triton.multi_kernel': 0, 'triton.use_block_ptr': True, 'triton.enable_persistent_tma_matmul': True, 'triton.autotune_at_compile_time': False, 'triton.cooperative_reductions': False, 'cuda.compile_opt_level': '-O2', 'cuda.enable_cuda_lto': True, 'combo_kernels': False, 'benchmark_combo_kernel': True, 'combo_kernel_foreach_dynamic_shapes': True, 'enable_auto_functionalized_v2': False}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}
  41. INFO 03-08 07:39:00 [parallel_state.py:1212] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.0.2.91:36873 backend=nccl
  42. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  43. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  44. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  45. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  46. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  47. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
  48. INFO 03-08 07:39:00 [parallel_state.py:1423] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
  49. /opt/conda/lib/python3.13/site-packages/pydantic/type_adapter.py:605: UserWarning: Pydantic serializer warnings:
  50.  PydanticSerializationUnexpectedValue(Expected `enum` - serialized value may not be as expected [field_name='mode', input_value=3, input_type=int])
  51.  return self.serializer.to_python(
  52. INFO 03-08 07:39:00 [topk_topp_sampler.py:47] Using FlashInfer for top-p & top-k sampling.
  53. INFO 03-08 07:39:00 [gpu_model_runner.py:4033] Starting to load model unsloth/Qwen3-8B-FP8...
  54. INFO 03-08 07:39:01 [cuda.py:364] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
  55. <frozen importlib._bootstrap_external>:1328: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
  56. <frozen importlib._bootstrap_external>:1328: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
  57. Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.65it/s]
  58. INFO 03-08 07:39:03 [default_loader.py:291] Loading weights took 1.28 seconds
  59. INFO 03-08 07:39:03 [punica_selector.py:20] Using PunicaWrapperGPU.
  60. INFO 03-08 07:39:04 [gpu_model_runner.py:4130] Model loading took 8.96 GiB memory and 2.207885 seconds
  61. INFO 03-08 07:39:16 [backends.py:812] Using cache directory: /home/jovyan/.cache/vllm/torch_compile_cache/d772c4f9dd/rank_0_0/backbone for vLLM's torch.compile
  62. INFO 03-08 07:39:16 [backends.py:872] Dynamo bytecode transform time: 11.27 s
  63. Unsloth: Compiling kernels: 100%|██████████| 4/4 [00:01<00:00,  3.44it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_4]
  64. INFO 03-08 07:39:25 [backends.py:302] Cache the graph of compile range (1, 8192) for later use
  65.  
  66. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 13.42it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  67. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 1069.62it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  68. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 1124.91it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  69. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 1129.63it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  70. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 1131.02it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  71. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 1132.59it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  72. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 1072.91it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  73. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 786.19it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  74. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 972.51it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  75. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 953.10it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  76. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 972.67it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  77. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 964.40it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  78. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 947.99it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  79. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 965.13it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  80. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 987.66it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  81. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 969.72it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  82. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 971.58it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  83. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 979.19it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  84. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 966.78it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  85. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 970.17it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  86. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 955.55it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  87. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 952.57it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  88. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 920.24it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  89. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 959.26it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  90. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 960.49it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  91. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 936.83it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  92. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 974.64it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  93. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 972.83it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  94. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 958.79it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  95. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 940.40it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  96. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 913.68it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  97. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 963.73it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  98. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 956.61it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  99. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 963.13it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  100. Unsloth: Compiling kernels: 100%|██████████| 7/7 [00:00<00:00, 959.20it/s, triton_poi_fused__to_copy_add_index_select_mean_mul_pow_rsqrt_split_split_with_sizes_sub_unsqueeze_view_6]
  101. Unsloth: Compiling kernels: 100%|██████████| 3/3 [00:00<00:00, 18.02it/s, triton_red_fused__to_copy_add_mean_mul_pow_rsqrt_2]
  102. INFO 03-08 07:39:32 [backends.py:319] Compiling a graph for compile range (1, 8192) takes 10.56 s
  103. INFO 03-08 07:39:32 [monitor.py:34] torch.compile takes 21.83 s in total
  104.  
  105. ---------------------------------------------------------------------------
  106. RuntimeError                              Traceback (most recent call last)
  107. File /opt/conda/lib/python3.13/site-packages/unsloth_zoo/vllm_utils.py:2257, in load_vllm(model_name, config, gpu_memory_utilization, max_seq_length, dtype, training, float8_kv_cache, random_state, enable_lora, max_lora_rank, max_loras, use_async, use_engine, disable_log_stats, enforce_eager, enable_prefix_caching, compilation_config, conservativeness, max_logprobs, use_bitsandbytes, unsloth_vllm_standby, is_vision_model, return_args, max_num_seqs, fp8_mode)
  108.    2256 else:
  109. -> 2257     llm = LLM(**engine_args)
  110.    2258 pass
  111.  
  112. File /opt/conda/lib/python3.13/site-packages/vllm/entrypoints/llm.py:334, in LLM.__init__(self, model, runner, convert, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, allowed_local_media_path, allowed_media_domains, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, cpu_offload_gb, enforce_eager, enable_return_routed_experts, disable_custom_all_reduce, hf_token, hf_overrides, mm_processor_kwargs, pooler_config, structured_outputs_config, profiler_config, attention_config, kv_cache_memory_bytes, compilation_config, logits_processors, **kwargs)
  113.     332 log_non_default_args(engine_args)
  114. --> 334 self.llm_engine = LLMEngine.from_engine_args(
  115.     335     engine_args=engine_args, usage_context=UsageContext.LLM_CLASS
  116.     336 )
  117.     337 self.engine_class = type(self.llm_engine)
  118.  
  119. File /opt/conda/lib/python3.13/site-packages/vllm/v1/engine/llm_engine.py:172, in LLMEngine.from_engine_args(cls, engine_args, usage_context, stat_loggers, enable_multiprocessing)
  120.     171 # Create the LLMEngine.
  121. --> 172 return cls(
  122.     173     vllm_config=vllm_config,
  123.     174     executor_class=executor_class,
  124.     175     log_stats=not engine_args.disable_log_stats,
  125.     176     usage_context=usage_context,
  126.     177     stat_loggers=stat_loggers,
  127.     178     multiprocess_mode=enable_multiprocessing,
  128.     179 )
  129.  
  130. File /opt/conda/lib/python3.13/site-packages/vllm/v1/engine/llm_engine.py:106, in LLMEngine.__init__(self, vllm_config, executor_class, log_stats, aggregate_engine_logging, usage_context, stat_loggers, mm_registry, use_cached_outputs, multiprocess_mode)
  131.     105 # EngineCore (gets EngineCoreRequests and gives EngineCoreOutputs)
  132. --> 106 self.engine_core = EngineCoreClient.make_client(
  133.     107     multiprocess_mode=multiprocess_mode,
  134.     108     asyncio_mode=False,
  135.     109     vllm_config=vllm_config,
  136.     110     executor_class=executor_class,
  137.     111     log_stats=self.log_stats,
  138.     112 )
  139.     114 self.logger_manager: StatLoggerManager | None = None
  140.  
  141. File /opt/conda/lib/python3.13/site-packages/vllm/v1/engine/core_client.py:96, in EngineCoreClient.make_client(multiprocess_mode, asyncio_mode, vllm_config, executor_class, log_stats)
  142.      94     return SyncMPClient(vllm_config, executor_class, log_stats)
  143. ---> 96 return InprocClient(vllm_config, executor_class, log_stats)
  144.  
  145. File /opt/conda/lib/python3.13/site-packages/vllm/v1/engine/core_client.py:269, in InprocClient.__init__(self, *args, **kwargs)
  146.     268 def __init__(self, *args, **kwargs):
  147. --> 269     self.engine_core = EngineCore(*args, **kwargs)
  148.  
  149. File /opt/conda/lib/python3.13/site-packages/vllm/v1/engine/core.py:112, in EngineCore.__init__(self, vllm_config, executor_class, log_stats, executor_fail_callback, include_finished_set)
  150.     111 # Setup KV Caches and update CacheConfig after profiling.
  151. --> 112 num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
  152.     113     vllm_config
  153.     114 )
  154.     116 vllm_config.cache_config.num_gpu_blocks = num_gpu_blocks
  155.  
  156. File /opt/conda/lib/python3.13/site-packages/vllm/v1/engine/core.py:242, in EngineCore._initialize_kv_caches(self, vllm_config)
  157.     239 else:
  158.     240     # Profiles the peak memory usage of the model to determine how
  159.     241     # much memory can be allocated for kv cache.
  160. --> 242     available_gpu_memory = self.model_executor.determine_available_memory()
  161.     243     self.available_gpu_memory_for_kv_cache = available_gpu_memory[0]
  162.  
  163. File /opt/conda/lib/python3.13/site-packages/vllm/v1/executor/abstract.py:126, in Executor.determine_available_memory(self)
  164.     125 def determine_available_memory(self) -> list[int]:  # in bytes
  165. --> 126     return self.collective_rpc("determine_available_memory")
  166.  
  167. File /opt/conda/lib/python3.13/site-packages/vllm/v1/executor/uniproc_executor.py:75, in UniProcExecutor.collective_rpc(self, method, timeout, args, kwargs, non_block, single_value)
  168.      74 if not non_block:
  169. ---> 75     result = run_method(self.driver_worker, method, args, kwargs)
  170.      76     return result if single_value else [result]
  171.  
  172. File /opt/conda/lib/python3.13/site-packages/vllm/v1/serial_utils.py:461, in run_method(obj, method, args, kwargs)
  173.     460     func = partial(method, obj)  # type: ignore
  174. --> 461 return func(*args, **kwargs)
  175.  
  176. File /opt/conda/lib/python3.13/site-packages/torch/utils/_contextlib.py:120, in context_decorator.<locals>.decorate_context(*args, **kwargs)
  177.     119 with ctx_factory():
  178. --> 120     return func(*args, **kwargs)
  179.  
  180. File /opt/conda/lib/python3.13/site-packages/vllm/v1/worker/gpu_worker.py:322, in Worker.determine_available_memory(self)
  181.     318 with memory_profiling(
  182.     319     self.init_snapshot,
  183.     320     weights_memory=int(self.model_runner.model_memory_usage),
  184.     321 ) as profile_result:
  185. --> 322     self.model_runner.profile_run()
  186.     324 self.non_torch_memory = profile_result.non_torch_increase
  187.  
  188. File /opt/conda/lib/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py:4993, in GPUModelRunner.profile_run(self)
  189.    4992     else:
  190. -> 4993         output = self._dummy_sampler_run(last_hidden_states)
  191.    4994 else:
  192.  
  193. File /opt/conda/lib/python3.13/site-packages/torch/utils/_contextlib.py:120, in context_decorator.<locals>.decorate_context(*args, **kwargs)
  194.     119 with ctx_factory():
  195. --> 120     return func(*args, **kwargs)
  196.  
  197. File /opt/conda/lib/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py:4824, in GPUModelRunner._dummy_sampler_run(self, hidden_states)
  198.    4823     else:
  199. -> 4824         raise e
  200.    4825 if self.speculative_config:
  201.  
  202. File /opt/conda/lib/python3.13/site-packages/vllm/v1/worker/gpu_model_runner.py:4812, in GPUModelRunner._dummy_sampler_run(self, hidden_states)
  203.    4811 try:
  204. -> 4812     sampler_output = self.sampler(
  205.    4813         logits=logits, sampling_metadata=dummy_metadata
  206.    4814     )
  207.    4815 except RuntimeError as e:
  208.  
  209. File /opt/conda/lib/python3.13/site-packages/torch/nn/modules/module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
  210.    1774 else:
  211. -> 1775     return self._call_impl(*args, **kwargs)
  212.  
  213. File /opt/conda/lib/python3.13/site-packages/torch/nn/modules/module.py:1786, in Module._call_impl(self, *args, **kwargs)
  214.    1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
  215.    1784         or _global_backward_pre_hooks or _global_backward_hooks
  216.    1785         or _global_forward_hooks or _global_forward_pre_hooks):
  217. -> 1786     return forward_call(*args, **kwargs)
  218.    1788 result = None
  219.  
  220. File /opt/conda/lib/python3.13/site-packages/vllm/v1/sample/sampler.py:96, in Sampler.forward(self, logits, sampling_metadata, predict_bonus_token, logprobs_mode_override)
  221.      95 # Sample the next token.
  222. ---> 96 sampled, processed_logprobs = self.sample(logits, sampling_metadata)
  223.      97 if processed_logprobs is not None:
  224.  
  225. File /opt/conda/lib/python3.13/site-packages/vllm/v1/sample/sampler.py:187, in Sampler.sample(self, logits, sampling_metadata, logprobs_mode_override)
  226.     186 # Apply top_k and/or top_p.
  227. --> 187 random_sampled, processed_logprobs = self.topk_topp_sampler(
  228.     188     logits,
  229.     189     sampling_metadata.generators,
  230.     190     sampling_metadata.top_k,
  231.     191     sampling_metadata.top_p,
  232.     192 )
  233.     194 if greedy_sampled is None:
  234.  
  235. File /opt/conda/lib/python3.13/site-packages/torch/nn/modules/module.py:1775, in Module._wrapped_call_impl(self, *args, **kwargs)
  236.    1774 else:
  237. -> 1775     return self._call_impl(*args, **kwargs)
  238.  
  239. File /opt/conda/lib/python3.13/site-packages/torch/nn/modules/module.py:1786, in Module._call_impl(self, *args, **kwargs)
  240.    1783 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
  241.    1784         or _global_backward_pre_hooks or _global_backward_hooks
  242.    1785         or _global_forward_hooks or _global_forward_pre_hooks):
  243. -> 1786     return forward_call(*args, **kwargs)
  244.    1788 result = None
  245.  
  246. File /opt/conda/lib/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py:138, in TopKTopPSampler.forward_cuda(self, logits, generators, k, p)
  247.     135 # flashinfer sampling functions expect contiguous logits.
  248.     136 # In flex_attn/triton_attn fp32 inference, logits can be non-contiguous
  249.     137 # because of slicing operation in logits_processor.
  250. --> 138 return flashinfer_sample(logits.contiguous(), k, p, generators), None
  251.  
  252. File /opt/conda/lib/python3.13/site-packages/vllm/v1/sample/ops/topk_topp_sampler.py:377, in flashinfer_sample(logits, k, p, generators)
  253.     375 else:
  254.     376     # Both top-k and top-p.
  255. --> 377     next_token_ids = flashinfer.sampling.top_k_top_p_sampling_from_logits(
  256.     378         logits, k, p, deterministic=True
  257.     379     )
  258.     381 return next_token_ids.view(-1)
  259.  
  260. File /opt/conda/lib/python3.13/site-packages/flashinfer/sampling.py:1128, in top_k_top_p_sampling_from_logits(logits, top_k, top_p, indices, filter_apply_order, deterministic, generator, check_nan, seed, offset)
  261.    1127 if filter_apply_order == "top_k_first":
  262. -> 1128     masked_logits = top_k_mask_logits(logits, top_k)
  263.    1129     probs = torch.softmax(masked_logits, dim=-1)
  264.  
  265. File /opt/conda/lib/python3.13/site-packages/flashinfer/sampling.py:1495, in top_k_mask_logits(logits, top_k)
  266.    1492 # Note: row_states_buffer is zero-initialized on first allocation by _get_cache_buf
  267.    1493 # Kernel will reset arrival_counter to 0 at the end of each launch
  268. -> 1495 return get_sampling_module().top_k_mask_logits(
  269.    1496     logits, *_to_tensor_scalar_tuple(top_k), row_states_buffer
  270.    1497 )
  271.  
  272. File /opt/conda/lib/python3.13/site-packages/flashinfer/sampling.py:54, in get_sampling_module()
  273.      52 @functools.cache
  274.      53 def get_sampling_module():
  275. ---> 54     module = gen_sampling_module().build_and_load()
  276.      56     @register_custom_op("flashinfer::softmax", mutates_args=("workspace_buffer",))
  277.      57     def softmax(
  278.      58         workspace_buffer: torch.Tensor,
  279.    (...)     62         enable_pdl: bool,
  280.      63     ) -> torch.Tensor:
  281.  
  282. File /opt/conda/lib/python3.13/site-packages/flashinfer/jit/core.py:316, in JitSpec.build_and_load(self)
  283.     315 verbose = os.environ.get("FLASHINFER_JIT_VERBOSE", "0") == "1"
  284. --> 316 self.build(verbose, need_lock=False)
  285.     317 result = self.load(so_path)
  286.  
  287. File /opt/conda/lib/python3.13/site-packages/flashinfer/jit/core.py:301, in JitSpec.build(self, verbose, need_lock)
  288.     300 if not self.is_ninja_generated:
  289. --> 301     self.write_ninja()
  290.     302 run_ninja(self.build_dir, self.ninja_path, verbose)
  291.  
  292. File /opt/conda/lib/python3.13/site-packages/flashinfer/jit/core.py:272, in JitSpec.write_ninja(self)
  293.     271 self.build_dir.mkdir(parents=True, exist_ok=True)
  294. --> 272 content = generate_ninja_build_for_op(
  295.     273     name=self.name,
  296.     274     sources=self.sources,
  297.     275     extra_cflags=self.extra_cflags,
  298.     276     extra_cuda_cflags=self.extra_cuda_cflags,
  299.     277     extra_ldflags=self.extra_ldflags,
  300.     278     extra_include_dirs=self.extra_include_dirs,
  301.     279     needs_device_linking=self.needs_device_linking,
  302.     280 )
  303.     281 write_if_different(ninja_path, content)
  304.  
  305. File /opt/conda/lib/python3.13/site-packages/flashinfer/jit/cpp_ext.py:212, in generate_ninja_build_for_op(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_dirs, needs_device_linking)
  306.     203 def generate_ninja_build_for_op(
  307.     204     name: str,
  308.     205     sources: List[Path],
  309.    (...)    210     needs_device_linking: bool = False,
  310.     211 ) -> str:
  311. --> 212     cuda_home = get_cuda_path()
  312.     213     common_cflags = build_common_cflags(cuda_home, extra_include_dirs)
  313.  
  314. File /opt/conda/lib/python3.13/site-packages/flashinfer/jit/cpp_ext.py:57, in get_cuda_path()
  315.      56     if not os.path.exists(cuda_home):
  316. ---> 57         raise RuntimeError(
  317.      58             f"Could not find nvcc and default {cuda_home=} doesn't exist"
  318.      59         )
  319.      60 return cuda_home
  320.  
  321. RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist
  322.  
  323. During handling of the above exception, another exception occurred:
  324.  
  325. RuntimeError                              Traceback (most recent call last)
  326. Cell In[5], line 3
  327.      1 from unsloth import FastLanguageModel
  328. ----> 3 model, tokenizer = FastLanguageModel.from_pretrained(
  329.      4     model_name=MODEL_NAME,
  330.      5     max_seq_length=MAX_SEQ_LENGTH,
  331.      6     load_in_4bit=False,   # False = 16-bit LoRA (not 4-bit)
  332.      7     fast_inference=True,  # enable vLLM as the GRPO rollout engine
  333.      8     max_lora_rank=LORA_RANK,  # pre-allocate vLLM memory for LoRA
  334.      9     load_in_fp8=True,     # Float8 training — the key FP8 flag
  335.     10 )
  336.     11 print('Base model loaded.')
  337.  
  338. File /opt/conda/lib/python3.13/site-packages/unsloth/models/loader.py:704, in FastLanguageModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, load_in_8bit, load_in_16bit, full_finetuning, token, device_map, rope_scaling, fix_tokenizer, trust_remote_code, use_gradient_checkpointing, resize_model_vocab, revision, use_exact_model_name, offload_embedding, float32_mixed_precision, fast_inference, gpu_memory_utilization, float8_kv_cache, random_state, max_lora_rank, disable_log_stats, qat_scheme, load_in_fp8, unsloth_tiled_mlp, *args, **kwargs)
  339.    701     load_in_4bit_kwargs = False
  340.    702     load_in_8bit_kwargs = False
  341. --> 704 model, tokenizer = dispatch_model.from_pretrained(
  342.    705     model_name = model_name,
  343.    706     max_seq_length = max_seq_length,
  344.    707     dtype = _get_dtype(dtype),
  345.    708     load_in_4bit = load_in_4bit_kwargs,
  346.    709     token = token,
  347.    710     device_map = device_map,
  348.    711     rope_scaling = rope_scaling,
  349.    712     fix_tokenizer = fix_tokenizer,
  350.    713     model_patcher = dispatch_model,
  351.    714     tokenizer_name = tokenizer_name,
  352.    715     trust_remote_code = trust_remote_code,
  353.    716     revision = revision if not is_peft else None,
  354.    717     fast_inference = fast_inference,
  355.    718     gpu_memory_utilization = gpu_memory_utilization,
  356.    719     float8_kv_cache = float8_kv_cache,
  357.    720     random_state = random_state,
  358.    721     max_lora_rank = max_lora_rank,
  359.    722     disable_log_stats = disable_log_stats,
  360.    723     load_in_fp8 = load_in_fp8,
  361.    724     *args,
  362.    725     **kwargs,
  363.    726 )
  364.    728 if resize_model_vocab is not None:
  365.    729     model.resize_token_embeddings(resize_model_vocab)
  366.  
  367. File /opt/conda/lib/python3.13/site-packages/unsloth/models/qwen3.py:462, in FastQwen3Model.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, model_patcher, tokenizer_name, trust_remote_code, **kwargs)
  368.    447 @staticmethod
  369.    448 def from_pretrained(  # TODO: Change after release
  370.    449     model_name = "Qwen/Qwen3-7B",
  371.   (...)    460     **kwargs,
  372.    461 ):
  373. --> 462     return FastLlamaModel.from_pretrained(
  374.    463         model_name = model_name,
  375.    464         max_seq_length = max_seq_length,
  376.    465         dtype = dtype,
  377.    466         load_in_4bit = load_in_4bit,
  378.    467         token = token,
  379.    468         device_map = device_map,
  380.    469         rope_scaling = rope_scaling,
  381.    470         fix_tokenizer = fix_tokenizer,
  382.    471         model_patcher = FastQwen3Model,
  383.    472         tokenizer_name = tokenizer_name,
  384.    473         trust_remote_code = trust_remote_code,
  385.    474         **kwargs,
  386.    475     )
  387.  
  388. File /opt/conda/lib/python3.13/site-packages/unsloth/models/llama.py:2479, in FastLlamaModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, model_patcher, tokenizer_name, trust_remote_code, revision, fast_inference, gpu_memory_utilization, float8_kv_cache, random_state, max_lora_rank, disable_log_stats, unsloth_vllm_standby, num_labels, qat_scheme, load_in_fp8, **kwargs)
  389.   2476 pass
  390.   2478 # Load vLLM first
  391. -> 2479 llm = load_vllm(**load_vllm_kwargs)
  392.   2481 # Convert to HF format
  393.   2482 _, quant_state_dict = get_vllm_state_dict(
  394.   2483     llm,
  395.   2484     config = model_config,
  396.   2485     load_in_fp8 = load_in_fp8,
  397.   2486 )
  398.  
  399. File /opt/conda/lib/python3.13/site-packages/unsloth_zoo/vllm_utils.py:2281, in load_vllm(model_name, config, gpu_memory_utilization, max_seq_length, dtype, training, float8_kv_cache, random_state, enable_lora, max_lora_rank, max_loras, use_async, use_engine, disable_log_stats, enforce_eager, enable_prefix_caching, compilation_config, conservativeness, max_logprobs, use_bitsandbytes, unsloth_vllm_standby, is_vision_model, return_args, max_num_seqs, fp8_mode)
  400.   2271     if unsloth_vllm_standby and ("memory" in error.lower() or "alloc" in error.lower()):
  401.   2272         raise MemoryError(
  402.   2273             f"Unsloth: Your GPU ran out of memory loading vLLM with standby mode enabled.\n"
  403.   2274             f"Your GPU has {total_gb:.1f} GB VRAM with gpu_memory_utilization={gpu_memory_utilization:.3f}.\n"
  404.   (...)   2279             f"Original error: {error}"
  405.   2280         )
  406. -> 2281     raise RuntimeError(error)
  407.   2283 if "gpu_memory_utilization" in error or "memory" in error:
  408.   2284     approx_max_num_seqs = max(int(approx_max_num_seqs * 0.75), 1)
  409.  
  410. RuntimeError: Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist
  411.  
Tags: RL jupyter Conda llama QWEN northflank h100 unsloth grpo
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!