API tools faq
paste
machinez

Llama.cpp server Qwen2.5 7b

machinez
Jan 19th, 2025
27
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 10.59 KB | None | 0 0
raw download clone embed print report
  1. $ ./build/bin/llama-server -m ~/models/Qwen2.5-7B-Instruct-Q8_0.gguf -n -2 -e -ngl 33 -t 4 -c 4096 --host 0.0.0.0
  2. ggml_vulkan: Found 1 Vulkan devices:
  3. ggml_vulkan: 0 = AMD Radeon Graphics (RADV NAVI10) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: none
  4. build: 4510 (b9daaffe) with cc (GCC) 14.0.1 20240411 (Red Hat 14.0.1-0) for x86_64-redhat-linux
  5. system info: n_threads = 4, n_threads_batch = 4, total_threads = 12
  6.  
  7. system_info: n_threads = 4 (n_threads_batch = 4) / 12 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
  8.  
  9. main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 11
  10. main: loading model
  11. srv load_model: loading model '/home/user/models/Qwen2.5-7B-Instruct-Q8_0.gguf'
  12. llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon Graphics (RADV NAVI10)) - 10240 MiB free
  13. llama_model_loader: loaded meta data with 38 key-value pairs and 339 tensors from /home/user/models/Qwen2.5-7B-Instruct-Q8_0.gguf (version GGUF V3 (latest))
  14. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  15. llama_model_loader: - kv 0: general.architecture str = qwen2
  16. llama_model_loader: - kv 1: general.type str = model
  17. llama_model_loader: - kv 2: general.name str = Qwen2.5 7B Instruct
  18. llama_model_loader: - kv 3: general.finetune str = Instruct
  19. llama_model_loader: - kv 4: general.basename str = Qwen2.5
  20. llama_model_loader: - kv 5: general.size_label str = 7B
  21. llama_model_loader: - kv 6: general.license str = apache-2.0
  22. llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/Qwen/Qwen2.5-7...
  23. llama_model_loader: - kv 8: general.base_model.count u32 = 1
  24. llama_model_loader: - kv 9: general.base_model.0.name str = Qwen2.5 7B
  25. llama_model_loader: - kv 10: general.base_model.0.organization str = Qwen
  26. llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/Qwen/Qwen2.5-7B
  27. llama_model_loader: - kv 12: general.tags arr[str,2] = ["chat", "text-generation"]
  28. llama_model_loader: - kv 13: general.languages arr[str,1] = ["en"]
  29. llama_model_loader: - kv 14: qwen2.block_count u32 = 28
  30. llama_model_loader: - kv 15: qwen2.context_length u32 = 32768
  31. llama_model_loader: - kv 16: qwen2.embedding_length u32 = 3584
  32. llama_model_loader: - kv 17: qwen2.feed_forward_length u32 = 18944
  33. llama_model_loader: - kv 18: qwen2.attention.head_count u32 = 28
  34. llama_model_loader: - kv 19: qwen2.attention.head_count_kv u32 = 4
  35. llama_model_loader: - kv 20: qwen2.rope.freq_base f32 = 1000000.000000
  36. llama_model_loader: - kv 21: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001
  37. llama_model_loader: - kv 22: general.file_type u32 = 7
  38. llama_model_loader: - kv 23: tokenizer.ggml.model str = gpt2
  39. llama_model_loader: - kv 24: tokenizer.ggml.pre str = qwen2
  40. llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ...
  41. llama_model_loader: - kv 26: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  42. llama_model_loader: - kv 27: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
  43. llama_model_loader: - kv 28: tokenizer.ggml.eos_token_id u32 = 151645
  44. llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 151643
  45. llama_model_loader: - kv 30: tokenizer.ggml.bos_token_id u32 = 151643
  46. llama_model_loader: - kv 31: tokenizer.ggml.add_bos_token bool = false
  47. llama_model_loader: - kv 32: tokenizer.chat_template str = {%- if tools %}\n {{- '<|im_start|>...
  48. llama_model_loader: - kv 33: general.quantization_version u32 = 2
  49. llama_model_loader: - kv 34: quantize.imatrix.file str = /models_out/Qwen2.5-7B-Instruct-GGUF/...
  50. llama_model_loader: - kv 35: quantize.imatrix.dataset str = /training_dir/calibration_datav3.txt
  51. llama_model_loader: - kv 36: quantize.imatrix.entries_count i32 = 196
  52. llama_model_loader: - kv 37: quantize.imatrix.chunks_count i32 = 128
  53. llama_model_loader: - type f32: 141 tensors
  54. llama_model_loader: - type q8_0: 198 tensors
  55. print_info: file format = GGUF V3 (latest)
  56. print_info: file type = Q8_0
  57. print_info: file size = 7.54 GiB (8.50 BPW)
  58. load: special tokens cache size = 22
  59. load: token to piece cache size = 0.9310 MB
  60. print_info: arch = qwen2
  61. print_info: vocab_only = 0
  62. print_info: n_ctx_train = 32768
  63. print_info: n_embd = 3584
  64. print_info: n_layer = 28
  65. print_info: n_head = 28
  66. print_info: n_head_kv = 4
  67. print_info: n_rot = 128
  68. print_info: n_swa = 0
  69. print_info: n_embd_head_k = 128
  70. print_info: n_embd_head_v = 128
  71. print_info: n_gqa = 7
  72. print_info: n_embd_k_gqa = 512
  73. print_info: n_embd_v_gqa = 512
  74. print_info: f_norm_eps = 0.0e+00
  75. print_info: f_norm_rms_eps = 1.0e-06
  76. print_info: f_clamp_kqv = 0.0e+00
  77. print_info: f_max_alibi_bias = 0.0e+00
  78. print_info: f_logit_scale = 0.0e+00
  79. print_info: n_ff = 18944
  80. print_info: n_expert = 0
  81. print_info: n_expert_used = 0
  82. print_info: causal attn = 1
  83. print_info: pooling type = 0
  84. print_info: rope type = 2
  85. print_info: rope scaling = linear
  86. print_info: freq_base_train = 1000000.0
  87. print_info: freq_scale_train = 1
  88. print_info: n_ctx_orig_yarn = 32768
  89. print_info: rope_finetuned = unknown
  90. print_info: ssm_d_conv = 0
  91. print_info: ssm_d_inner = 0
  92. print_info: ssm_d_state = 0
  93. print_info: ssm_dt_rank = 0
  94. print_info: ssm_dt_b_c_rms = 0
  95. print_info: model type = 7B
  96. print_info: model params = 7.62 B
  97. print_info: general.name = Qwen2.5 7B Instruct
  98. print_info: vocab type = BPE
  99. print_info: n_vocab = 152064
  100. print_info: n_merges = 151387
  101. print_info: BOS token = 151643 '<|endoftext|>'
  102. print_info: EOS token = 151645 '<|im_end|>'
  103. print_info: EOT token = 151645 '<|im_end|>'
  104. print_info: PAD token = 151643 '<|endoftext|>'
  105. print_info: LF token = 148848 'ÄĬ'
  106. print_info: FIM PRE token = 151659 '<|fim_prefix|>'
  107. print_info: FIM SUF token = 151661 '<|fim_suffix|>'
  108. print_info: FIM MID token = 151660 '<|fim_middle|>'
  109. print_info: FIM PAD token = 151662 '<|fim_pad|>'
  110. print_info: FIM REP token = 151663 '<|repo_name|>'
  111. print_info: FIM SEP token = 151664 '<|file_sep|>'
  112. print_info: EOG token = 151643 '<|endoftext|>'
  113. print_info: EOG token = 151645 '<|im_end|>'
  114. print_info: EOG token = 151662 '<|fim_pad|>'
  115. print_info: EOG token = 151663 '<|repo_name|>'
  116. print_info: EOG token = 151664 '<|file_sep|>'
  117. print_info: max token length = 256
  118. ggml_vulkan: Compiling shaders.............................................Done!
  119. load_tensors: offloading 28 repeating layers to GPU
  120. load_tensors: offloading output layer to GPU
  121. load_tensors: offloaded 29/29 layers to GPU
  122. load_tensors: Vulkan0 model buffer size = 7165.44 MiB
  123. load_tensors: CPU_Mapped model buffer size = 552.23 MiB
  124. llama_init_from_model: n_seq_max = 1
  125. llama_init_from_model: n_ctx = 4096
  126. llama_init_from_model: n_ctx_per_seq = 4096
  127. llama_init_from_model: n_batch = 2048
  128. llama_init_from_model: n_ubatch = 512
  129. llama_init_from_model: flash_attn = 0
  130. llama_init_from_model: freq_base = 1000000.0
  131. llama_init_from_model: freq_scale = 1
  132. llama_init_from_model: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
  133. llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 28, can_shift = 1
  134. llama_kv_cache_init: Vulkan0 KV buffer size = 224.00 MiB
  135. llama_init_from_model: KV self size = 224.00 MiB, K (f16): 112.00 MiB, V (f16): 112.00 MiB
  136. llama_init_from_model: Vulkan_Host output buffer size = 0.58 MiB
  137. llama_init_from_model: Vulkan0 compute buffer size = 304.00 MiB
  138. llama_init_from_model: Vulkan_Host compute buffer size = 15.01 MiB
  139. llama_init_from_model: graph nodes = 986
  140. llama_init_from_model: graph splits = 2
  141. common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
  142. common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
  143. srv init: initializing slots, n_slots = 1
  144. slot init: id 0 | task -1 | new slot n_ctx_slot = 4096
  145. main: model loaded
  146. main: chat template, chat_template: (built-in), example_format: '<|im_start|>system
  147. You are a helpful assistant<|im_end|>
  148. <|im_start|>user
  149. Hello<|im_end|>
  150. <|im_start|>assistant
  151. Hi there<|im_end|>
  152. <|im_start|>user
  153. How are you?<|im_end|>
  154. <|im_start|>assistant
  155. '
  156. main: server is listening on http://0.0.0.0:8080 - starting the main loop
  157. srv update_slots: all slots are idle
  158. slot launch_slot_: id 0 | task 0 | processing task
  159. slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 33
  160. slot update_slots: id 0 | task 0 | kv cache rm [0, end)
  161. slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 33, n_tokens = 33, progress = 1.000000
  162. slot update_slots: id 0 | task 0 | prompt done, n_past = 33, n_tokens = 33
  163. slot release: id 0 | task 0 | stop processing: n_past = 515, truncated = 0
  164. slot print_timing: id 0 | task 0 |
  165. prompt eval time = 329.50 ms / 33 tokens ( 9.98 ms per token, 100.15 tokens per second)
  166. eval time = 14331.91 ms / 483 tokens ( 29.67 ms per token, 33.70 tokens per second)
  167. total time = 14661.41 ms / 516 tokens
  168. srv update_slots: all slots are idle
Tags: llama.cpp QWEN
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!