API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jan 24th, 2025
1,838
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 13.29 KB | None | 0 0
raw download clone embed print report
  1.  
  2. ```
  3. 7 experts at 2bit are equivalent to 8 GB worth model weights.
  4.  
  5.  
  6. I use an NVMe SSD (western digital sn850x 2TB) the bandwidth of my SSD is 7.3 GB/s
  7.  
  8. When I run the model, I believe the shared expert is run in RAM, leaving only 7 out of 8 active experts loading from SSD each forward pass. (37B - > 32B)
  9.  
  10. The model is 2bit: (IQ2XXS the deepseek r1 model is specifically 2.08 BPW) and this leaves us with 32÷4=8GB worth of active parameter weights.
  11.  
  12. This is why I can get 1 t/s without swap. I have removed and disabled swapfile entirely. (htop display is 0k/0k) nothing is loaded into vram, I built without cuda support.
  13. ```
  14.  
  15.  
  16. bin/llama-cli -m /home/user/Storage/DeepSeek-R1-IQ2_XXS-00001-of-00005.gguf -p "here's an intresting experiment:"
  17. build: 4447 (f7cd1330) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
  18. main: llama backend init
  19. main: load the model and apply lora adapter, if any
  20. llama_model_loader: additional 4 GGUFs metadata loaded.
  21. llama_model_loader: loaded meta data with 51 key-value pairs and 1025 tensors from /home/user/Storage/DeepSeek-R1-IQ2_XXS-00001-of-00005.gguf (version GGUF V3 (latest))
  22. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  23. llama_model_loader: - kv 0: general.architecture str = deepseek2
  24. llama_model_loader: - kv 1: general.type str = model
  25. llama_model_loader: - kv 2: general.name str = DeepSeek R1
  26. llama_model_loader: - kv 3: general.size_label str = 256x20B
  27. llama_model_loader: - kv 4: general.tags arr[str,1] = ["text-generation"]
  28. llama_model_loader: - kv 5: deepseek2.block_count u32 = 61
  29. llama_model_loader: - kv 6: deepseek2.context_length u32 = 163840
  30. llama_model_loader: - kv 7: deepseek2.embedding_length u32 = 7168
  31. llama_model_loader: - kv 8: deepseek2.feed_forward_length u32 = 18432
  32. llama_model_loader: - kv 9: deepseek2.attention.head_count u32 = 128
  33. llama_model_loader: - kv 10: deepseek2.attention.head_count_kv u32 = 128
  34. llama_model_loader: - kv 11: deepseek2.rope.freq_base f32 = 10000.000000
  35. llama_model_loader: - kv 12: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
  36. llama_model_loader: - kv 13: deepseek2.expert_used_count u32 = 8
  37. llama_model_loader: - kv 14: deepseek2.leading_dense_block_count u32 = 3
  38. llama_model_loader: - kv 15: deepseek2.vocab_size u32 = 129280
  39. llama_model_loader: - kv 16: deepseek2.attention.q_lora_rank u32 = 1536
  40. llama_model_loader: - kv 17: deepseek2.attention.kv_lora_rank u32 = 512
  41. llama_model_loader: - kv 18: deepseek2.attention.key_length u32 = 192
  42. llama_model_loader: - kv 19: deepseek2.attention.value_length u32 = 128
  43. llama_model_loader: - kv 20: deepseek2.expert_feed_forward_length u32 = 2048
  44. llama_model_loader: - kv 21: deepseek2.expert_count u32 = 256
  45. llama_model_loader: - kv 22: deepseek2.expert_shared_count u32 = 1
  46. llama_model_loader: - kv 23: deepseek2.expert_weights_scale f32 = 2.500000
  47. llama_model_loader: - kv 24: deepseek2.expert_weights_norm bool = true
  48. llama_model_loader: - kv 25: deepseek2.expert_gating_func u32 = 2
  49. llama_model_loader: - kv 26: deepseek2.rope.dimension_count u32 = 64
  50. llama_model_loader: - kv 27: deepseek2.rope.scaling.type str = yarn
  51. llama_model_loader: - kv 28: deepseek2.rope.scaling.factor f32 = 40.000000
  52. llama_model_loader: - kv 29: deepseek2.rope.scaling.original_context_length u32 = 4096
  53. llama_model_loader: - kv 30: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
  54. llama_model_loader: - kv 31: tokenizer.ggml.model str = gpt2
  55. llama_model_loader: - kv 32: tokenizer.ggml.pre str = deepseek-v3
  56. llama_model_loader: - kv 33: tokenizer.ggml.tokens arr[str,129280] = ["<｜begin▁of▁sentence｜>", "<�...
  57. llama_model_loader: - kv 34: tokenizer.ggml.token_type arr[i32,129280] = [3, 3, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  58. llama_model_loader: - kv 35: tokenizer.ggml.merges arr[str,127741] = ["Ġ t", "Ġ a", "i n", "Ġ Ġ", "h e...
  59. llama_model_loader: - kv 36: tokenizer.ggml.bos_token_id u32 = 0
  60. llama_model_loader: - kv 37: tokenizer.ggml.eos_token_id u32 = 1
  61. llama_model_loader: - kv 38: tokenizer.ggml.padding_token_id u32 = 1
  62. llama_model_loader: - kv 39: tokenizer.ggml.add_bos_token bool = true
  63. llama_model_loader: - kv 40: tokenizer.ggml.add_eos_token bool = false
  64. llama_model_loader: - kv 41: tokenizer.chat_template str = {% if not add_generation_prompt is de...
  65. llama_model_loader: - kv 42: general.quantization_version u32 = 2
  66. llama_model_loader: - kv 43: general.file_type u32 = 19
  67. llama_model_loader: - kv 44: quantize.imatrix.file str = /models_out/DeepSeek-R1-GGUF/DeepSeek...
  68. llama_model_loader: - kv 45: quantize.imatrix.dataset str = /training_data/calibration_datav3.txt
  69. llama_model_loader: - kv 46: quantize.imatrix.entries_count i32 = 720
  70. llama_model_loader: - kv 47: quantize.imatrix.chunks_count i32 = 124
  71. llama_model_loader: - kv 48: split.no u16 = 0
  72. llama_model_loader: - kv 49: split.tensors.count i32 = 1025
  73. llama_model_loader: - kv 50: split.count u16 = 5
  74. llama_model_loader: - type f32: 361 tensors
  75. llama_model_loader: - type q2_K: 8 tensors
  76. llama_model_loader: - type q5_K: 1 tensors
  77. llama_model_loader: - type iq2_xxs: 655 tensors
  78. llm_load_vocab: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
  79. llm_load_vocab: special tokens cache size = 818
  80. llm_load_vocab: token to piece cache size = 0.8223 MB
  81. llm_load_print_meta: format = GGUF V3 (latest)
  82. llm_load_print_meta: arch = deepseek2
  83. llm_load_print_meta: vocab type = BPE
  84. llm_load_print_meta: n_vocab = 129280
  85. llm_load_print_meta: n_merges = 127741
  86. llm_load_print_meta: vocab_only = 0
  87. llm_load_print_meta: n_ctx_train = 163840
  88. llm_load_print_meta: n_embd = 7168
  89. llm_load_print_meta: n_layer = 61
  90. llm_load_print_meta: n_head = 128
  91. llm_load_print_meta: n_head_kv = 128
  92. llm_load_print_meta: n_rot = 64
  93. llm_load_print_meta: n_swa = 0
  94. llm_load_print_meta: n_embd_head_k = 192
  95. llm_load_print_meta: n_embd_head_v = 128
  96. llm_load_print_meta: n_gqa = 1
  97. llm_load_print_meta: n_embd_k_gqa = 24576
  98. llm_load_print_meta: n_embd_v_gqa = 16384
  99. llm_load_print_meta: f_norm_eps = 0.0e+00
  100. llm_load_print_meta: f_norm_rms_eps = 1.0e-06
  101. llm_load_print_meta: f_clamp_kqv = 0.0e+00
  102. llm_load_print_meta: f_max_alibi_bias = 0.0e+00
  103. llm_load_print_meta: f_logit_scale = 0.0e+00
  104. llm_load_print_meta: n_ff = 18432
  105. llm_load_print_meta: n_expert = 256
  106. llm_load_print_meta: n_expert_used = 8
  107. llm_load_print_meta: causal attn = 1
  108. llm_load_print_meta: pooling type = 0
  109. llm_load_print_meta: rope type = 0
  110. llm_load_print_meta: rope scaling = yarn
  111. llm_load_print_meta: freq_base_train = 10000.0
  112. llm_load_print_meta: freq_scale_train = 0.025
  113. llm_load_print_meta: n_ctx_orig_yarn = 4096
  114. llm_load_print_meta: rope_finetuned = unknown
  115. llm_load_print_meta: ssm_d_conv = 0
  116. llm_load_print_meta: ssm_d_inner = 0
  117. llm_load_print_meta: ssm_d_state = 0
  118. llm_load_print_meta: ssm_dt_rank = 0
  119. llm_load_print_meta: ssm_dt_b_c_rms = 0
  120. llm_load_print_meta: model type = 671B
  121. llm_load_print_meta: model ftype = IQ2_XXS - 2.0625 bpw
  122. llm_load_print_meta: model params = 671.03 B
  123. llm_load_print_meta: model size = 162.44 GiB (2.08 BPW)
  124. llm_load_print_meta: general.name = DeepSeek R1
  125. llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>'
  126. llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>'
  127. llm_load_print_meta: EOT token = 1 '<｜end▁of▁sentence｜>'
  128. llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>'
  129. llm_load_print_meta: LF token = 131 'Ä'
  130. llm_load_print_meta: FIM PRE token = 128801 '<｜fim▁begin｜>'
  131. llm_load_print_meta: FIM SUF token = 128800 '<｜fim▁hole｜>'
  132. llm_load_print_meta: FIM MID token = 128802 '<｜fim▁end｜>'
  133. llm_load_print_meta: EOG token = 1 '<｜end▁of▁sentence｜>'
  134. llm_load_print_meta: max token length = 256
  135. llm_load_print_meta: n_layer_dense_lead = 3
  136. llm_load_print_meta: n_lora_q = 1536
  137. llm_load_print_meta: n_lora_kv = 512
  138. llm_load_print_meta: n_ff_exp = 2048
  139. llm_load_print_meta: n_expert_shared = 1
  140. llm_load_print_meta: expert_weights_scale = 2.5
  141. llm_load_print_meta: expert_weights_norm = 1
  142. llm_load_print_meta: expert_gating_func = sigmoid
  143. llm_load_print_meta: rope_yarn_log_mul = 0.1000
  144. llm_load_tensors: CPU_Mapped model buffer size = 37799.72 MiB
  145. llm_load_tensors: CPU_Mapped model buffer size = 37840.28 MiB
  146. llm_load_tensors: CPU_Mapped model buffer size = 37794.24 MiB
  147. llm_load_tensors: CPU_Mapped model buffer size = 37801.27 MiB
  148. llm_load_tensors: CPU_Mapped model buffer size = 15107.08 MiB
  149. ....................................................................................................
  150. llama_new_context_with_model: n_seq_max = 1
  151. llama_new_context_with_model: n_ctx = 4096
  152. llama_new_context_with_model: n_ctx_per_seq = 4096
  153. llama_new_context_with_model: n_batch = 2048
  154. llama_new_context_with_model: n_ubatch = 512
  155. llama_new_context_with_model: flash_attn = 0
  156. llama_new_context_with_model: freq_base = 10000.0
  157. llama_new_context_with_model: freq_scale = 0.025
  158. llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (163840) -- the full capacity of the model will not be utilized
  159. llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 61, can_shift = 0
  160. llama_kv_cache_init: CPU KV buffer size = 19520.00 MiB
  161. llama_new_context_with_model: KV self size = 19520.00 MiB, K (f16): 11712.00 MiB, V (f16): 7808.00 MiB
  162. llama_new_context_with_model: CPU output buffer size = 0.49 MiB
  163. llama_new_context_with_model: CPU compute buffer size = 1186.01 MiB
  164. llama_new_context_with_model: graph nodes = 5025
  165. llama_new_context_with_model: graph splits = 1
  166. common_init_from_params: KV cache shifting is not supported for this model, disabling KV cache shifting
  167. common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
  168. common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
  169. main: llama threadpool init, n_threads = 16
  170.  
  171. system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
  172.  
  173. sampler seed: 3369376391
  174. sampler params:
  175. repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
  176. dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
  177. top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, temp = 0.800
  178. mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
  179. sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
  180. generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 1
  181.  
  182. here's an intresting experiment: let's take a small sample of the dataset and check what's inside. Let's start with the first example.
  183.  
  184. In [1]:
  185. from __future__ import print_function
  186.  
  187. import json
  188.  
  189. with open("data/sherlock_holmes_1.json") as f:
  190. first_example = json.load(f)
  191.  
  192. print(first_example.keys())
  193. print()
  194. print("Title:", first
  195. llama_perf_sampler_print: sampling time = 8.98 ms / 86 runs ( 0.10 ms per token, 9577.90 tokens per second)
  196. llama_perf_context_print: load time = 87451.86 ms
  197. llama_perf_context_print: prompt eval time = 8725.74 ms / 9 tokens ( 969.53 ms per token, 1.03 tokens per second)
  198. llama_perf_context_print: eval time = 74440.10 ms / 76 runs ( 979.48 ms per token, 1.02 tokens per second)
  199. llama_perf_context_print: total time = 83671.35 ms / 85 tokens
  200.  
  201.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!