Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
- llama_model_loader: - kv 0: general.architecture str = command-r
- llama_model_loader: - kv 1: general.name str = c4ai-command-r-v01
- llama_model_loader: - kv 2: command-r.block_count u32 = 40
- llama_model_loader: - kv 3: command-r.context_length u32 = 131072
- llama_model_loader: - kv 4: command-r.embedding_length u32 = 8192
- llama_model_loader: - kv 5: command-r.feed_forward_length u32 = 22528
- llama_model_loader: - kv 6: command-r.attention.head_count u32 = 64
- llama_model_loader: - kv 7: command-r.attention.head_count_kv u32 = 64
- llama_model_loader: - kv 8: command-r.rope.freq_base f32 = 8000000.000000
- llama_model_loader: - kv 9: command-r.attention.layer_norm_epsilon f32 = 0.000010
- llama_model_loader: - kv 10: general.file_type u32 = 11
- llama_model_loader: - kv 11: command-r.logit_scale f32 = 0.062500
- llama_model_loader: - kv 12: command-r.rope.scaling.type str = none
- llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
- llama_model_loader: - kv 14: tokenizer.ggml.tokens arr[str,256000] = ["<PAD>", "<UNK>", "<CLS>", "<SEP>", ...
- llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,256000] = [3, 3, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, ...
- llama_model_loader: - kv 16: tokenizer.ggml.merges arr[str,253333] = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ a...
- llama_model_loader: - kv 17: tokenizer.ggml.bos_token_id u32 = 5
- llama_model_loader: - kv 18: tokenizer.ggml.eos_token_id u32 = 255001
- llama_model_loader: - kv 19: tokenizer.ggml.padding_token_id u32 = 0
- llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true
- llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false
- llama_model_loader: - kv 22: general.quantization_version u32 = 2
- llama_model_loader: - type f32: 41 tensors
- llama_model_loader: - type q3_K: 280 tensors
- llama_model_loader: - type q6_K: 1 tensors
- llm_load_vocab: special tokens definition check successful ( 1008/256000 ).
- llm_load_print_meta: format = GGUF V3 (latest)
- llm_load_print_meta: arch = command-r
- llm_load_print_meta: vocab type = BPE
- llm_load_print_meta: n_vocab = 256000
- llm_load_print_meta: n_merges = 253333
- llm_load_print_meta: n_ctx_train = 131072
- llm_load_print_meta: n_embd = 8192
- llm_load_print_meta: n_head = 64
- llm_load_print_meta: n_head_kv = 64
- llm_load_print_meta: n_layer = 40
- llm_load_print_meta: n_rot = 128
- llm_load_print_meta: n_embd_head_k = 128
- llm_load_print_meta: n_embd_head_v = 128
- llm_load_print_meta: n_gqa = 1
- llm_load_print_meta: n_embd_k_gqa = 8192
- llm_load_print_meta: n_embd_v_gqa = 8192
- llm_load_print_meta: f_norm_eps = 1.0e-05
- llm_load_print_meta: f_norm_rms_eps = 0.0e+00
- llm_load_print_meta: f_clamp_kqv = 0.0e+00
- llm_load_print_meta: f_max_alibi_bias = 0.0e+00
- llm_load_print_meta: f_logit_scale = 6.2e-02
- llm_load_print_meta: n_ff = 22528
- llm_load_print_meta: n_expert = 0
- llm_load_print_meta: n_expert_used = 0
- llm_load_print_meta: causal attn = 1
- llm_load_print_meta: pooling type = 0
- llm_load_print_meta: rope type = 0
- llm_load_print_meta: rope scaling = none
- llm_load_print_meta: freq_base_train = 8000000.0
- llm_load_print_meta: freq_scale_train = 1
- llm_load_print_meta: n_yarn_orig_ctx = 131072
- llm_load_print_meta: rope_finetuned = unknown
- llm_load_print_meta: ssm_d_conv = 0
- llm_load_print_meta: ssm_d_inner = 0
- llm_load_print_meta: ssm_d_state = 0
- llm_load_print_meta: ssm_dt_rank = 0
- llm_load_print_meta: model type = 35B
- llm_load_print_meta: model ftype = Q3_K - Small
- llm_load_print_meta: model params = 34.98 B
- llm_load_print_meta: model size = 14.76 GiB (3.63 BPW)
- llm_load_print_meta: general.name = c4ai-command-r-v01
- llm_load_print_meta: BOS token = 5 '<BOS_TOKEN>'
- llm_load_print_meta: EOS token = 255001 '<|END_OF_TURN_TOKEN|>'
- llm_load_print_meta: PAD token = 0 '<PAD>'
- llm_load_print_meta: LF token = 136 'Ä'
- llm_load_tensors: ggml ctx size = 0.49 MiB
- llm_load_tensors: offloading 36 repeating layers to GPU
- llm_load_tensors: offloaded 36/41 layers to GPU
- llm_load_tensors: CPU buffer size = 15116.91 MiB
- llm_load_tensors: CUDA0 buffer size = 3705.97 MiB
- llm_load_tensors: CUDA1 buffer size = 3369.06 MiB
- llm_load_tensors: CUDA2 buffer size = 5053.59 MiB
- ...................................................................................
- llama_new_context_with_model: n_ctx = 2048
- llama_new_context_with_model: n_batch = 512
- llama_new_context_with_model: n_ubatch = 512
- llama_new_context_with_model: freq_base = 8000000.0
- llama_new_context_with_model: freq_scale = 1
- llama_kv_cache_init: CUDA_Host KV buffer size = 256.00 MiB
- llama_kv_cache_init: CUDA0 KV buffer size = 704.00 MiB
- llama_kv_cache_init: CUDA1 KV buffer size = 640.00 MiB
- llama_kv_cache_init: CUDA2 KV buffer size = 960.00 MiB
- llama_new_context_with_model: KV self size = 2560.00 MiB, K (f16): 1280.00 MiB, V (f16): 1280.00 MiB
- llama_new_context_with_model: CUDA_Host output buffer size = 516.00 MiB
- llama_new_context_with_model: CUDA0 compute buffer size = 2156.62 MiB
- llama_new_context_with_model: CUDA1 compute buffer size = 324.00 MiB
- llama_new_context_with_model: CUDA2 compute buffer size = 324.00 MiB
- llama_new_context_with_model: CUDA_Host compute buffer size = 36.00 MiB
- llama_new_context_with_model: graph nodes = 1245
- llama_new_context_with_model: graph splits = 46
- {"function":"initialize","level":"INFO","line":444,"msg":"initializing slots","n_slots":1,"tid":"21568","timestamp":1712661638}
- {"function":"initialize","level":"INFO","line":456,"msg":"new slot","n_ctx_slot":2048,"slot_id":0,"tid":"21568","timestamp":1712661638}
- time=2024-04-09T16:50:38.720+05:30 level=INFO source=dyn_ext_server.go:159 msg="Starting llama main loop"
- {"function":"update_slots","level":"INFO","line":1574,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"27556","timestamp":1712661638}
- {"function":"launch_slot_with_data","level":"INFO","line":829,"msg":"slot is processing task","slot_id":0,"task_id":0,"tid":"27556","timestamp":1712661638}
- {"function":"update_slots","ga_i":0,"level":"INFO","line":1812,"msg":"slot progression","n_past":0,"n_past_se":0,"n_prompt_tokens_processed":77,"slot_id":0,"task_id":0,"tid":"27556","timestamp":1712661638}
- {"function":"update_slots","level":"INFO","line":1836,"msg":"kv cache rm [p0, end)","p0":0,"slot_id":0,"task_id":0,"tid":"27556","timestamp":1712661638}
- CUDA error: out of memory
- current device: 0, in function alloc at C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda.cu:532
- cuMemSetAccess(pool_addr + pool_size, reserve_size, &access, 1)
- GGML_ASSERT: C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda.cu:193: !"CUDA error"
Add Comment
Please, Sign In to add comment
