API tools faq
paste
Guest User

ollama logs

a guest
Jul 8th, 2024
38
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 61.43 KB | None | 0 0
raw download clone embed print report
  1. 2024/07/09 03:05:38 routes.go:1033: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:10m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:2 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:4 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
  2. time=2024-07-09T03:05:38.609Z level=INFO source=images.go:751 msg="total blobs: 74"
  3. time=2024-07-09T03:05:38.611Z level=INFO source=images.go:758 msg="total unused blobs removed: 0"
  4. time=2024-07-09T03:05:38.612Z level=INFO source=routes.go:1080 msg="Listening on [::]:11434 (version 0.2.0)"
  5. time=2024-07-09T03:05:38.613Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1577350832/runners
  6. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu file=build/linux/x86_64/cpu/bin/ollama_llama_server.gz
  7. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx file=build/linux/x86_64/cpu_avx/bin/ollama_llama_server.gz
  8. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=cpu_avx2 file=build/linux/x86_64/cpu_avx2/bin/ollama_llama_server.gz
  9. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublas.so.11.gz
  10. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcublasLt.so.11.gz
  11. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/libcudart.so.11.0.gz
  12. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=cuda_v11 file=build/linux/x86_64/cuda_v11/bin/ollama_llama_server.gz
  13. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60101 file=build/linux/x86_64/rocm_v60101/bin/deps.txt.gz
  14. time=2024-07-09T03:05:38.613Z level=DEBUG source=payload.go:182 msg=extracting variant=rocm_v60101 file=build/linux/x86_64/rocm_v60101/bin/ollama_llama_server.gz
  15. time=2024-07-09T03:05:41.535Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu/ollama_llama_server
  16. time=2024-07-09T03:05:41.535Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx/ollama_llama_server
  17. time=2024-07-09T03:05:41.535Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx2/ollama_llama_server
  18. time=2024-07-09T03:05:41.535Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server
  19. time=2024-07-09T03:05:41.535Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/rocm_v60101/ollama_llama_server
  20. time=2024-07-09T03:05:41.535Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v11 rocm_v60101 cpu cpu_avx cpu_avx2]"
  21. time=2024-07-09T03:05:41.535Z level=DEBUG source=payload.go:45 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
  22. time=2024-07-09T03:05:41.535Z level=DEBUG source=sched.go:102 msg="starting llm scheduler"
  23. time=2024-07-09T03:05:41.535Z level=INFO source=gpu.go:205 msg="looking for compatible GPUs"
  24. time=2024-07-09T03:05:41.535Z level=DEBUG source=gpu.go:91 msg="searching for GPU discovery libraries for NVIDIA"
  25. time=2024-07-09T03:05:41.535Z level=DEBUG source=gpu.go:438 msg="Searching for GPU library" name=libcuda.so*
  26. time=2024-07-09T03:05:41.535Z level=DEBUG source=gpu.go:457 msg="gpu library search" globs="[/usr/local/nvidia/lib/libcuda.so** /usr/local/nvidia/lib64/libcuda.so** /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
  27. time=2024-07-09T03:05:41.536Z level=DEBUG source=gpu.go:491 msg="discovered GPU libraries" paths=[/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01]
  28. CUDA driver version: 12.2
  29. time=2024-07-09T03:05:41.558Z level=DEBUG source=gpu.go:124 msg="detected GPUs" count=2 library=/usr/lib/x86_64-linux-gnu/libcuda.so.535.183.01
  30. [GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25] CUDA totalMem 22191 mb
  31. [GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25] CUDA freeMem 22013 mb
  32. [GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25] Compute Capability 7.5
  33. [GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0] CUDA totalMem 22191 mb
  34. [GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0] CUDA freeMem 22024 mb
  35. [GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0] Compute Capability 7.5
  36. time=2024-07-09T03:05:41.758Z level=DEBUG source=amd_linux.go:356 msg="amdgpu driver not detected /sys/module/amdgpu"
  37. releasing cuda driver library
  38. time=2024-07-09T03:05:41.758Z level=INFO source=types.go:103 msg="inference compute" id=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 library=cuda compute=7.5 driver=12.2 name="NVIDIA GeForce RTX 2080 Ti" total="21.7 GiB" available="21.5 GiB"
  39. time=2024-07-09T03:05:41.758Z level=INFO source=types.go:103 msg="inference compute" id=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 library=cuda compute=7.5 driver=12.2 name="NVIDIA GeForce RTX 2080 Ti" total="21.7 GiB" available="21.5 GiB"
  40. [GIN] 2024/07/09 - 03:05:46 | 200 | 28.322µs | 127.0.0.1 | HEAD "/"
  41. [GIN] 2024/07/09 - 03:05:46 | 200 | 34.124968ms | 127.0.0.1 | POST "/api/show"
  42. time=2024-07-09T03:05:46.681Z level=DEBUG source=gpu.go:336 msg="updating system memory data" before.total="30.3 GiB" before.free="28.1 GiB" now.total="30.3 GiB" now.free="26.3 GiB"
  43. CUDA driver version: 12.2
  44. time=2024-07-09T03:05:46.788Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="178.0 MiB"
  45. time=2024-07-09T03:05:46.903Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="167.6 MiB"
  46. releasing cuda driver library
  47. time=2024-07-09T03:05:46.931Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=2 available="[21.5 GiB 21.5 GiB]"
  48. time=2024-07-09T03:05:46.932Z level=DEBUG source=sched.go:254 msg="loading first model" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  49. time=2024-07-09T03:05:46.932Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[21.5 GiB]"
  50. time=2024-07-09T03:05:46.933Z level=INFO source=sched.go:741 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 parallel=4 available=23093903360 required="6.2 GiB"
  51. time=2024-07-09T03:05:46.933Z level=DEBUG source=server.go:98 msg="system memory" total="30.3 GiB" free=27599036416
  52. time=2024-07-09T03:05:46.934Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[21.5 GiB]"
  53. time=2024-07-09T03:05:46.935Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[21.5 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
  54. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu/ollama_llama_server
  55. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx/ollama_llama_server
  56. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx2/ollama_llama_server
  57. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server
  58. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/rocm_v60101/ollama_llama_server
  59. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu/ollama_llama_server
  60. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx/ollama_llama_server
  61. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx2/ollama_llama_server
  62. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server
  63. time=2024-07-09T03:05:46.941Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/rocm_v60101/ollama_llama_server
  64. time=2024-07-09T03:05:46.941Z level=INFO source=server.go:375 msg="starting llama server" cmd="/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 41871"
  65. time=2024-07-09T03:05:46.941Z level=DEBUG source=server.go:390 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama1577350832/runners/cuda_v11:/tmp/ollama1577350832/runners:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0]"
  66. time=2024-07-09T03:05:46.945Z level=INFO source=sched.go:477 msg="loaded runners" count=1
  67. time=2024-07-09T03:05:46.945Z level=INFO source=server.go:563 msg="waiting for llama runner to start responding"
  68. time=2024-07-09T03:05:46.945Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server error"
  69. INFO [main] build info | build=1 commit="a8db2a9" tid="140553717837824" timestamp=1720494346
  70. INFO [main] system info | n_threads=4 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="140553717837824" timestamp=1720494346 total_threads=8
  71. INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="41871" tid="140553717837824" timestamp=1720494346
  72. llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from /root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa (version GGUF V3 (latest))
  73. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  74. llama_model_loader: - kv 0: general.architecture str = llama
  75. llama_model_loader: - kv 1: general.name str = Meta-Llama-3-8B-Instruct
  76. llama_model_loader: - kv 2: llama.block_count u32 = 32
  77. llama_model_loader: - kv 3: llama.context_length u32 = 8192
  78. llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
  79. llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
  80. llama_model_loader: - kv 6: llama.attention.head_count u32 = 32
  81. llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
  82. llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000
  83. llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
  84. llama_model_loader: - kv 10: general.file_type u32 = 2
  85. llama_model_loader: - kv 11: llama.vocab_size u32 = 128256
  86. llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
  87. llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
  88. llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe
  89. llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
  90. llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  91. time=2024-07-09T03:05:47.197Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server loading model"
  92. llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
  93. llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000
  94. llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128009
  95. llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
  96. llama_model_loader: - kv 21: general.quantization_version u32 = 2
  97. llama_model_loader: - type f32: 65 tensors
  98. llama_model_loader: - type q4_0: 225 tensors
  99. llama_model_loader: - type q6_K: 1 tensors
  100. llm_load_vocab: special tokens cache size = 256
  101. llm_load_vocab: token to piece cache size = 0.8000 MB
  102. llm_load_print_meta: format = GGUF V3 (latest)
  103. llm_load_print_meta: arch = llama
  104. llm_load_print_meta: vocab type = BPE
  105. llm_load_print_meta: n_vocab = 128256
  106. llm_load_print_meta: n_merges = 280147
  107. llm_load_print_meta: vocab_only = 0
  108. llm_load_print_meta: n_ctx_train = 8192
  109. llm_load_print_meta: n_embd = 4096
  110. llm_load_print_meta: n_layer = 32
  111. llm_load_print_meta: n_head = 32
  112. llm_load_print_meta: n_head_kv = 8
  113. llm_load_print_meta: n_rot = 128
  114. llm_load_print_meta: n_swa = 0
  115. llm_load_print_meta: n_embd_head_k = 128
  116. llm_load_print_meta: n_embd_head_v = 128
  117. llm_load_print_meta: n_gqa = 4
  118. llm_load_print_meta: n_embd_k_gqa = 1024
  119. llm_load_print_meta: n_embd_v_gqa = 1024
  120. llm_load_print_meta: f_norm_eps = 0.0e+00
  121. llm_load_print_meta: f_norm_rms_eps = 1.0e-05
  122. llm_load_print_meta: f_clamp_kqv = 0.0e+00
  123. llm_load_print_meta: f_max_alibi_bias = 0.0e+00
  124. llm_load_print_meta: f_logit_scale = 0.0e+00
  125. llm_load_print_meta: n_ff = 14336
  126. llm_load_print_meta: n_expert = 0
  127. llm_load_print_meta: n_expert_used = 0
  128. llm_load_print_meta: causal attn = 1
  129. llm_load_print_meta: pooling type = 0
  130. llm_load_print_meta: rope type = 0
  131. llm_load_print_meta: rope scaling = linear
  132. llm_load_print_meta: freq_base_train = 500000.0
  133. llm_load_print_meta: freq_scale_train = 1
  134. llm_load_print_meta: n_ctx_orig_yarn = 8192
  135. llm_load_print_meta: rope_finetuned = unknown
  136. llm_load_print_meta: ssm_d_conv = 0
  137. llm_load_print_meta: ssm_d_inner = 0
  138. llm_load_print_meta: ssm_d_state = 0
  139. llm_load_print_meta: ssm_dt_rank = 0
  140. llm_load_print_meta: model type = 8B
  141. llm_load_print_meta: model ftype = Q4_0
  142. llm_load_print_meta: model params = 8.03 B
  143. llm_load_print_meta: model size = 4.33 GiB (4.64 BPW)
  144. llm_load_print_meta: general.name = Meta-Llama-3-8B-Instruct
  145. llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
  146. llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
  147. llm_load_print_meta: LF token = 128 'Ä'
  148. llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
  149. llm_load_print_meta: max token length = 256
  150. ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
  151. ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  152. ggml_cuda_init: found 1 CUDA devices:
  153. Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
  154. llm_load_tensors: ggml ctx size = 0.27 MiB
  155. llm_load_tensors: offloading 32 repeating layers to GPU
  156. llm_load_tensors: offloading non-repeating layers to GPU
  157. llm_load_tensors: offloaded 33/33 layers to GPU
  158. llm_load_tensors: CPU buffer size = 281.81 MiB
  159. llm_load_tensors: CUDA0 buffer size = 4155.99 MiB
  160. time=2024-07-09T03:05:48.451Z level=DEBUG source=server.go:615 msg="model load progress 0.06"
  161. time=2024-07-09T03:05:48.702Z level=DEBUG source=server.go:615 msg="model load progress 0.45"
  162. time=2024-07-09T03:05:48.953Z level=DEBUG source=server.go:615 msg="model load progress 0.77"
  163. time=2024-07-09T03:05:49.204Z level=DEBUG source=server.go:615 msg="model load progress 0.99"
  164. llama_new_context_with_model: n_ctx = 8192
  165. llama_new_context_with_model: n_batch = 512
  166. llama_new_context_with_model: n_ubatch = 512
  167. llama_new_context_with_model: flash_attn = 0
  168. llama_new_context_with_model: freq_base = 500000.0
  169. llama_new_context_with_model: freq_scale = 1
  170. llama_kv_cache_init: CUDA0 KV buffer size = 1024.00 MiB
  171. llama_new_context_with_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB
  172. llama_new_context_with_model: CUDA_Host output buffer size = 2.02 MiB
  173. llama_new_context_with_model: CUDA0 compute buffer size = 560.00 MiB
  174. llama_new_context_with_model: CUDA_Host compute buffer size = 24.01 MiB
  175. llama_new_context_with_model: graph nodes = 1030
  176. llama_new_context_with_model: graph splits = 2
  177. time=2024-07-09T03:05:49.455Z level=DEBUG source=server.go:615 msg="model load progress 1.00"
  178. time=2024-07-09T03:05:49.706Z level=DEBUG source=server.go:618 msg="model load completed, waiting for server to become available" status="llm server loading model"
  179. DEBUG [initialize] initializing slots | n_slots=4 tid="140553717837824" timestamp=1720494349
  180. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="140553717837824" timestamp=1720494349
  181. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=1 tid="140553717837824" timestamp=1720494349
  182. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=2 tid="140553717837824" timestamp=1720494349
  183. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=3 tid="140553717837824" timestamp=1720494349
  184. INFO [main] model loaded | tid="140553717837824" timestamp=1720494349
  185. DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="140553717837824" timestamp=1720494349
  186. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=0 tid="140553717837824" timestamp=1720494349
  187. time=2024-07-09T03:05:49.959Z level=INFO source=server.go:609 msg="llama runner started in 3.01 seconds"
  188. time=2024-07-09T03:05:49.959Z level=DEBUG source=sched.go:490 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  189. time=2024-07-09T03:05:49.959Z level=DEBUG source=prompt.go:168 msg="prompt now fits in context window" required=1 window=2048
  190. [GIN] 2024/07/09 - 03:05:49 | 200 | 3.299985599s | 127.0.0.1 | POST "/api/chat"
  191. time=2024-07-09T03:05:49.960Z level=DEBUG source=sched.go:494 msg="context for request finished"
  192. time=2024-07-09T03:05:49.960Z level=DEBUG source=sched.go:366 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa duration=10m0s
  193. time=2024-07-09T03:05:49.960Z level=DEBUG source=sched.go:384 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa refCount=0
  194. time=2024-07-09T03:05:51.326Z level=DEBUG source=sched.go:603 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  195. time=2024-07-09T03:05:51.326Z level=DEBUG source=sched.go:310 msg="resetting model to expire immediately to make room" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa refCount=0
  196. time=2024-07-09T03:05:51.326Z level=DEBUG source=sched.go:323 msg="waiting for pending requests to complete and unload to occur" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  197. time=2024-07-09T03:05:51.326Z level=DEBUG source=sched.go:387 msg="runner expired event received" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  198. time=2024-07-09T03:05:51.326Z level=DEBUG source=sched.go:403 msg="got lock to unload" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  199. time=2024-07-09T03:05:51.326Z level=DEBUG source=gpu.go:336 msg="updating system memory data" before.total="30.3 GiB" before.free="26.3 GiB" now.total="30.3 GiB" now.free="24.6 GiB"
  200. CUDA driver version: 12.2
  201. time=2024-07-09T03:05:51.443Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="178.0 MiB"
  202. time=2024-07-09T03:05:51.505Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="15.5 GiB" now.used="6.2 GiB"
  203. releasing cuda driver library
  204. time=2024-07-09T03:05:51.505Z level=DEBUG source=server.go:1026 msg="stopping llama server"
  205. time=2024-07-09T03:05:51.505Z level=DEBUG source=server.go:1032 msg="waiting for llama server to exit"
  206. time=2024-07-09T03:05:51.581Z level=DEBUG source=server.go:1036 msg="llama server stopped"
  207. time=2024-07-09T03:05:51.581Z level=DEBUG source=sched.go:408 msg="runner released" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  208. time=2024-07-09T03:05:51.756Z level=DEBUG source=gpu.go:336 msg="updating system memory data" before.total="30.3 GiB" before.free="24.6 GiB" now.total="30.3 GiB" now.free="25.4 GiB"
  209. CUDA driver version: 12.2
  210. time=2024-07-09T03:05:51.848Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="178.0 MiB"
  211. time=2024-07-09T03:05:51.922Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="15.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="167.6 MiB"
  212. releasing cuda driver library
  213. time=2024-07-09T03:05:51.922Z level=DEBUG source=sched.go:687 msg="gpu VRAM free memory converged after 0.60 seconds" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  214. time=2024-07-09T03:05:51.922Z level=DEBUG source=sched.go:412 msg="sending an unloaded event" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  215. time=2024-07-09T03:05:51.922Z level=DEBUG source=sched.go:329 msg="unload completed" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  216. time=2024-07-09T03:05:51.922Z level=DEBUG source=gpu.go:336 msg="updating system memory data" before.total="30.3 GiB" before.free="25.4 GiB" now.total="30.3 GiB" now.free="25.4 GiB"
  217. CUDA driver version: 12.2
  218. time=2024-07-09T03:05:51.996Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="178.0 MiB"
  219. time=2024-07-09T03:05:52.070Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="167.6 MiB"
  220. releasing cuda driver library
  221. time=2024-07-09T03:05:52.090Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=2 available="[21.5 GiB 21.5 GiB]"
  222. time=2024-07-09T03:05:52.091Z level=DEBUG source=sched.go:254 msg="loading first model" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  223. time=2024-07-09T03:05:52.091Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[21.5 GiB]"
  224. time=2024-07-09T03:05:52.092Z level=INFO source=sched.go:741 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 parallel=4 available=23093903360 required="6.2 GiB"
  225. time=2024-07-09T03:05:52.092Z level=DEBUG source=server.go:98 msg="system memory" total="30.3 GiB" free=27231694848
  226. time=2024-07-09T03:05:52.092Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[21.5 GiB]"
  227. time=2024-07-09T03:05:52.092Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[21.5 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
  228. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu/ollama_llama_server
  229. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx/ollama_llama_server
  230. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx2/ollama_llama_server
  231. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server
  232. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/rocm_v60101/ollama_llama_server
  233. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu/ollama_llama_server
  234. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx/ollama_llama_server
  235. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx2/ollama_llama_server
  236. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server
  237. time=2024-07-09T03:05:52.093Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/rocm_v60101/ollama_llama_server
  238. time=2024-07-09T03:05:52.093Z level=INFO source=server.go:375 msg="starting llama server" cmd="/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 39689"
  239. time=2024-07-09T03:05:52.093Z level=DEBUG source=server.go:390 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama1577350832/runners/cuda_v11:/tmp/ollama1577350832/runners:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0]"
  240. time=2024-07-09T03:05:52.093Z level=INFO source=sched.go:477 msg="loaded runners" count=1
  241. time=2024-07-09T03:05:52.093Z level=INFO source=server.go:563 msg="waiting for llama runner to start responding"
  242. time=2024-07-09T03:05:52.093Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server error"
  243. INFO [main] build info | build=1 commit="a8db2a9" tid="139722130096128" timestamp=1720494352
  244. INFO [main] system info | n_threads=4 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="139722130096128" timestamp=1720494352 total_threads=8
  245. INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="39689" tid="139722130096128" timestamp=1720494352
  246. llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from /root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa (version GGUF V3 (latest))
  247. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  248. llama_model_loader: - kv 0: general.architecture str = llama
  249. llama_model_loader: - kv 1: general.name str = Meta-Llama-3-8B-Instruct
  250. llama_model_loader: - kv 2: llama.block_count u32 = 32
  251. llama_model_loader: - kv 3: llama.context_length u32 = 8192
  252. llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
  253. llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
  254. llama_model_loader: - kv 6: llama.attention.head_count u32 = 32
  255. llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
  256. llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000
  257. llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
  258. llama_model_loader: - kv 10: general.file_type u32 = 2
  259. llama_model_loader: - kv 11: llama.vocab_size u32 = 128256
  260. llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
  261. llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
  262. llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe
  263. llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
  264. llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  265. time=2024-07-09T03:05:52.344Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server loading model"
  266. llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
  267. llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000
  268. llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128009
  269. llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
  270. llama_model_loader: - kv 21: general.quantization_version u32 = 2
  271. llama_model_loader: - type f32: 65 tensors
  272. llama_model_loader: - type q4_0: 225 tensors
  273. llama_model_loader: - type q6_K: 1 tensors
  274. llm_load_vocab: special tokens cache size = 256
  275. llm_load_vocab: token to piece cache size = 0.8000 MB
  276. llm_load_print_meta: format = GGUF V3 (latest)
  277. llm_load_print_meta: arch = llama
  278. llm_load_print_meta: vocab type = BPE
  279. llm_load_print_meta: n_vocab = 128256
  280. llm_load_print_meta: n_merges = 280147
  281. llm_load_print_meta: vocab_only = 0
  282. llm_load_print_meta: n_ctx_train = 8192
  283. llm_load_print_meta: n_embd = 4096
  284. llm_load_print_meta: n_layer = 32
  285. llm_load_print_meta: n_head = 32
  286. llm_load_print_meta: n_head_kv = 8
  287. llm_load_print_meta: n_rot = 128
  288. llm_load_print_meta: n_swa = 0
  289. llm_load_print_meta: n_embd_head_k = 128
  290. llm_load_print_meta: n_embd_head_v = 128
  291. llm_load_print_meta: n_gqa = 4
  292. llm_load_print_meta: n_embd_k_gqa = 1024
  293. llm_load_print_meta: n_embd_v_gqa = 1024
  294. llm_load_print_meta: f_norm_eps = 0.0e+00
  295. llm_load_print_meta: f_norm_rms_eps = 1.0e-05
  296. llm_load_print_meta: f_clamp_kqv = 0.0e+00
  297. llm_load_print_meta: f_max_alibi_bias = 0.0e+00
  298. llm_load_print_meta: f_logit_scale = 0.0e+00
  299. llm_load_print_meta: n_ff = 14336
  300. llm_load_print_meta: n_expert = 0
  301. llm_load_print_meta: n_expert_used = 0
  302. llm_load_print_meta: causal attn = 1
  303. llm_load_print_meta: pooling type = 0
  304. llm_load_print_meta: rope type = 0
  305. llm_load_print_meta: rope scaling = linear
  306. llm_load_print_meta: freq_base_train = 500000.0
  307. llm_load_print_meta: freq_scale_train = 1
  308. llm_load_print_meta: n_ctx_orig_yarn = 8192
  309. llm_load_print_meta: rope_finetuned = unknown
  310. llm_load_print_meta: ssm_d_conv = 0
  311. llm_load_print_meta: ssm_d_inner = 0
  312. llm_load_print_meta: ssm_d_state = 0
  313. llm_load_print_meta: ssm_dt_rank = 0
  314. llm_load_print_meta: model type = 8B
  315. llm_load_print_meta: model ftype = Q4_0
  316. llm_load_print_meta: model params = 8.03 B
  317. llm_load_print_meta: model size = 4.33 GiB (4.64 BPW)
  318. llm_load_print_meta: general.name = Meta-Llama-3-8B-Instruct
  319. llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
  320. llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
  321. llm_load_print_meta: LF token = 128 'Ä'
  322. llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
  323. llm_load_print_meta: max token length = 256
  324. ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
  325. ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  326. ggml_cuda_init: found 1 CUDA devices:
  327. Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
  328. llm_load_tensors: ggml ctx size = 0.27 MiB
  329. llm_load_tensors: offloading 32 repeating layers to GPU
  330. llm_load_tensors: offloading non-repeating layers to GPU
  331. llm_load_tensors: offloaded 33/33 layers to GPU
  332. llm_load_tensors: CPU buffer size = 281.81 MiB
  333. llm_load_tensors: CUDA0 buffer size = 4155.99 MiB
  334. time=2024-07-09T03:05:53.598Z level=DEBUG source=server.go:615 msg="model load progress 0.18"
  335. time=2024-07-09T03:05:53.849Z level=DEBUG source=server.go:615 msg="model load progress 0.49"
  336. time=2024-07-09T03:05:54.099Z level=DEBUG source=server.go:615 msg="model load progress 0.81"
  337. time=2024-07-09T03:05:54.350Z level=DEBUG source=server.go:615 msg="model load progress 0.99"
  338. llama_new_context_with_model: n_ctx = 8192
  339. llama_new_context_with_model: n_batch = 512
  340. llama_new_context_with_model: n_ubatch = 512
  341. llama_new_context_with_model: flash_attn = 0
  342. llama_new_context_with_model: freq_base = 500000.0
  343. llama_new_context_with_model: freq_scale = 1
  344. llama_kv_cache_init: CUDA0 KV buffer size = 1024.00 MiB
  345. llama_new_context_with_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB
  346. llama_new_context_with_model: CUDA_Host output buffer size = 2.02 MiB
  347. llama_new_context_with_model: CUDA0 compute buffer size = 560.00 MiB
  348. llama_new_context_with_model: CUDA_Host compute buffer size = 24.01 MiB
  349. llama_new_context_with_model: graph nodes = 1030
  350. llama_new_context_with_model: graph splits = 2
  351. time=2024-07-09T03:05:54.601Z level=DEBUG source=server.go:615 msg="model load progress 1.00"
  352. DEBUG [initialize] initializing slots | n_slots=4 tid="139722130096128" timestamp=1720494354
  353. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="139722130096128" timestamp=1720494354
  354. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=1 tid="139722130096128" timestamp=1720494354
  355. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=2 tid="139722130096128" timestamp=1720494354
  356. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=3 tid="139722130096128" timestamp=1720494354
  357. INFO [main] model loaded | tid="139722130096128" timestamp=1720494354
  358. DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="139722130096128" timestamp=1720494354
  359. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=0 tid="139722130096128" timestamp=1720494354
  360. time=2024-07-09T03:05:54.852Z level=INFO source=server.go:609 msg="llama runner started in 2.76 seconds"
  361. time=2024-07-09T03:05:54.852Z level=DEBUG source=sched.go:490 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  362. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=1 tid="139722130096128" timestamp=1720494354
  363. DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=44292 status=200 tid="139721594368000" timestamp=1720494354
  364. time=2024-07-09T03:05:54.941Z level=DEBUG source=prompt.go:168 msg="prompt now fits in context window" required=12 window=2048
  365. time=2024-07-09T03:05:54.941Z level=DEBUG source=routes.go:1334 msg="chat handler" prompt="<|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" images=0
  366. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=2 tid="139722130096128" timestamp=1720494354
  367. DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=3 tid="139722130096128" timestamp=1720494354
  368. DEBUG [update_slots] slot progression | ga_i=0 n_past=0 n_past_se=0 n_prompt_tokens_processed=11 slot_id=0 task_id=3 tid="139722130096128" timestamp=1720494354
  369. DEBUG [update_slots] kv cache rm [p0, end) | p0=0 slot_id=0 task_id=3 tid="139722130096128" timestamp=1720494354
  370. DEBUG [print_timings] prompt eval time = 36.72 ms / 11 tokens ( 3.34 ms per token, 299.60 tokens per second) | n_prompt_tokens_processed=11 n_tokens_second=299.5969059810437 slot_id=0 t_prompt_processing=36.716 t_token=3.337818181818182 task_id=3 tid="139722130096128" timestamp=1720494355
  371. DEBUG [print_timings] generation eval time = 293.62 ms / 26 runs ( 11.29 ms per token, 88.55 tokens per second) | n_decoded=26 n_tokens_second=88.5513342301994 slot_id=0 t_token=11.292884615384615 t_token_generation=293.615 task_id=3 tid="139722130096128" timestamp=1720494355
  372. DEBUG [print_timings] total time = 330.33 ms | slot_id=0 t_prompt_processing=36.716 t_token_generation=293.615 t_total=330.331 task_id=3 tid="139722130096128" timestamp=1720494355
  373. DEBUG [update_slots] slot released | n_cache_tokens=37 n_ctx=8192 n_past=36 n_system_tokens=0 slot_id=0 task_id=3 tid="139722130096128" timestamp=1720494355 truncated=false
  374. DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=44292 status=200 tid="139721594368000" timestamp=1720494355
  375. [GIN] 2024/07/09 - 03:05:55 | 200 | 4.009726723s | 127.0.0.1 | POST "/api/chat"
  376. time=2024-07-09T03:05:55.316Z level=DEBUG source=sched.go:494 msg="context for request finished"
  377. time=2024-07-09T03:05:55.316Z level=DEBUG source=sched.go:366 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa duration=10m0s
  378. time=2024-07-09T03:05:55.316Z level=DEBUG source=sched.go:384 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa refCount=0
  379. [GIN] 2024/07/09 - 03:05:59 | 200 | 15.832243ms | 172.21.0.4 | GET "/api/tags"
  380. time=2024-07-09T03:05:59.327Z level=DEBUG source=sched.go:603 msg="evaluating already loaded" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  381. time=2024-07-09T03:05:59.327Z level=DEBUG source=sched.go:310 msg="resetting model to expire immediately to make room" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa refCount=0
  382. time=2024-07-09T03:05:59.327Z level=DEBUG source=sched.go:323 msg="waiting for pending requests to complete and unload to occur" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  383. time=2024-07-09T03:05:59.327Z level=DEBUG source=sched.go:387 msg="runner expired event received" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  384. time=2024-07-09T03:05:59.327Z level=DEBUG source=sched.go:403 msg="got lock to unload" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  385. time=2024-07-09T03:05:59.327Z level=DEBUG source=gpu.go:336 msg="updating system memory data" before.total="30.3 GiB" before.free="25.4 GiB" now.total="30.3 GiB" now.free="24.6 GiB"
  386. CUDA driver version: 12.2
  387. time=2024-07-09T03:05:59.434Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="178.0 MiB"
  388. time=2024-07-09T03:05:59.504Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="15.5 GiB" now.used="6.2 GiB"
  389. releasing cuda driver library
  390. time=2024-07-09T03:05:59.504Z level=DEBUG source=server.go:1026 msg="stopping llama server"
  391. time=2024-07-09T03:05:59.504Z level=DEBUG source=server.go:1032 msg="waiting for llama server to exit"
  392. time=2024-07-09T03:05:59.574Z level=DEBUG source=server.go:1036 msg="llama server stopped"
  393. time=2024-07-09T03:05:59.574Z level=DEBUG source=sched.go:408 msg="runner released" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  394. time=2024-07-09T03:05:59.754Z level=DEBUG source=gpu.go:336 msg="updating system memory data" before.total="30.3 GiB" before.free="24.6 GiB" now.total="30.3 GiB" now.free="25.4 GiB"
  395. CUDA driver version: 12.2
  396. time=2024-07-09T03:05:59.832Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="178.0 MiB"
  397. time=2024-07-09T03:05:59.903Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="15.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="167.6 MiB"
  398. releasing cuda driver library
  399. time=2024-07-09T03:05:59.903Z level=DEBUG source=sched.go:687 msg="gpu VRAM free memory converged after 0.58 seconds" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  400. time=2024-07-09T03:05:59.903Z level=DEBUG source=sched.go:412 msg="sending an unloaded event" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  401. time=2024-07-09T03:05:59.903Z level=DEBUG source=sched.go:329 msg="unload completed" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  402. time=2024-07-09T03:05:59.903Z level=DEBUG source=gpu.go:336 msg="updating system memory data" before.total="30.3 GiB" before.free="25.4 GiB" now.total="30.3 GiB" now.free="25.3 GiB"
  403. CUDA driver version: 12.2
  404. time=2024-07-09T03:05:59.976Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-ec2a76a4-bee1-631e-1b81-cdc6e6fe0d25 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="178.0 MiB"
  405. time=2024-07-09T03:06:00.046Z level=DEBUG source=gpu.go:377 msg="updating cuda memory data" gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 name="NVIDIA GeForce RTX 2080 Ti" before.total="21.7 GiB" before.free="21.5 GiB" now.total="21.7 GiB" now.free="21.5 GiB" now.used="167.6 MiB"
  406. releasing cuda driver library
  407. time=2024-07-09T03:06:00.064Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=2 available="[21.5 GiB 21.5 GiB]"
  408. time=2024-07-09T03:06:00.064Z level=DEBUG source=sched.go:254 msg="loading first model" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  409. time=2024-07-09T03:06:00.064Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[21.5 GiB]"
  410. time=2024-07-09T03:06:00.065Z level=INFO source=sched.go:741 msg="new model will fit in available VRAM in single GPU, loading" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa gpu=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0 parallel=4 available=23093903360 required="6.2 GiB"
  411. time=2024-07-09T03:06:00.065Z level=DEBUG source=server.go:98 msg="system memory" total="30.3 GiB" free=27210125312
  412. time=2024-07-09T03:06:00.065Z level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[21.5 GiB]"
  413. time=2024-07-09T03:06:00.065Z level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=33 layers.offload=33 layers.split="" memory.available="[21.5 GiB]" memory.required.full="6.2 GiB" memory.required.partial="6.2 GiB" memory.required.kv="1.0 GiB" memory.required.allocations="[6.2 GiB]" memory.weights.total="4.7 GiB" memory.weights.repeating="4.3 GiB" memory.weights.nonrepeating="411.0 MiB" memory.graph.full="560.0 MiB" memory.graph.partial="677.5 MiB"
  414. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu/ollama_llama_server
  415. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx/ollama_llama_server
  416. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx2/ollama_llama_server
  417. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server
  418. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/rocm_v60101/ollama_llama_server
  419. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu/ollama_llama_server
  420. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx/ollama_llama_server
  421. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cpu_avx2/ollama_llama_server
  422. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server
  423. time=2024-07-09T03:06:00.065Z level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama1577350832/runners/rocm_v60101/ollama_llama_server
  424. time=2024-07-09T03:06:00.065Z level=INFO source=server.go:375 msg="starting llama server" cmd="/tmp/ollama1577350832/runners/cuda_v11/ollama_llama_server --model /root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 33 --verbose --parallel 4 --port 33565"
  425. time=2024-07-09T03:06:00.065Z level=DEBUG source=server.go:390 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LD_LIBRARY_PATH=/tmp/ollama1577350832/runners/cuda_v11:/tmp/ollama1577350832/runners:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 CUDA_VISIBLE_DEVICES=GPU-544bf8b0-7a5c-07f4-b172-733cff9988f0]"
  426. time=2024-07-09T03:06:00.066Z level=INFO source=sched.go:477 msg="loaded runners" count=1
  427. time=2024-07-09T03:06:00.066Z level=INFO source=server.go:563 msg="waiting for llama runner to start responding"
  428. time=2024-07-09T03:06:00.066Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server error"
  429. INFO [main] build info | build=1 commit="a8db2a9" tid="139854183272448" timestamp=1720494360
  430. INFO [main] system info | n_threads=4 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | " tid="139854183272448" timestamp=1720494360 total_threads=8
  431. INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="7" port="33565" tid="139854183272448" timestamp=1720494360
  432. llama_model_loader: loaded meta data with 22 key-value pairs and 291 tensors from /root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa (version GGUF V3 (latest))
  433. llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
  434. llama_model_loader: - kv 0: general.architecture str = llama
  435. llama_model_loader: - kv 1: general.name str = Meta-Llama-3-8B-Instruct
  436. llama_model_loader: - kv 2: llama.block_count u32 = 32
  437. llama_model_loader: - kv 3: llama.context_length u32 = 8192
  438. llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
  439. llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
  440. llama_model_loader: - kv 6: llama.attention.head_count u32 = 32
  441. llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
  442. llama_model_loader: - kv 8: llama.rope.freq_base f32 = 500000.000000
  443. llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
  444. llama_model_loader: - kv 10: general.file_type u32 = 2
  445. llama_model_loader: - kv 11: llama.vocab_size u32 = 128256
  446. llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
  447. llama_model_loader: - kv 13: tokenizer.ggml.model str = gpt2
  448. llama_model_loader: - kv 14: tokenizer.ggml.pre str = llama-bpe
  449. llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,128256] = ["!", "\"", "#", "$", "%", "&", "'", ...
  450. llama_model_loader: - kv 16: tokenizer.ggml.token_type arr[i32,128256] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
  451. time=2024-07-09T03:06:00.317Z level=INFO source=server.go:604 msg="waiting for server to become available" status="llm server loading model"
  452. llama_model_loader: - kv 17: tokenizer.ggml.merges arr[str,280147] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
  453. llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 128000
  454. llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 128009
  455. llama_model_loader: - kv 20: tokenizer.chat_template str = {% set loop_messages = messages %}{% ...
  456. llama_model_loader: - kv 21: general.quantization_version u32 = 2
  457. llama_model_loader: - type f32: 65 tensors
  458. llama_model_loader: - type q4_0: 225 tensors
  459. llama_model_loader: - type q6_K: 1 tensors
  460. llm_load_vocab: special tokens cache size = 256
  461. llm_load_vocab: token to piece cache size = 0.8000 MB
  462. llm_load_print_meta: format = GGUF V3 (latest)
  463. llm_load_print_meta: arch = llama
  464. llm_load_print_meta: vocab type = BPE
  465. llm_load_print_meta: n_vocab = 128256
  466. llm_load_print_meta: n_merges = 280147
  467. llm_load_print_meta: vocab_only = 0
  468. llm_load_print_meta: n_ctx_train = 8192
  469. llm_load_print_meta: n_embd = 4096
  470. llm_load_print_meta: n_layer = 32
  471. llm_load_print_meta: n_head = 32
  472. llm_load_print_meta: n_head_kv = 8
  473. llm_load_print_meta: n_rot = 128
  474. llm_load_print_meta: n_swa = 0
  475. llm_load_print_meta: n_embd_head_k = 128
  476. llm_load_print_meta: n_embd_head_v = 128
  477. llm_load_print_meta: n_gqa = 4
  478. llm_load_print_meta: n_embd_k_gqa = 1024
  479. llm_load_print_meta: n_embd_v_gqa = 1024
  480. llm_load_print_meta: f_norm_eps = 0.0e+00
  481. llm_load_print_meta: f_norm_rms_eps = 1.0e-05
  482. llm_load_print_meta: f_clamp_kqv = 0.0e+00
  483. llm_load_print_meta: f_max_alibi_bias = 0.0e+00
  484. llm_load_print_meta: f_logit_scale = 0.0e+00
  485. llm_load_print_meta: n_ff = 14336
  486. llm_load_print_meta: n_expert = 0
  487. llm_load_print_meta: n_expert_used = 0
  488. llm_load_print_meta: causal attn = 1
  489. llm_load_print_meta: pooling type = 0
  490. llm_load_print_meta: rope type = 0
  491. llm_load_print_meta: rope scaling = linear
  492. llm_load_print_meta: freq_base_train = 500000.0
  493. llm_load_print_meta: freq_scale_train = 1
  494. llm_load_print_meta: n_ctx_orig_yarn = 8192
  495. llm_load_print_meta: rope_finetuned = unknown
  496. llm_load_print_meta: ssm_d_conv = 0
  497. llm_load_print_meta: ssm_d_inner = 0
  498. llm_load_print_meta: ssm_d_state = 0
  499. llm_load_print_meta: ssm_dt_rank = 0
  500. llm_load_print_meta: model type = 8B
  501. llm_load_print_meta: model ftype = Q4_0
  502. llm_load_print_meta: model params = 8.03 B
  503. llm_load_print_meta: model size = 4.33 GiB (4.64 BPW)
  504. llm_load_print_meta: general.name = Meta-Llama-3-8B-Instruct
  505. llm_load_print_meta: BOS token = 128000 '<|begin_of_text|>'
  506. llm_load_print_meta: EOS token = 128009 '<|eot_id|>'
  507. llm_load_print_meta: LF token = 128 'Ä'
  508. llm_load_print_meta: EOT token = 128009 '<|eot_id|>'
  509. llm_load_print_meta: max token length = 256
  510. ggml_cuda_init: GGML_CUDA_FORCE_MMQ: yes
  511. ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
  512. ggml_cuda_init: found 1 CUDA devices:
  513. Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
  514. llm_load_tensors: ggml ctx size = 0.27 MiB
  515. llm_load_tensors: offloading 32 repeating layers to GPU
  516. llm_load_tensors: offloading non-repeating layers to GPU
  517. llm_load_tensors: offloaded 33/33 layers to GPU
  518. llm_load_tensors: CPU buffer size = 281.81 MiB
  519. llm_load_tensors: CUDA0 buffer size = 4155.99 MiB
  520. time=2024-07-09T03:06:01.320Z level=DEBUG source=server.go:615 msg="model load progress 0.06"
  521. time=2024-07-09T03:06:01.571Z level=DEBUG source=server.go:615 msg="model load progress 0.46"
  522. time=2024-07-09T03:06:01.822Z level=DEBUG source=server.go:615 msg="model load progress 0.77"
  523. time=2024-07-09T03:06:02.073Z level=DEBUG source=server.go:615 msg="model load progress 0.99"
  524. llama_new_context_with_model: n_ctx = 8192
  525. llama_new_context_with_model: n_batch = 512
  526. llama_new_context_with_model: n_ubatch = 512
  527. llama_new_context_with_model: flash_attn = 0
  528. llama_new_context_with_model: freq_base = 500000.0
  529. llama_new_context_with_model: freq_scale = 1
  530. llama_kv_cache_init: CUDA0 KV buffer size = 1024.00 MiB
  531. llama_new_context_with_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB
  532. llama_new_context_with_model: CUDA_Host output buffer size = 2.02 MiB
  533. llama_new_context_with_model: CUDA0 compute buffer size = 560.00 MiB
  534. llama_new_context_with_model: CUDA_Host compute buffer size = 24.01 MiB
  535. llama_new_context_with_model: graph nodes = 1030
  536. llama_new_context_with_model: graph splits = 2
  537. time=2024-07-09T03:06:02.324Z level=DEBUG source=server.go:615 msg="model load progress 1.00"
  538. time=2024-07-09T03:06:02.575Z level=DEBUG source=server.go:618 msg="model load completed, waiting for server to become available" status="llm server loading model"
  539. DEBUG [initialize] initializing slots | n_slots=4 tid="139854183272448" timestamp=1720494362
  540. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=0 tid="139854183272448" timestamp=1720494362
  541. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=1 tid="139854183272448" timestamp=1720494362
  542. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=2 tid="139854183272448" timestamp=1720494362
  543. DEBUG [initialize] new slot | n_ctx_slot=2048 slot_id=3 tid="139854183272448" timestamp=1720494362
  544. INFO [main] model loaded | tid="139854183272448" timestamp=1720494362
  545. DEBUG [update_slots] all slots are idle and system prompt is empty, clear the KV cache | tid="139854183272448" timestamp=1720494362
  546. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=0 tid="139854183272448" timestamp=1720494362
  547. time=2024-07-09T03:06:02.829Z level=INFO source=server.go:609 msg="llama runner started in 2.76 seconds"
  548. time=2024-07-09T03:06:02.829Z level=DEBUG source=sched.go:490 msg="finished setting up runner" model=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa
  549. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=1 tid="139854183272448" timestamp=1720494362
  550. DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=37778 status=200 tid="139853712080896" timestamp=1720494362
  551. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=2 tid="139854183272448" timestamp=1720494362
  552. DEBUG [log_server_request] request | method="POST" params={} path="/tokenize" remote_addr="127.0.0.1" remote_port=37778 status=200 tid="139853712080896" timestamp=1720494362
  553. time=2024-07-09T03:06:02.962Z level=DEBUG source=prompt.go:168 msg="prompt now fits in context window" required=50 window=2048
  554. time=2024-07-09T03:06:02.962Z level=DEBUG source=routes.go:1334 msg="chat handler" prompt="<|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nHello! It's nice to meet you. Is there something I can help you with, or would you like to chat?<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat's up<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" images=0
  555. DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=3 tid="139854183272448" timestamp=1720494362
  556. DEBUG [launch_slot_with_data] slot is processing task | slot_id=0 task_id=4 tid="139854183272448" timestamp=1720494362
  557. DEBUG [update_slots] slot progression | ga_i=0 n_past=0 n_past_se=0 n_prompt_tokens_processed=49 slot_id=0 task_id=4 tid="139854183272448" timestamp=1720494362
  558. DEBUG [update_slots] kv cache rm [p0, end) | p0=0 slot_id=0 task_id=4 tid="139854183272448" timestamp=1720494362
  559. DEBUG [print_timings] prompt eval time = 35.35 ms / 49 tokens ( 0.72 ms per token, 1385.98 tokens per second) | n_prompt_tokens_processed=49 n_tokens_second=1385.9817842394073 slot_id=0 t_prompt_processing=35.354 t_token=0.7215102040816327 task_id=4 tid="139854183272448" timestamp=1720494363
  560. DEBUG [print_timings] generation eval time = 395.42 ms / 35 runs ( 11.30 ms per token, 88.51 tokens per second) | n_decoded=35 n_tokens_second=88.51325549224751 slot_id=0 t_token=11.297742857142858 t_token_generation=395.421 task_id=4 tid="139854183272448" timestamp=1720494363
  561. DEBUG [print_timings] total time = 430.77 ms | slot_id=0 t_prompt_processing=35.354 t_token_generation=395.421 t_total=430.775 task_id=4 tid="139854183272448" timestamp=1720494363
  562. DEBUG [update_slots] slot released | n_cache_tokens=84 n_ctx=8192 n_past=83 n_system_tokens=0 slot_id=0 task_id=4 tid="139854183272448" timestamp=1720494363 truncated=false
  563. DEBUG [log_server_request] request | method="POST" params={} path="/completion" remote_addr="127.0.0.1" remote_port=37780 status=200 tid="139853703688192" timestamp=1720494363
  564. [GIN] 2024/07/09 - 03:06:03 | 200 | 4.096197133s | 127.0.0.1 | POST "/api/chat"
  565. time=2024-07-09T03:06:03.398Z level=DEBUG source=sched.go:494 msg="context for request finished"
  566. time=2024-07-09T03:06:03.398Z level=DEBUG source=sched.go:366 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa duration=10m0s
  567. time=2024-07-09T03:06:03.398Z level=DEBUG source=sched.go:384 msg="after processing request finished event" modelPath=/root/.ollama/models/blobs/sha256-6a0746a1ec1aef3e7ec53868f220ff6e389f6f8ef87a01d77c96807de94ca2aa refCount=0
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!