API tools faq
paste
Guest User

sdxl_lora_error

a guest
Oct 29th, 2023
26
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 20.51 KB | None | 0 0
raw download clone embed print report
  1. 17:12:00-648383 INFO Version: v22.0.1
  2.  
  3. 17:12:00-660230 INFO nVidia toolkit detected
  4. 17:12:02-253904 INFO Torch 2.0.1+cu118
  5. 17:12:02-280180 INFO Torch backend: nVidia CUDA 11.8 cuDNN 8700
  6. 17:12:02-295041 INFO Torch detected GPU: NVIDIA GeForce RTX 3090 VRAM 24260
  7. Arch (8, 6) Cores 82
  8. 17:12:02-297403 INFO Verifying modules installation status from
  9. /workspace/kohya_ss/requirements_runpod.txt...
  10. 17:12:02-310830 INFO Verifying modules installation status from
  11. requirements.txt...
  12. 17:12:10-195676 INFO headless: True
  13. 17:12:10-207025 INFO Load CSS...
  14. Running on local URL: http://0.0.0.0:3011
  15.  
  16. To create a public link, set `share=True` in `launch()`.
  17. 17:26:04-447990 INFO Loading config...
  18. 17:37:04-696329 INFO Copy /workspace/SCARLETT to
  19. /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila
  20. morrone woman...
  21. 17:37:05-214138 INFO Copy /workspace/WOMAN to
  22. /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman...
  23. 17:37:16-843933 INFO Done creating kohya_ss training folder structure at
  24. /workspace/SCARLETT-CAMILA_MORRONE...
  25. 17:39:16-467957 INFO Start training LoRA Standard ...
  26. 17:39:16-472754 INFO Checking for duplicate image filenames in training data
  27. directory...
  28. 17:39:16-479060 INFO Valid image folder names found in:
  29. /workspace/SCARLETT-CAMILA_MORRONE/img
  30. 17:39:16-484032 INFO Valid image folder names found in:
  31. /workspace/SCARLETT-CAMILA_MORRONE/reg
  32. 17:39:16-486844 INFO Headless mode, skipping verification if model already
  33. exist... if model already exist it will be
  34. overwritten...
  35. 17:39:16-491651 INFO Folder 25_camila morrone woman: 53 images found
  36. 17:39:16-494328 INFO Folder 25_camila morrone woman: 1325 steps
  37. 17:39:16-497702 WARNING Regularisation images are used... Will double the
  38. number of steps required...
  39. 17:39:16-503442 INFO Total steps: 1325
  40. 17:39:16-508258 INFO Train batch size: 1
  41. 17:39:16-510891 INFO Gradient accumulation steps: 1
  42. 17:39:16-513517 INFO Epoch: 10
  43. 17:39:16-515946 INFO Regulatization factor: 2
  44. 17:39:16-518322 INFO max_train_steps (1325 / 1 / 1 * 10 * 2) = 26500
  45. 17:39:16-521131 INFO stop_text_encoder_training = 0
  46. 17:39:16-523630 INFO lr_warmup_steps = 0
  47. 17:39:16-526828 INFO Saving training config to
  48. /workspace/SCARLETT-CAMILA_MORRONE/model/SCARLETT-CAMIL
  49. A_MORRONE_20231029-173916.json...
  50. 17:39:16-535180 INFO accelerate launch --num_cpu_threads_per_process=2
  51. "./sdxl_train_network.py" --enable_bucket
  52. --min_bucket_reso=256 --max_bucket_reso=2048
  53. --pretrained_model_name_or_path="/workspace/stable-diff
  54. usion-webui/models/Stable-diffusion/sd_xl_base_1.0_0.9v
  55. ae.safetensors"
  56. --train_data_dir="/workspace/SCARLETT-CAMILA_MORRONE/im
  57. g"
  58. --reg_data_dir="/workspace/SCARLETT-CAMILA_MORRONE/reg"
  59. --resolution="1024,1024"
  60. --output_dir="/workspace/SCARLETT-CAMILA_MORRONE/model"
  61. --logging_dir="/workspace/SCARLETT-CAMILA_MORRONE/log"
  62. --network_alpha="1" --save_model_as=safetensors
  63. --network_module=networks.lora --text_encoder_lr=0.0003
  64. --unet_lr=0.0003 --network_dim=256
  65. --output_name="SCARLETT-CAMILA_MORRONE"
  66. --lr_scheduler_num_cycles="10" --no_half_vae
  67. --learning_rate="0.0003" --lr_scheduler="constant"
  68. --train_batch_size="1" --max_train_steps="26500"
  69. --save_every_n_epochs="1" --mixed_precision="bf16"
  70. --save_precision="bf16" --caption_extension=".txt"
  71. --cache_latents --cache_latents_to_disk
  72. --optimizer_type="Adafactor" --optimizer_args
  73. scale_parameter=False relative_step=False
  74. warmup_init=False --max_data_loader_n_workers="0"
  75. --bucket_reso_steps=64 --gradient_checkpointing
  76. --xformers --bucket_no_upscale --noise_offset=0.0
  77. 2023-10-29 17:39:30.632691: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  78. 2023-10-29 17:39:30.632948: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  79. 2023-10-29 17:39:30.639821: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  80. 2023-10-29 17:39:31.077390: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
  81. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
  82. 2023-10-29 17:39:36.617810: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
  83. prepare tokenizers
  84.  
  85. Downloading (…)olve/main/vocab.json: 0%| | 0.00/862k [00:00<?, ?B/s]
  86. Downloading (…)olve/main/vocab.json: 100%|██████████| 862k/862k [00:00<00:00, 2.10MB/s]
  87. Downloading (…)olve/main/vocab.json: 100%|██████████| 862k/862k [00:00<00:00, 2.09MB/s]
  88.  
  89. Downloading (…)olve/main/merges.txt: 0%| | 0.00/525k [00:00<?, ?B/s]
  90. Downloading (…)olve/main/merges.txt: 100%|██████████| 525k/525k [00:00<00:00, 1.69MB/s]
  91. Downloading (…)olve/main/merges.txt: 100%|██████████| 525k/525k [00:00<00:00, 1.69MB/s]
  92.  
  93. Downloading (…)cial_tokens_map.json: 0%| | 0.00/389 [00:00<?, ?B/s]
  94. Downloading (…)cial_tokens_map.json: 100%|██████████| 389/389 [00:00<00:00, 1.86MB/s]
  95.  
  96. Downloading (…)okenizer_config.json: 0%| | 0.00/904 [00:00<?, ?B/s]
  97. Downloading (…)okenizer_config.json: 100%|██████████| 904/904 [00:00<00:00, 4.69MB/s]
  98. Using DreamBooth method.
  99. prepare images.
  100. found directory /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman contains 53 image files
  101. No caption file found for 53 images. Training will continue without captions for these images. If class token exists, it will be used. / 53枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。
  102. /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman/SCARLETT_test1_face\MissScarlett23 - 1625852000322867200 - 3_1625850588486574088.jpg
  103. /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman/SCARLETT_test1_face\MissScarlett23 - 1625958978705412096 - 3_1625958652002664455.jpg
  104. /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman/SCARLETT_test1_face\MissScarlett23 - 1626160233855737856 - 3_1361587561626554368.jpg
  105. /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman/SCARLETT_test1_face\MissScarlett23 - 1628674856437809152 - 3_1496411943132930061.jpg
  106. /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman/SCARLETT_test1_face\MissScarlett23 - 1629102572203122688 - 3_1496831840203816964.jpg
  107. /workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman/SCARLETT_test1_face\MissScarlett23 - 1630126904459100161 - 3_1630126204647882753.jpg... and 48 more
  108. found directory /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman contains 1000 image files
  109. No caption file found for 1000 images. Training will continue without captions for these images. If class token exists, it will be used. / 1000枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。
  110. /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman/1024_WOMAN\woman_0001.jpg
  111. /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman/1024_WOMAN\woman_0002.jpg
  112. /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman/1024_WOMAN\woman_0003.jpg
  113. /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman/1024_WOMAN\woman_0004.jpg
  114. /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman/1024_WOMAN\woman_0005.jpg
  115. /workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman/1024_WOMAN\woman_0006.jpg... and 995 more
  116. 1325 train images with repeating.
  117. 1000 reg images.
  118. [Dataset 0]
  119. batch_size: 1
  120. resolution: (1024, 1024)
  121. enable_bucket: True
  122. min_bucket_reso: 256
  123. max_bucket_reso: 2048
  124. bucket_reso_steps: 64
  125. bucket_no_upscale: True
  126.  
  127. [Subset 0 of Dataset 0]
  128. image_dir: "/workspace/SCARLETT-CAMILA_MORRONE/img/25_camila morrone woman"
  129. image_count: 53
  130. num_repeats: 25
  131. shuffle_caption: False
  132. keep_tokens: 0
  133. caption_dropout_rate: 0.0
  134. caption_dropout_every_n_epoches: 0
  135. caption_tag_dropout_rate: 0.0
  136. caption_prefix: None
  137. caption_suffix: None
  138. color_aug: False
  139. flip_aug: False
  140. face_crop_aug_range: None
  141. random_crop: False
  142. token_warmup_min: 1,
  143. token_warmup_step: 0,
  144. is_reg: False
  145. class_tokens: camila morrone woman
  146. caption_extension: .txt
  147.  
  148. [Subset 1 of Dataset 0]
  149. image_dir: "/workspace/SCARLETT-CAMILA_MORRONE/reg/1_woman"
  150. image_count: 1000
  151. num_repeats: 1
  152. shuffle_caption: False
  153. keep_tokens: 0
  154. caption_dropout_rate: 0.0
  155. caption_dropout_every_n_epoches: 0
  156. caption_tag_dropout_rate: 0.0
  157. caption_prefix: None
  158. caption_suffix: None
  159. color_aug: False
  160. flip_aug: False
  161. face_crop_aug_range: None
  162. random_crop: False
  163. token_warmup_min: 1,
  164. token_warmup_step: 0,
  165. is_reg: True
  166. class_tokens: woman
  167. caption_extension: .txt
  168.  
  169.  
  170. [Dataset 0]
  171. loading image sizes.
  172.  
  173. 0%| | 0/1053 [00:00<?, ?it/s]
  174. 100%|██████████| 1053/1053 [00:02<00:00, 383.33it/s]
  175. make buckets
  176. min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます
  177. number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
  178. bucket 0: resolution (576, 576), count: 25
  179. bucket 1: resolution (576, 1024), count: 25
  180. bucket 2: resolution (640, 1152), count: 125
  181. bucket 3: resolution (832, 1088), count: 750
  182. bucket 4: resolution (896, 1088), count: 300
  183. bucket 5: resolution (896, 1152), count: 25
  184. bucket 6: resolution (1024, 1024), count: 1350
  185. bucket 7: resolution (1088, 832), count: 25
  186. bucket 8: resolution (1152, 640), count: 25
  187. mean ar error (without repeats): 0.0007962156055837949
  188. preparing accelerator
  189. loading model for process 0/1
  190. load StableDiffusion checkpoint: /workspace/stable-diffusion-webui/models/Stable-diffusion/sd_xl_base_1.0_0.9vae.safetensors
  191. building U-Net
  192. loading U-Net from checkpoint
  193. U-Net: <All keys matched successfully>
  194. building text encoders
  195. loading text encoders from checkpoint
  196. text encoder 1: <All keys matched successfully>
  197. text encoder 2: <All keys matched successfully>
  198. building VAE
  199. loading VAE from checkpoint
  200. VAE: <All keys matched successfully>
  201. Enable xformers for U-Net
  202. import network module: networks.lora
  203. [Dataset 0]
  204. caching latents.
  205. checking cache validity...
  206.  
  207. 0%| | 0/1053 [00:00<?, ?it/s]
  208. 100%|██████████| 1053/1053 [00:00<00:00, 5177.35it/s]
  209. caching latents...
  210.  
  211. 0%| | 0/1053 [00:00<?, ?it/s]
  212. 100%|██████████| 1053/1053 [05:39<00:00, 3.10it/s]
  213. create LoRA network. base dim (rank): 256, alpha: 1.0
  214. neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
  215. create LoRA for Text Encoder 1:
  216. create LoRA for Text Encoder 2:
  217. create LoRA for Text Encoder: 264 modules.
  218. create LoRA for U-Net: 722 modules.
  219. enable LoRA for text encoder
  220. enable LoRA for U-Net
  221. prepare optimizer, data loader etc.
  222. use Adafactor optimizer | {'scale_parameter': False, 'relative_step': False, 'warmup_init': False}
  223. because max_grad_norm is set, clip_grad_norm is enabled. consider set to 0 / max_grad_normが設定されているためclip_grad_normが有効になります。0に設定して無効にしたほうがいいかもしれません
  224. constant_with_warmup will be good / スケジューラはconstant_with_warmupが良いかもしれません
  225. running training / 学習開始
  226. num train images * repeats / 学習画像の数×繰り返し回数: 1325
  227. num reg images / 正則化画像の数: 1000
  228. num batches per epoch / 1epochのバッチ数: 2650
  229. num epochs / epoch数: 10
  230. batch size per device / バッチサイズ: 1
  231. gradient accumulation steps / 勾配を合計するステップ数 = 1
  232. total optimization steps / 学習ステップ数: 26500
  233.  
  234. steps: 0%| | 0/26500 [00:00<?, ?it/s]
  235. epoch 1/10
  236. Traceback (most recent call last):
  237. File "/workspace/kohya_ss/./sdxl_train_network.py", line 183, in <module>
  238. trainer.train(args)
  239. File "/workspace/kohya_ss/train_network.py", line 792, in train
  240. noise_pred = self.call_unet(
  241. File "/workspace/kohya_ss/./sdxl_train_network.py", line 163, in call_unet
  242. noise_pred = unet(noisy_latents, timesteps, text_embedding, vector_embedding)
  243. File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  244. return forward_call(*args, **kwargs)
  245. File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 636, in forward
  246. return model_forward(*args, **kwargs)
  247. File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 624, in __call__
  248. return convert_to_fp32(self.model_forward(*args, **kwargs))
  249. File "/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
  250. return func(*args, **kwargs)
  251. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 1088, in forward
  252. h = call_module(module, h, emb, context)
  253. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 1073, in call_module
  254. x = layer(x, context)
  255. File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  256. return forward_call(*args, **kwargs)
  257. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 728, in forward
  258. hidden_states = block(hidden_states, context=encoder_hidden_states, timestep=timestep)
  259. File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  260. return forward_call(*args, **kwargs)
  261. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 647, in forward
  262. output = torch.utils.checkpoint.checkpoint(
  263. File "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py", line 249, in checkpoint
  264. return CheckpointFunction.apply(function, preserve, *args)
  265. File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 506, in apply
  266. return super().apply(*args, **kwargs) # type: ignore[misc]
  267. File "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py", line 107, in forward
  268. outputs = run_function(*args)
  269. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 643, in custom_forward
  270. return func(*inputs)
  271. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 633, in forward_body
  272. hidden_states = self.ff(self.norm3(hidden_states)) + hidden_states
  273. File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  274. return forward_call(*args, **kwargs)
  275. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 577, in forward
  276. hidden_states = module(hidden_states)
  277. File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  278. return forward_call(*args, **kwargs)
  279. File "/workspace/kohya_ss/library/sdxl_original_unet.py", line 555, in forward
  280. hidden_states, gate = self.proj(hidden_states).chunk(2, dim=-1)
  281. File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
  282. return forward_call(*args, **kwargs)
  283. File "/workspace/kohya_ss/networks/lora.py", line 117, in forward
  284. return org_forwarded + lx * self.multiplier * scale
  285. torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 18.00 MiB (GPU 0; 23.69 GiB total capacity; 12.75 GiB already allocated; 12.69 MiB free; 12.90 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  286.  
  287. steps: 0%| | 0/26500 [00:00<?, ?it/s]
  288. Traceback (most recent call last):
  289. File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
  290. sys.exit(main())
  291. File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
  292. args.func(args)
  293. File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 986, in launch_command
  294. simple_launcher(args)
  295. File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 628, in simple_launcher
  296. raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
  297. subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python3', './sdxl_train_network.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=/workspace/stable-diffusion-webui/models/Stable-diffusion/sd_xl_base_1.0_0.9vae.safetensors', '--train_data_dir=/workspace/SCARLETT-CAMILA_MORRONE/img', '--reg_data_dir=/workspace/SCARLETT-CAMILA_MORRONE/reg', '--resolution=1024,1024', '--output_dir=/workspace/SCARLETT-CAMILA_MORRONE/model', '--logging_dir=/workspace/SCARLETT-CAMILA_MORRONE/log', '--network_alpha=1', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=0.0003', '--unet_lr=0.0003', '--network_dim=256', '--output_name=SCARLETT-CAMILA_MORRONE', '--lr_scheduler_num_cycles=10', '--no_half_vae', '--learning_rate=0.0003', '--lr_scheduler=constant', '--train_batch_size=1', '--max_train_steps=26500', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16', '--caption_extension=.txt', '--cache_latents', '--cache_latents_to_disk', '--optimizer_type=Adafactor', '--optimizer_args', 'scale_parameter=False', 'relative_step=False', 'warmup_init=False', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--gradient_checkpointing', '--xformers', '--bucket_no_upscale', '--noise_offset=0.0']' returned non-zero exit status 1.
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!