kohya_ss.log

                                                                                                                                                                               20:58:16-892180 INFO     Start training LoRA Standard ...
20:58:16-893710 INFO     Checking for duplicate image filenames in training data
                         directory...
20:58:16-895126 INFO     Valid image folder names found in:
                         /workspace/Salida/img
20:58:16-896272 INFO     Valid image folder names found in:
                         /workspace/Salida/reg
20:58:16-897397 INFO     Headless mode, skipping verification if model already
                         exist... if model already exist it will be
                         overwritten...
20:58:16-898666 INFO     Folder 20_victorahumada man: 21 images found
20:58:16-899658 INFO     Folder 20_victorahumada man: 420 steps
20:58:16-900698 WARNING  Regularisation images are used... Will double the
                         number of steps required...
20:58:16-902043 INFO     Total steps: 420
20:58:16-903153 INFO     Train batch size: 1
20:58:16-904249 INFO     Gradient accumulation steps: 1
20:58:16-905428 INFO     Epoch: 10
20:58:16-906524 INFO     Regulatization factor: 2
20:58:16-907636 INFO     max_train_steps (420 / 1 / 1 * 10 * 2) = 8400
20:58:16-909052 INFO     stop_text_encoder_training = 0
20:58:16-910235 INFO     lr_warmup_steps = 0
20:58:16-911500 INFO     Saving training config to
                         /workspace/Salida/model/victorahumada_20231219-205816.j
                         son...
20:58:16-913270 INFO     accelerate launch --num_cpu_threads_per_process=2
                         "./sdxl_train_network.py" --enable_bucket
                         --min_bucket_reso=256 --max_bucket_reso=2048
                         --pretrained_model_name_or_path="/workspace/stable-diff
                         usion-webui/models/Stable-diffusion/sd_xl_base_1.0.safe
                         tensors" --train_data_dir="/workspace/Salida/img"
                         --reg_data_dir="/workspace/Salida/reg"
                         --resolution="1024,1024"
                         --output_dir="/workspace/Salida/model"
                         --logging_dir="/workspace/Salida/log"
                         --network_alpha="1" --save_model_as=safetensors
                         --network_module=networks.lora --text_encoder_lr=0.0003
                         --unet_lr=0.0003 --network_dim=256
                         --output_name="victorahumada"
                         --lr_scheduler_num_cycles="10" --no_half_vae
                         --learning_rate="0.0003" --lr_scheduler="constant"
                         --train_batch_size="1" --max_train_steps="8400"
                         --save_every_n_epochs="1" --mixed_precision="bf16"
                         --save_precision="bf16" --caption_extension=".txt"
                         --cache_latents --cache_latents_to_disk
                         --optimizer_type="Adafactor" --optimizer_args
                         scale_parameter=False relative_step=False
                         warmup_init=False --max_data_loader_n_workers="0"
                         --bucket_reso_steps=64 --gradient_checkpointing
                         --xformers --bucket_no_upscale --noise_offset=0.0
2023-12-19 20:58:22.461886: I tensorflow/core/util/port.cc:111] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-12-19 20:58:22.517911: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2023-12-19 20:58:22.517965: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2023-12-19 20:58:22.518014: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-12-19 20:58:22.529595: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-12-19 20:58:23.626118: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
prepare tokenizers
Using DreamBooth method.
prepare images.
found directory /workspace/Salida/img/20_victorahumada man contains 21 image files
No caption file found for 21 images. Training will continue without captions for these images. If class token exists, it will be used. / 21枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。
/workspace/Salida/img/20_victorahumada man/victor (1).jpg
/workspace/Salida/img/20_victorahumada man/victor (10).JPG
/workspace/Salida/img/20_victorahumada man/victor (11).JPG
/workspace/Salida/img/20_victorahumada man/victor (12).JPG
/workspace/Salida/img/20_victorahumada man/victor (13).JPG
/workspace/Salida/img/20_victorahumada man/victor (14).JPG... and 16 more
found directory /workspace/Salida/reg/1_man contains 1000 image files
No caption file found for 1000 images. Training will continue without captions for these images. If class token exists, it will be used. / 1000枚の画像にキャプションファイルが見つかりませんでした。これらの画像についてはキャプションなしで学習を続行します。class tokenが存在する場合はそれを使います。
/workspace/Salida/reg/1_man/man_0001.jpg
/workspace/Salida/reg/1_man/man_0002.jpg
/workspace/Salida/reg/1_man/man_0003.jpg
/workspace/Salida/reg/1_man/man_0004.jpg
/workspace/Salida/reg/1_man/man_0005.jpg
/workspace/Salida/reg/1_man/man_0006.jpg... and 995 more
420 train images with repeating.
1000 reg images.
some of reg images are not used / 正則化画像の数が多いので、一部使用されない正則化画像があります
[Dataset 0]
  batch_size: 1
  resolution: (1024, 1024)
  enable_bucket: True
  min_bucket_reso: 256
  max_bucket_reso: 2048
  bucket_reso_steps: 64
  bucket_no_upscale: True

  [Subset 0 of Dataset 0]
    image_dir: "/workspace/Salida/img/20_victorahumada man"
    image_count: 21
    num_repeats: 20
    shuffle_caption: False
    keep_tokens: 0
    caption_dropout_rate: 0.0
    caption_dropout_every_n_epoches: 0
    caption_tag_dropout_rate: 0.0
    caption_prefix: None
    caption_suffix: None
    color_aug: False
    flip_aug: False
    face_crop_aug_range: None
    random_crop: False
    token_warmup_min: 1,
    token_warmup_step: 0,
    is_reg: False
    class_tokens: victorahumada man
    caption_extension: .txt

  [Subset 1 of Dataset 0]
    image_dir: "/workspace/Salida/reg/1_man"
    image_count: 1000
    num_repeats: 1
    shuffle_caption: False
    keep_tokens: 0
    caption_dropout_rate: 0.0
    caption_dropout_every_n_epoches: 0
    caption_tag_dropout_rate: 0.0
    caption_prefix: None
    caption_suffix: None
    color_aug: False
    flip_aug: False
    face_crop_aug_range: None
    random_crop: False
    token_warmup_min: 1,
    token_warmup_step: 0,
    is_reg: True
    class_tokens: man
    caption_extension: .txt


[Dataset 0]
loading image sizes.

  0%|          | 0/441 [00:00<?, ?it/s]
100%|██████████| 441/441 [00:00<00:00, 8442.21it/s]
make buckets
min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is set, because bucket reso is defined by image size automatically / bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計算されるため、min_bucket_resoとmax_bucket_resoは無視されます
number of images (including repeats) / 各bucketの画像枚数（繰り返し回数を含む）
bucket 0: resolution (448, 640), count: 20
bucket 1: resolution (640, 448), count: 80
bucket 2: resolution (1024, 1024), count: 420
bucket 3: resolution (1088, 832), count: 280
bucket 4: resolution (1216, 832), count: 40
mean ar error (without repeats): 0.00195763833959853
preparing accelerator
loading model for process 0/1
load StableDiffusion checkpoint: /sd-models/sd_xl_base_1.0.safetensors
building U-Net
loading U-Net from checkpoint
U-Net:  <All keys matched successfully>
building text encoders
loading text encoders from checkpoint
text encoder 1: <All keys matched successfully>
text encoder 2: <All keys matched successfully>
building VAE
loading VAE from checkpoint
VAE: <All keys matched successfully>
Enable xformers for U-Net
import network module: networks.lora
[Dataset 0]
caching latents.
checking cache validity...

  0%|          | 0/441 [00:00<?, ?it/s]
100%|██████████| 441/441 [00:00<00:00, 159373.43it/s]
caching latents...

  0%|          | 0/441 [00:00<?, ?it/s]
  0%|          | 1/441 [00:00<01:37,  4.53it/s]
  1%|          | 3/441 [00:00<00:34, 12.81it/s]
  1%|          | 5/441 [00:00<00:34, 12.51it/s]
  2%|▏         | 7/441 [00:01<02:01,  3.56it/s]
  2%|▏         | 8/441 [00:01<01:55,  3.75it/s]
  2%|▏         | 9/441 [00:01<02:18,  3.12it/s]
  2%|▏         | 10/441 [00:01<02:01,  3.55it/s]
  2%|▏         | 11/441 [00:02<02:00,  3.57it/s]
  3%|▎         | 12/441 [00:02<01:59,  3.58it/s]
  3%|▎         | 13/441 [00:02<02:00,  3.55it/s]
  3%|▎         | 14/441 [00:03<01:59,  3.57it/s]
  3%|▎         | 15/441 [00:03<01:59,  3.58it/s]
  4%|▎         | 16/441 [00:03<02:00,  3.52it/s]
  4%|▍         | 17/441 [00:03<01:57,  3.61it/s]
  4%|▍         | 18/441 [00:04<02:03,  3.41it/s]
  4%|▍         | 19/441 [00:04<02:00,  3.49it/s]
  5%|▍         | 20/441 [00:04<02:16,  3.07it/s]
  5%|▍         | 21/441 [00:05<02:32,  2.75it/s]
  5%|▍         | 22/441 [00:05<02:13,  3.13it/s]
  5%|▌         | 23/441 [00:05<02:04,  3.37it/s]
  5%|▌         | 24/441 [00:06<02:10,  3.19it/s]
  6%|▌         | 25/441 [00:06<02:10,  3.20it/s]
  6%|▌         | 26/441 [00:06<02:03,  3.36it/s]
  6%|▌         | 27/441 [00:07<02:07,  3.25it/s]
  6%|▋         | 28/441 [00:07<02:02,  3.37it/s]
  7%|▋         | 29/441 [00:07<02:08,  3.21it/s]
  7%|▋         | 30/441 [00:07<02:08,  3.19it/s]
  7%|▋         | 31/441 [00:08<02:01,  3.39it/s]
  7%|▋         | 32/441 [00:08<02:07,  3.21it/s]
  7%|▋         | 33/441 [00:08<02:07,  3.21it/s]
  8%|▊         | 34/441 [00:09<02:00,  3.38it/s]
  8%|▊         | 35/441 [00:09<02:05,  3.23it/s]
  8%|▊         | 36/441 [00:09<02:03,  3.28it/s]
  8%|▊         | 37/441 [00:10<02:00,  3.36it/s]
  9%|▊         | 38/441 [00:10<02:07,  3.16it/s]
  9%|▉         | 39/441 [00:10<02:01,  3.30it/s]
  9%|▉         | 40/441 [00:11<01:59,  3.36it/s]
  9%|▉         | 41/441 [00:11<02:02,  3.26it/s]
 10%|▉         | 42/441 [00:11<02:06,  3.17it/s]
 10%|▉         | 43/441 [00:11<01:59,  3.34it/s]
 10%|▉         | 44/441 [00:12<02:04,  3.19it/s]
 10%|█         | 45/441 [00:12<02:06,  3.12it/s]
 10%|█         | 46/441 [00:12<01:57,  3.37it/s]
 11%|█         | 47/441 [00:13<02:05,  3.13it/s]
 11%|█         | 48/441 [00:13<02:03,  3.19it/s]
 11%|█         | 49/441 [00:13<01:57,  3.33it/s]
 11%|█▏        | 50/441 [00:14<02:02,  3.19it/s]
 12%|█▏        | 51/441 [00:14<02:02,  3.19it/s]
 12%|█▏        | 52/441 [00:14<01:57,  3.32it/s]
 12%|█▏        | 53/441 [00:15<02:00,  3.23it/s]
 12%|█▏        | 54/441 [00:15<02:00,  3.20it/s]
 12%|█▏        | 55/441 [00:15<01:55,  3.34it/s]
 13%|█▎        | 56/441 [00:15<01:58,  3.25it/s]
 13%|█▎        | 57/441 [00:16<02:01,  3.16it/s]
 13%|█▎        | 58/441 [00:16<01:53,  3.39it/s]
 13%|█▎        | 59/441 [00:16<02:03,  3.09it/s]
 14%|█▎        | 60/441 [00:17<02:01,  3.13it/s]
 14%|█▍        | 61/441 [00:17<01:53,  3.36it/s]
 14%|█▍        | 62/441 [00:17<01:59,  3.16it/s]
 14%|█▍        | 63/441 [00:18<01:59,  3.18it/s]
 15%|█▍        | 64/441 [00:18<01:52,  3.34it/s]
 15%|█▍        | 65/441 [00:18<02:00,  3.12it/s]
 15%|█▍        | 66/441 [00:19<01:58,  3.17it/s]
 15%|█▌        | 67/441 [00:19<01:52,  3.32it/s]
 15%|█▌        | 68/441 [00:19<01:59,  3.13it/s]
 16%|█▌        | 69/441 [00:20<01:58,  3.13it/s]
 16%|█▌        | 70/441 [00:20<01:51,  3.32it/s]
 16%|█▌        | 71/441 [00:20<01:56,  3.18it/s]
 16%|█▋        | 72/441 [00:20<01:53,  3.25it/s]
 17%|█▋        | 73/441 [00:21<01:50,  3.32it/s]
 17%|█▋        | 74/441 [00:21<01:57,  3.13it/s]
 17%|█▋        | 75/441 [00:21<01:55,  3.16it/s]
 17%|█▋        | 76/441 [00:22<01:49,  3.33it/s]
 17%|█▋        | 77/441 [00:22<01:53,  3.21it/s]
 18%|█▊        | 78/441 [00:22<01:58,  3.07it/s]
 18%|█▊        | 79/441 [00:23<01:48,  3.33it/s]
 18%|█▊        | 80/441 [00:23<01:51,  3.24it/s]
 18%|█▊        | 81/441 [00:23<01:53,  3.16it/s]
 19%|█▊        | 82/441 [00:24<01:46,  3.37it/s]
 19%|█▉        | 83/441 [00:24<01:55,  3.11it/s]
 19%|█▉        | 84/441 [00:24<01:53,  3.15it/s]
 19%|█▉        | 85/441 [00:24<01:46,  3.35it/s]
 20%|█▉        | 86/441 [00:25<01:50,  3.21it/s]
 20%|█▉        | 87/441 [00:25<01:53,  3.11it/s]
 20%|█▉        | 88/441 [00:25<01:45,  3.33it/s]
 20%|██        | 89/441 [00:26<01:53,  3.10it/s]
 20%|██        | 90/441 [00:26<01:51,  3.16it/s]
 21%|██        | 91/441 [00:26<01:46,  3.29it/s]
 21%|██        | 92/441 [00:27<01:50,  3.17it/s]
 21%|██        | 93/441 [00:27<01:50,  3.15it/s]
 21%|██▏       | 94/441 [00:27<01:48,  3.19it/s]
 22%|██▏       | 95/441 [00:28<01:49,  3.17it/s]
 22%|██▏       | 96/441 [00:28<01:44,  3.30it/s]
 22%|██▏       | 97/441 [00:28<01:49,  3.15it/s]
 22%|██▏       | 98/441 [00:29<01:45,  3.24it/s]
 22%|██▏       | 99/441 [00:29<01:41,  3.36it/s]
 23%|██▎       | 100/441 [00:29<01:46,  3.19it/s]
 23%|██▎       | 101/441 [00:29<01:46,  3.19it/s]
 23%|██▎       | 102/441 [00:30<01:41,  3.34it/s]
 23%|██▎       | 103/441 [00:30<01:46,  3.17it/s]
 24%|██▎       | 104/441 [00:30<01:47,  3.13it/s]
 24%|██▍       | 105/441 [00:31<01:40,  3.33it/s]
 24%|██▍       | 106/441 [00:31<01:45,  3.18it/s]
 24%|██▍       | 107/441 [00:31<01:45,  3.16it/s]
 24%|██▍       | 108/441 [00:32<01:41,  3.30it/s]
 25%|██▍       | 109/441 [00:32<01:45,  3.16it/s]
 25%|██▍       | 110/441 [00:32<01:42,  3.24it/s]
 25%|██▌       | 111/441 [00:33<01:43,  3.20it/s]
 25%|██▌       | 112/441 [00:33<01:43,  3.18it/s]
 26%|██▌       | 113/441 [00:33<01:38,  3.32it/s]
 26%|██▌       | 114/441 [00:34<01:43,  3.17it/s]
 26%|██▌       | 115/441 [00:34<01:42,  3.19it/s]
 26%|██▋       | 116/441 [00:34<01:36,  3.35it/s]
 27%|██▋       | 117/441 [00:34<01:42,  3.15it/s]
 27%|██▋       | 118/441 [00:35<01:40,  3.23it/s]
 27%|██▋       | 119/441 [00:35<01:36,  3.34it/s]
 27%|██▋       | 120/441 [00:35<01:42,  3.14it/s]
 27%|██▋       | 121/441 [00:36<01:44,  3.07it/s]
 28%|██▊       | 122/441 [00:36<01:37,  3.28it/s]
 28%|██▊       | 123/441 [00:36<01:38,  3.23it/s]
 28%|██▊       | 124/441 [00:37<01:38,  3.22it/s]
 28%|██▊       | 125/441 [00:37<01:33,  3.37it/s]
 29%|██▊       | 126/441 [00:37<01:38,  3.21it/s]
 29%|██▉       | 127/441 [00:38<01:42,  3.07it/s]
 29%|██▉       | 128/441 [00:38<01:33,  3.35it/s]
 29%|██▉       | 129/441 [00:38<01:36,  3.24it/s]
 29%|██▉       | 130/441 [00:38<01:35,  3.27it/s]
 30%|██▉       | 131/441 [00:39<01:32,  3.37it/s]
 30%|██▉       | 132/441 [00:39<01:37,  3.18it/s]
 30%|███       | 133/441 [00:39<01:38,  3.14it/s]
 30%|███       | 134/441 [00:40<01:32,  3.32it/s]
 31%|███       | 135/441 [00:40<01:37,  3.13it/s]
 31%|███       | 136/441 [00:40<01:35,  3.19it/s]
 31%|███       | 137/441 [00:41<01:35,  3.17it/s]
 31%|███▏      | 138/441 [00:41<01:34,  3.20it/s]
 32%|███▏      | 139/441 [00:41<01:31,  3.31it/s]
 32%|███▏      | 140/441 [00:42<01:35,  3.15it/s]
 32%|███▏      | 141/441 [00:42<01:34,  3.17it/s]
 32%|███▏      | 142/441 [00:42<01:33,  3.20it/s]
 32%|███▏      | 143/441 [00:43<01:32,  3.21it/s]
 33%|███▎      | 144/441 [00:43<01:30,  3.30it/s]
 33%|███▎      | 145/441 [00:43<01:32,  3.20it/s]
 33%|███▎      | 146/441 [00:43<01:33,  3.16it/s]
 33%|███▎      | 147/441 [00:44<01:29,  3.28it/s]
 34%|███▎      | 148/441 [00:44<01:30,  3.24it/s]
 34%|███▍      | 149/441 [00:44<01:31,  3.20it/s]
 34%|███▍      | 150/441 [00:45<01:27,  3.32it/s]
 34%|███▍      | 151/441 [00:45<01:30,  3.21it/s]
 34%|███▍      | 152/441 [00:45<01:25,  3.37it/s]
 35%|███▍      | 153/441 [00:46<01:31,  3.14it/s]
 35%|███▍      | 154/441 [00:46<01:31,  3.14it/s]
 35%|███▌      | 155/441 [00:46<01:25,  3.33it/s]
 35%|███▌      | 156/441 [00:47<01:29,  3.17it/s]
 36%|███▌      | 157/441 [00:47<01:27,  3.24it/s]
 36%|███▌      | 158/441 [00:47<01:24,  3.34it/s]
 36%|███▌      | 159/441 [00:47<01:31,  3.09it/s]
 36%|███▋      | 160/441 [00:48<01:31,  3.08it/s]
 37%|███▋      | 161/441 [00:48<01:24,  3.32it/s]
 37%|███▋      | 162/441 [00:48<01:25,  3.26it/s]
 37%|███▋      | 163/441 [00:49<01:23,  3.33it/s]
 37%|███▋      | 164/441 [00:49<01:27,  3.18it/s]
 37%|███▋      | 165/441 [00:49<01:31,  3.01it/s]
 38%|███▊      | 166/441 [00:50<01:23,  3.30it/s]
 38%|███▊      | 167/441 [00:50<01:25,  3.19it/s]
 38%|███▊      | 168/441 [00:50<01:25,  3.20it/s]
 38%|███▊      | 169/441 [00:51<01:24,  3.23it/s]
 39%|███▊      | 170/441 [00:51<01:26,  3.12it/s]
 39%|███▉      | 171/441 [00:51<01:20,  3.35it/s]
 39%|███▉      | 172/441 [00:52<01:30,  2.99it/s]
 39%|███▉      | 173/441 [00:52<01:23,  3.21it/s]
 39%|███▉      | 174/441 [00:52<01:20,  3.32it/s]
 40%|███▉      | 175/441 [00:52<01:24,  3.15it/s]
 40%|███▉      | 176/441 [00:53<01:24,  3.12it/s]
 40%|████      | 177/441 [00:53<01:19,  3.30it/s]
 40%|████      | 178/441 [00:53<01:23,  3.16it/s]
 41%|████      | 179/441 [00:54<01:23,  3.15it/s]
 41%|████      | 180/441 [00:54<01:20,  3.25it/s]
 41%|████      | 181/441 [00:54<01:21,  3.20it/s]
 41%|████▏     | 182/441 [00:55<01:20,  3.22it/s]
 41%|████▏     | 183/441 [00:55<01:16,  3.35it/s]
 42%|████▏     | 184/441 [00:55<01:21,  3.16it/s]
 42%|████▏     | 185/441 [00:56<01:21,  3.16it/s]
 42%|████▏     | 186/441 [00:56<01:17,  3.31it/s]
 42%|████▏     | 187/441 [00:56<01:18,  3.23it/s]
 43%|████▎     | 188/441 [00:57<01:19,  3.20it/s]
 43%|████▎     | 189/441 [00:57<01:15,  3.34it/s]
 43%|████▎     | 190/441 [00:57<01:17,  3.25it/s]
 43%|████▎     | 191/441 [00:57<01:19,  3.14it/s]
 44%|████▎     | 192/441 [00:58<01:14,  3.33it/s]
 44%|████▍     | 193/441 [00:58<01:17,  3.19it/s]
 44%|████▍     | 194/441 [00:58<01:18,  3.16it/s]
 44%|████▍     | 195/441 [00:59<01:13,  3.34it/s]
 44%|████▍     | 196/441 [00:59<01:18,  3.13it/s]
 45%|████▍     | 197/441 [00:59<01:17,  3.15it/s]
 45%|████▍     | 198/441 [01:00<01:13,  3.30it/s]
 45%|████▌     | 199/441 [01:00<01:15,  3.19it/s]
 45%|████▌     | 200/441 [01:00<01:15,  3.20it/s]
 46%|████▌     | 201/441 [01:01<01:13,  3.25it/s]
 46%|████▌     | 202/441 [01:01<01:15,  3.16it/s]
 46%|████▌     | 203/441 [01:01<01:17,  3.08it/s]
 46%|████▋     | 204/441 [01:01<01:13,  3.21it/s]
 46%|████▋     | 205/441 [01:02<01:19,  2.99it/s]
 47%|████▋     | 206/441 [01:02<01:16,  3.08it/s]
 47%|████▋     | 207/441 [01:02<01:11,  3.25it/s]
 47%|████▋     | 208/441 [01:03<01:14,  3.14it/s]
 47%|████▋     | 209/441 [01:03<01:15,  3.08it/s]
 48%|████▊     | 210/441 [01:03<01:11,  3.24it/s]
 48%|████▊     | 211/441 [01:04<01:17,  2.98it/s]
 48%|████▊     | 212/441 [01:04<01:14,  3.06it/s]
 48%|████▊     | 213/441 [01:04<01:11,  3.21it/s]
 49%|████▊     | 214/441 [01:05<01:18,  2.88it/s]
 49%|████▉     | 215/441 [01:05<01:12,  3.13it/s]
 49%|████▉     | 216/441 [01:05<01:09,  3.25it/s]
 49%|████▉     | 217/441 [01:06<01:09,  3.22it/s]
 49%|████▉     | 218/441 [01:06<01:06,  3.35it/s]
 50%|████▉     | 219/441 [01:06<01:08,  3.23it/s]
 50%|████▉     | 220/441 [01:07<01:11,  3.08it/s]
 50%|█████     | 221/441 [01:07<01:13,  2.99it/s]
 50%|█████     | 222/441 [01:07<01:05,  3.36it/s]
 51%|█████     | 223/441 [01:08<01:06,  3.30it/s]
 51%|█████     | 224/441 [01:08<01:08,  3.16it/s]
 51%|█████     | 225/441 [01:08<01:10,  3.08it/s]
 51%|█████     | 226/441 [01:08<01:07,  3.20it/s]
 51%|█████▏    | 227/441 [01:09<01:03,  3.34it/s]
 52%|█████▏    | 228/441 [01:09<01:06,  3.23it/s]
 52%|█████▏    | 229/441 [01:09<01:05,  3.22it/s]
 52%|█████▏    | 230/441 [01:10<01:03,  3.32it/s]
 52%|█████▏    | 231/441 [01:10<01:06,  3.15it/s]
 53%|█████▎    | 232/441 [01:10<01:05,  3.21it/s]
 53%|█████▎    | 233/441 [01:11<01:04,  3.21it/s]
 53%|█████▎    | 234/441 [01:11<01:05,  3.18it/s]
 53%|█████▎    | 235/441 [01:11<01:04,  3.17it/s]
 54%|█████▎    | 236/441 [01:12<01:02,  3.30it/s]
 54%|█████▎    | 237/441 [01:12<01:03,  3.19it/s]
 54%|█████▍    | 238/441 [01:12<01:03,  3.20it/s]
 54%|█████▍    | 239/441 [01:13<01:01,  3.31it/s]
 54%|█████▍    | 240/441 [01:13<01:03,  3.16it/s]
 55%|█████▍    | 241/441 [01:13<01:01,  3.23it/s]
 55%|█████▍    | 242/441 [01:13<01:00,  3.30it/s]
 55%|█████▌    | 243/441 [01:14<01:01,  3.22it/s]
 55%|█████▌    | 244/441 [01:14<01:01,  3.18it/s]
 56%|█████▌    | 245/441 [01:14<00:59,  3.32it/s]
 56%|█████▌    | 246/441 [01:15<01:01,  3.15it/s]
 56%|█████▌    | 247/441 [01:15<00:58,  3.31it/s]
 56%|█████▌    | 248/441 [01:15<01:02,  3.11it/s]
 56%|█████▋    | 249/441 [01:16<01:00,  3.19it/s]
 57%|█████▋    | 250/441 [01:16<00:59,  3.21it/s]
 57%|█████▋    | 251/441 [01:16<01:00,  3.16it/s]
 57%|█████▋    | 252/441 [01:17<00:59,  3.17it/s]
 57%|█████▋    | 253/441 [01:17<00:59,  3.14it/s]
 58%|█████▊    | 254/441 [01:17<00:57,  3.24it/s]
 58%|█████▊    | 255/441 [01:17<00:57,  3.24it/s]
 58%|█████▊    | 256/441 [01:18<00:58,  3.19it/s]
 58%|█████▊    | 257/441 [01:18<00:55,  3.33it/s]
 59%|█████▊    | 258/441 [01:18<01:00,  3.04it/s]
 59%|█████▊    | 259/441 [01:19<00:58,  3.10it/s]
 59%|█████▉    | 260/441 [01:19<00:54,  3.33it/s]
 59%|█████▉    | 261/441 [01:19<00:56,  3.20it/s]
 59%|█████▉    | 262/441 [01:20<00:57,  3.11it/s]
 60%|█████▉    | 263/441 [01:20<00:56,  3.17it/s]
 60%|█████▉    | 264/441 [01:20<00:54,  3.24it/s]
 60%|██████    | 265/441 [01:21<00:52,  3.33it/s]
 60%|██████    | 266/441 [01:21<00:56,  3.12it/s]
 61%|██████    | 267/441 [01:21<00:54,  3.17it/s]
 61%|██████    | 268/441 [01:22<00:52,  3.30it/s]
 61%|██████    | 269/441 [01:22<00:54,  3.14it/s]
 61%|██████    | 270/441 [01:22<00:53,  3.17it/s]
 61%|██████▏   | 271/441 [01:22<00:50,  3.35it/s]
 62%|██████▏   | 272/441 [01:23<00:53,  3.16it/s]
 62%|██████▏   | 273/441 [01:23<00:53,  3.16it/s]
 62%|██████▏   | 274/441 [01:23<00:49,  3.34it/s]
 62%|██████▏   | 275/441 [01:24<00:51,  3.24it/s]
 63%|██████▎   | 276/441 [01:24<00:54,  3.05it/s]
 63%|██████▎   | 277/441 [01:24<00:49,  3.33it/s]
 63%|██████▎   | 278/441 [01:25<00:51,  3.16it/s]
 63%|██████▎   | 279/441 [01:25<00:51,  3.17it/s]
 63%|██████▎   | 280/441 [01:25<00:48,  3.35it/s]
 64%|██████▎   | 281/441 [01:26<00:50,  3.19it/s]
 64%|██████▍   | 282/441 [01:26<00:49,  3.20it/s]
 64%|██████▍   | 283/441 [01:26<00:47,  3.34it/s]
 64%|██████▍   | 284/441 [01:27<00:49,  3.16it/s]
 65%|██████▍   | 285/441 [01:27<00:47,  3.26it/s]
 65%|██████▍   | 286/441 [01:27<00:46,  3.33it/s]
 65%|██████▌   | 287/441 [01:27<00:48,  3.19it/s]
 65%|██████▌   | 288/441 [01:28<00:49,  3.10it/s]
 66%|██████▌   | 289/441 [01:28<00:45,  3.34it/s]
 66%|██████▌   | 290/441 [01:28<00:47,  3.17it/s]
 66%|██████▌   | 291/441 [01:29<00:46,  3.22it/s]
 66%|██████▌   | 292/441 [01:29<00:46,  3.18it/s]
 66%|██████▋   | 293/441 [01:29<00:46,  3.19it/s]
 67%|██████▋   | 294/441 [01:30<00:43,  3.34it/s]
 67%|██████▋   | 295/441 [01:30<00:44,  3.26it/s]
 67%|██████▋   | 296/441 [01:30<00:44,  3.26it/s]
 67%|██████▋   | 297/441 [01:31<00:43,  3.31it/s]
 68%|██████▊   | 298/441 [01:31<00:44,  3.21it/s]
 68%|██████▊   | 299/441 [01:31<00:45,  3.15it/s]
 68%|██████▊   | 300/441 [01:31<00:45,  3.07it/s]
 68%|██████▊   | 301/441 [01:32<00:44,  3.18it/s]
 68%|██████▊   | 302/441 [01:32<00:43,  3.18it/s]
 69%|██████▊   | 303/441 [01:32<00:42,  3.27it/s]
 69%|██████▉   | 304/441 [01:33<00:46,  2.97it/s]
 69%|██████▉   | 305/441 [01:33<00:43,  3.12it/s]
 69%|██████▉   | 306/441 [01:33<00:41,  3.28it/s]
 70%|██████▉   | 307/441 [01:34<00:43,  3.07it/s]
 70%|██████▉   | 308/441 [01:34<00:43,  3.03it/s]
 70%|███████   | 309/441 [01:34<00:40,  3.25it/s]
 70%|███████   | 310/441 [01:35<00:42,  3.05it/s]
 71%|███████   | 311/441 [01:35<00:42,  3.04it/s]
 71%|███████   | 312/441 [01:35<00:39,  3.27it/s]
 71%|███████   | 313/441 [01:36<00:42,  3.01it/s]
 71%|███████   | 314/441 [01:36<00:42,  2.99it/s]
 71%|███████▏  | 315/441 [01:36<00:39,  3.19it/s]
 72%|███████▏  | 316/441 [01:37<00:38,  3.25it/s]
 72%|███████▏  | 317/441 [01:37<00:39,  3.17it/s]
 72%|███████▏  | 318/441 [01:37<00:36,  3.33it/s]
 72%|███████▏  | 319/441 [01:38<00:37,  3.23it/s]
 73%|███████▎  | 320/441 [01:38<00:41,  2.93it/s]
 73%|███████▎  | 321/441 [01:38<00:36,  3.30it/s]
 73%|███████▎  | 322/441 [01:38<00:37,  3.20it/s]
 73%|███████▎  | 323/441 [01:39<00:37,  3.17it/s]
 73%|███████▎  | 324/441 [01:39<00:35,  3.31it/s]
 74%|███████▎  | 325/441 [01:39<00:37,  3.13it/s]
 74%|███████▍  | 326/441 [01:40<00:36,  3.17it/s]
 74%|███████▍  | 327/441 [01:40<00:34,  3.30it/s]
 74%|███████▍  | 328/441 [01:40<00:35,  3.21it/s]
 75%|███████▍  | 329/441 [01:41<00:35,  3.15it/s]
 75%|███████▍  | 330/441 [01:41<00:33,  3.30it/s]
 75%|███████▌  | 331/441 [01:41<00:35,  3.12it/s]
 75%|███████▌  | 332/441 [01:42<00:34,  3.11it/s]
 76%|███████▌  | 333/441 [01:42<00:32,  3.33it/s]
 76%|███████▌  | 334/441 [01:42<00:33,  3.22it/s]
 76%|███████▌  | 335/441 [01:43<00:32,  3.29it/s]
 76%|███████▌  | 336/441 [01:43<00:32,  3.21it/s]
 76%|███████▋  | 337/441 [01:43<00:32,  3.16it/s]
 77%|███████▋  | 338/441 [01:43<00:30,  3.33it/s]
 77%|███████▋  | 339/441 [01:44<00:32,  3.17it/s]
 77%|███████▋  | 340/441 [01:44<00:31,  3.17it/s]
 77%|███████▋  | 341/441 [01:44<00:30,  3.30it/s]
 78%|███████▊  | 342/441 [01:45<00:31,  3.10it/s]
 78%|███████▊  | 343/441 [01:45<00:30,  3.19it/s]
 78%|███████▊  | 344/441 [01:45<00:29,  3.34it/s]
 78%|███████▊  | 345/441 [01:46<00:29,  3.26it/s]
 78%|███████▊  | 346/441 [01:46<00:29,  3.17it/s]
 79%|███████▊  | 347/441 [01:46<00:28,  3.30it/s]
 79%|███████▉  | 348/441 [01:47<00:29,  3.17it/s]
 79%|███████▉  | 349/441 [01:47<00:29,  3.14it/s]
 79%|███████▉  | 350/441 [01:47<00:27,  3.29it/s]
 80%|███████▉  | 351/441 [01:48<00:28,  3.17it/s]
 80%|███████▉  | 352/441 [01:48<00:28,  3.12it/s]
 80%|████████  | 353/441 [01:48<00:26,  3.38it/s]
 80%|████████  | 354/441 [01:48<00:27,  3.19it/s]
 80%|████████  | 355/441 [01:49<00:27,  3.13it/s]
 81%|████████  | 356/441 [01:49<00:25,  3.32it/s]
 81%|████████  | 357/441 [01:49<00:26,  3.15it/s]
 81%|████████  | 358/441 [01:50<00:26,  3.14it/s]
 81%|████████▏ | 359/441 [01:50<00:25,  3.23it/s]
 82%|████████▏ | 360/441 [01:50<00:25,  3.21it/s]
 82%|████████▏ | 361/441 [01:51<00:25,  3.16it/s]
 82%|████████▏ | 362/441 [01:51<00:23,  3.35it/s]
 82%|████████▏ | 363/441 [01:51<00:24,  3.14it/s]
 83%|████████▎ | 364/441 [01:52<00:24,  3.16it/s]
 83%|████████▎ | 365/441 [01:52<00:23,  3.27it/s]
 83%|████████▎ | 366/441 [01:52<00:23,  3.14it/s]
 83%|████████▎ | 367/441 [01:53<00:23,  3.09it/s]
 83%|████████▎ | 368/441 [01:53<00:22,  3.29it/s]
 84%|████████▎ | 369/441 [01:53<00:22,  3.14it/s]
 84%|████████▍ | 370/441 [01:53<00:23,  3.08it/s]
 84%|████████▍ | 371/441 [01:54<00:21,  3.20it/s]
 84%|████████▍ | 372/441 [01:54<00:22,  3.05it/s]
 85%|████████▍ | 373/441 [01:54<00:22,  3.06it/s]
 85%|████████▍ | 374/441 [01:55<00:20,  3.21it/s]
 85%|████████▌ | 375/441 [01:55<00:22,  2.98it/s]
 85%|████████▌ | 376/441 [01:55<00:21,  3.02it/s]
 85%|████████▌ | 377/441 [01:56<00:19,  3.23it/s]
 86%|████████▌ | 378/441 [01:56<00:21,  2.96it/s]
 86%|████████▌ | 379/441 [01:56<00:20,  3.02it/s]
 86%|████████▌ | 380/441 [01:57<00:19,  3.19it/s]
 86%|████████▋ | 381/441 [01:57<00:20,  2.99it/s]
 87%|████████▋ | 382/441 [01:57<00:17,  3.30it/s]
 87%|████████▋ | 383/441 [01:58<00:18,  3.08it/s]
 87%|████████▋ | 384/441 [01:58<00:18,  3.08it/s]
 87%|████████▋ | 385/441 [01:58<00:17,  3.23it/s]
 88%|████████▊ | 386/441 [01:59<00:18,  3.05it/s]
 88%|████████▊ | 387/441 [01:59<00:17,  3.04it/s]
 88%|████████▊ | 388/441 [01:59<00:16,  3.20it/s]
 88%|████████▊ | 389/441 [02:00<00:16,  3.08it/s]
 88%|████████▊ | 390/441 [02:00<00:15,  3.27it/s]
 89%|████████▊ | 391/441 [02:00<00:16,  3.12it/s]
 89%|████████▉ | 392/441 [02:01<00:15,  3.10it/s]
 89%|████████▉ | 393/441 [02:01<00:14,  3.26it/s]
 89%|████████▉ | 394/441 [02:01<00:14,  3.19it/s]
 90%|████████▉ | 395/441 [02:01<00:14,  3.08it/s]
 90%|████████▉ | 396/441 [02:02<00:13,  3.24it/s]
 90%|█████████ | 397/441 [02:02<00:14,  3.06it/s]
 90%|█████████ | 398/441 [02:02<00:13,  3.21it/s]
 90%|█████████ | 399/441 [02:03<00:12,  3.32it/s]
 91%|█████████ | 400/441 [02:03<00:12,  3.24it/s]
 91%|█████████ | 401/441 [02:03<00:12,  3.11it/s]
 91%|█████████ | 402/441 [02:04<00:11,  3.36it/s]
 91%|█████████▏| 403/441 [02:04<00:11,  3.22it/s]
 92%|█████████▏| 404/441 [02:04<00:11,  3.25it/s]
 92%|█████████▏| 405/441 [02:05<00:10,  3.31it/s]
 92%|█████████▏| 406/441 [02:05<00:10,  3.19it/s]
 92%|█████████▏| 407/441 [02:05<00:10,  3.27it/s]
 93%|█████████▎| 408/441 [02:06<00:09,  3.32it/s]
 93%|█████████▎| 409/441 [02:06<00:10,  3.16it/s]
 93%|█████████▎| 410/441 [02:06<00:09,  3.11it/s]
 93%|█████████▎| 411/441 [02:06<00:09,  3.29it/s]
 93%|█████████▎| 412/441 [02:07<00:09,  3.14it/s]
 94%|█████████▎| 413/441 [02:07<00:08,  3.12it/s]
 94%|█████████▍| 414/441 [02:07<00:08,  3.21it/s]
 94%|█████████▍| 415/441 [02:08<00:08,  3.17it/s]
 94%|█████████▍| 416/441 [02:08<00:07,  3.21it/s]
 95%|█████████▍| 417/441 [02:08<00:07,  3.36it/s]
 95%|█████████▍| 418/441 [02:09<00:07,  3.17it/s]
 95%|█████████▌| 419/441 [02:09<00:06,  3.21it/s]
 95%|█████████▌| 420/441 [02:09<00:06,  3.19it/s]
 95%|█████████▌| 421/441 [02:10<00:06,  3.16it/s]
 96%|█████████▌| 422/441 [02:10<00:06,  3.16it/s]
 96%|█████████▌| 423/441 [02:10<00:05,  3.35it/s]
 96%|█████████▌| 424/441 [02:11<00:05,  3.17it/s]
 96%|█████████▋| 425/441 [02:11<00:05,  3.17it/s]
 97%|█████████▋| 426/441 [02:11<00:04,  3.36it/s]
 97%|█████████▋| 427/441 [02:11<00:04,  3.14it/s]
 97%|█████████▋| 428/441 [02:12<00:04,  3.21it/s]
 97%|█████████▋| 429/441 [02:12<00:03,  3.22it/s]
 98%|█████████▊| 430/441 [02:12<00:03,  3.16it/s]
 98%|█████████▊| 431/441 [02:13<00:03,  3.30it/s]
 98%|█████████▊| 432/441 [02:13<00:02,  3.07it/s]
 98%|█████████▊| 433/441 [02:13<00:02,  3.14it/s]
 98%|█████████▊| 434/441 [02:14<00:02,  3.35it/s]
 99%|█████████▊| 435/441 [02:14<00:01,  3.18it/s]
 99%|█████████▉| 436/441 [02:14<00:01,  3.19it/s]
 99%|█████████▉| 437/441 [02:15<00:01,  3.33it/s]
 99%|█████████▉| 438/441 [02:15<00:00,  3.24it/s]
100%|█████████▉| 439/441 [02:15<00:00,  3.18it/s]
100%|█████████▉| 440/441 [02:15<00:00,  3.32it/s]
100%|██████████| 441/441 [02:16<00:00,  3.21it/s]
100%|██████████| 441/441 [02:16<00:00,  3.24it/s]
create LoRA network. base dim (rank): 256, alpha: 1.0
neuron dropout: p=None, rank dropout: p=None, module dropout: p=None
create LoRA for Text Encoder 1:
create LoRA for Text Encoder 2:
create LoRA for Text Encoder: 264 modules.
create LoRA for U-Net: 722 modules.
enable LoRA for text encoder
enable LoRA for U-Net
prepare optimizer, data loader etc.
use Adafactor optimizer | {'scale_parameter': False, 'relative_step': False, 'warmup_init': False}
because max_grad_norm is set, clip_grad_norm is enabled. consider set to 0 / max_grad_normが設定されているためclip_grad_normが有効になります。0に設定して無効にしたほうがいいかもしれません
constant_with_warmup will be good / スケジューラはconstant_with_warmupが良いかもしれません
running training / 学習開始
  num train images * repeats / 学習画像の数×繰り返し回数: 420
  num reg images / 正則化画像の数: 1000
  num batches per epoch / 1epochのバッチ数: 840
  num epochs / epoch数: 10
  batch size per device / バッチサイズ: 1
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 8400

steps:   0%|          | 0/8400 [00:00<?, ?it/s]
epoch 1/10

steps:   0%|          | 1/8400 [00:02<5:39:32,  2.43s/it]
steps:   0%|          | 1/8400 [00:02<5:39:35,  2.43s/it, avr_loss=0.204]Traceback (most recent call last):
  File "/workspace/kohya_ss/./sdxl_train_network.py", line 185, in <module>
    trainer.train(args)
  File "/workspace/kohya_ss/train_network.py", line 825, in train
    accelerator.backward(loss)
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 1985, in backward
    loss.backward(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 274, in apply
    return user_fn(self, *args)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py", line 157, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 23.68 GiB total capacity; 15.32 GiB already allocated; 1.38 MiB free; 15.63 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

steps:   0%|          | 1/8400 [00:03<9:13:56,  3.96s/it, avr_loss=0.204]
Traceback (most recent call last):
  File "/workspace/kohya_ss/venv/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 986, in launch_command
    simple_launcher(args)
  File "/workspace/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 628, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/workspace/kohya_ss/venv/bin/python3', './sdxl_train_network.py', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--pretrained_model_name_or_path=/workspace/stable-diffusion-webui/models/Stable-diffusion/sd_xl_base_1.0.safetensors', '--train_data_dir=/workspace/Salida/img', '--reg_data_dir=/workspace/Salida/reg', '--resolution=1024,1024', '--output_dir=/workspace/Salida/model', '--logging_dir=/workspace/Salida/log', '--network_alpha=1', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=0.0003', '--unet_lr=0.0003', '--network_dim=256', '--output_name=victorahumada', '--lr_scheduler_num_cycles=10', '--no_half_vae', '--learning_rate=0.0003', '--lr_scheduler=constant', '--train_batch_size=1', '--max_train_steps=8400', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16', '--caption_extension=.txt', '--cache_latents', '--cache_latents_to_disk', '--optimizer_type=Adafactor', '--optimizer_args', 'scale_parameter=False', 'relative_step=False', 'warmup_init=False', '--max_data_loader_n_workers=0', '--bucket_reso_steps=64', '--gradient_checkpointing', '--xformers', '--bucket_no_upscale', '--noise_offset=0.0']' returned non-zero exit status 1.
21:04:37-332630 INFO     There is no running process to kill.