Loading...
02/08/2019 19:50:49 INFO     Log level set to: INFO
Using TensorFlow backend.
02/08/2019 19:50:51 INFO     Model A Directory: C:\Users\ZeroCool22\faceswap\train\input_a
02/08/2019 19:50:51 INFO     Model B Directory: C:\Users\ZeroCool22\faceswap\train\input_b
02/08/2019 19:50:51 INFO     Training data directory: C:\Users\ZeroCool22\Desktop\models
02/08/2019 19:50:51 INFO     =====================================================================
02/08/2019 19:50:51 INFO     - Using live preview                                                -
02/08/2019 19:50:51 INFO     - Press 'ENTER' on the preview window to save and quit              -
02/08/2019 19:50:51 INFO     - Press 'S' on the preview window to save model weights immediately -
02/08/2019 19:50:51 INFO     =====================================================================
02/08/2019 19:50:52 INFO     Loading data, this may take a while...
02/08/2019 19:50:52 INFO     Loading Model from Original_Hires plugin...
02/08/2019 19:50:57 INFO     Loading config: 'C:\Users\ZeroCool22\faceswap\config\train.ini'
02/08/2019 19:50:57 INFO     Using configuration saved in state file
02/08/2019 19:51:01 INFO     Loaded model from disk: 'C:\Users\ZeroCool22\Desktop\models'
02/08/2019 19:51:01 INFO     Loading Trainer from Original plugin...
02/08/2019 19:51:01 INFO     Reading alignments from: 'C:\Users\ZeroCool22\faceswap\extract\input_A\alignments.json'
02/08/2019 19:51:02 INFO     Reading alignments from: 'C:\Users\ZeroCool22\faceswap\extract\input_B\alignments.json'
02/08/2019 19:51:02 INFO     Enabling TensorBoard Logging
02/08/2019 19:51:44 CRITICAL Error caught! Exiting...
02/08/2019 19:51:44 ERROR    Caught exception in thread: 'training_0'
02/08/2019 19:51:47 ERROR    Got Exception on main handler:
Traceback (most recent call last):
File "C:\Users\ZeroCool22\faceswap\lib\cli.py", line 90, in execute_script
process.process()
File "C:\Users\ZeroCool22\faceswap\scripts\train.py", line 97, in process
self.end_thread(thread, err)
File "C:\Users\ZeroCool22\faceswap\scripts\train.py", line 122, in end_thread
thread.join()
File "C:\Users\ZeroCool22\faceswap\lib\multithreading.py", line 179, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\ZeroCool22\faceswap\lib\multithreading.py", line 117, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\ZeroCool22\faceswap\scripts\train.py", line 148, in training
raise err
File "C:\Users\ZeroCool22\faceswap\scripts\train.py", line 138, in training
self.run_training_cycle(model, trainer)
File "C:\Users\ZeroCool22\faceswap\scripts\train.py", line 210, in run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "C:\Users\ZeroCool22\faceswap\plugins\train\trainer\_base.py", line 137, in train_one_step
loss[side] = batcher.train_one_batch(is_preview_iteration)
File "C:\Users\ZeroCool22\faceswap\plugins\train\trainer\_base.py", line 211, in train_one_batch
loss = self.model.predictors[self.side].train_on_batch(*batch)
File "C:\Users\ZeroCool22\AppData\Roaming\Python\Python36\site-packages\keras\engine\training.py", line 1217, in train_on_batch
outputs = self.train_function(ins)
File "C:\Users\ZeroCool22\AppData\Roaming\Python\Python36\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "C:\Users\ZeroCool22\AppData\Roaming\Python\Python36\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "C:\Users\ZeroCool22\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
run_metadata_ptr)
File "C:\Users\ZeroCool22\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[5,5,512,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node training_1/Adam/mul_18}} = Mul[T=DT_FLOAT, _class=["loc:@training_1/Adam/cond_18/Switch_1"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](training_1/Adam/truediv_34, training_1/Adam/gradients/decoder_1/conv2d_11/convolution_grad/Conv2DBackpropFilter)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

02/08/2019 19:51:47 CRITICAL An unexpected crash has occurred. Crash report written to C:\Users\ZeroCool22\Desktop\crash_report.2019.02.08.195144239062.log. Please verify you are running the latest version of faceswap before reporting
Process exited.