2019-11-13 19:17:34.728199: E tensorflow/core/kernels/check_numerics_op.cc:185] abnormal_detected_host @0x7f3554cab200 = {1, 0} Found Inf or NaN global norm.
Exiting due to exception: Found Inf or NaN global norm. : Tensor had NaN values
         [[node Tacotron_model/VerifyFinite/CheckNumerics (defined at /home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py:487)  = CheckNumerics[T=DT_FLOAT, message="Found Inf or NaN global norm.", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Tacotron_model/global_norm/global_norm)]]
         [[{{node Tacotron_model/clip_by_global_norm/mul_41/_543}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_7770_Tacotron_model/clip_by_global_norm/mul_41", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'Tacotron_model/VerifyFinite/CheckNumerics', defined at:
  File "synthesizer_train.py", line 55, in <module>
    tacotron_train(args, log_dir, hparams)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/train.py", line 392, in tacotron_train
    return train(log_dir, args, hparams)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/train.py", line 148, in train
    model, stats = model_train_mode(args, feeder, hparams, global_step)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/train.py", line 93, in model_train_mode
    model.add_optimizer(global_step)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py", line 487, in add_optimizer
    clipped_gradients, _ = tf.clip_by_global_norm(avg_grads, 1.)  # __mark 0.5 refer
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/clip_ops.py", line 265, in clip_by_global_norm
    "Found Inf or NaN global norm.")
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/numerics.py", line 47, in verify_tensor_all_finite
    verify_input = array_ops.check_numerics(t, message=msg)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 817, in check_numerics
    "CheckNumerics", tensor=tensor, message=message, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Found Inf or NaN global norm. : Tensor had NaN values
         [[node Tacotron_model/VerifyFinite/CheckNumerics (defined at /home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py:487)  = CheckNumerics[T=DT_FLOAT, message="Found Inf or NaN global norm.", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Tacotron_model/global_norm/global_norm)]]
         [[{{node Tacotron_model/clip_by_global_norm/mul_41/_543}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_7770_Tacotron_model/clip_by_global_norm/mul_41", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Found Inf or NaN global norm. : Tensor had NaN values
         [[{{node Tacotron_model/VerifyFinite/CheckNumerics}} = CheckNumerics[T=DT_FLOAT, message="Found Inf or NaN global norm.", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Tacotron_model/global_norm/global_norm)]]
         [[{{node Tacotron_model/clip_by_global_norm/mul_41/_543}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_7770_Tacotron_model/clip_by_global_norm/mul_41", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/train.py", line 210, in train
    step, loss, opt = sess.run([global_step, model.loss, model.optimize])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Found Inf or NaN global norm. : Tensor had NaN values
         [[node Tacotron_model/VerifyFinite/CheckNumerics (defined at /home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py:487)  = CheckNumerics[T=DT_FLOAT, message="Found Inf or NaN global norm.", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Tacotron_model/global_norm/global_norm)]]
         [[{{node Tacotron_model/clip_by_global_norm/mul_41/_543}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_7770_Tacotron_model/clip_by_global_norm/mul_41", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'Tacotron_model/VerifyFinite/CheckNumerics', defined at:
  File "synthesizer_train.py", line 55, in <module>
    tacotron_train(args, log_dir, hparams)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/train.py", line 392, in tacotron_train
    return train(log_dir, args, hparams)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/train.py", line 148, in train
    model, stats = model_train_mode(args, feeder, hparams, global_step)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/train.py", line 93, in model_train_mode
    model.add_optimizer(global_step)
  File "/home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py", line 487, in add_optimizer
    clipped_gradients, _ = tf.clip_by_global_norm(avg_grads, 1.)  # __mark 0.5 refer
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/clip_ops.py", line 265, in clip_by_global_norm
    "Found Inf or NaN global norm.")
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/numerics.py", line 47, in verify_tensor_all_finite
    verify_input = array_ops.check_numerics(t, message=msg)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 817, in check_numerics
    "CheckNumerics", tensor=tensor, message=message, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Found Inf or NaN global norm. : Tensor had NaN values
         [[node Tacotron_model/VerifyFinite/CheckNumerics (defined at /home/avatar/Documents/Real-Time-Voice-Cloning/synthesizer/models/tacotron.py:487)  = CheckNumerics[T=DT_FLOAT, message="Found Inf or NaN global norm.", _device="/job:localhost/replica:0/task:0/device:GPU:0"](Tacotron_model/global_norm/global_norm)]]
         [[{{node Tacotron_model/clip_by_global_norm/mul_41/_543}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_7770_Tacotron_model/clip_by_global_norm/mul_41", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

2019-11-13 19:17:34.743579: W tensorflow/core/kernels/queue_base.cc:277] _1_datafeeder/eval_queue: Skipping cancelled enqueue attempt with queue not closed
2019-11-13 19:17:34.743685: W tensorflow/core/kernels/queue_base.cc:277] _0_datafeeder/input_queue: Skipping cancelled enqueue attempt with queue not closed