2018-03-13 03:45:32.181075: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
2018-03-13 03:45:32.214919: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
2018-03-13 03:45:32.236942: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
2018-03-13 03:45:32.250349: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
2018-03-13 03:45:32.275450: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
2018-03-13 03:45:32.289917: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
2018-03-13 03:45:32.307290: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
2018-03-13 03:45:32.318013: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at mkl_conv_ops.h:497 : Aborted: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
Traceback (most recent call last):
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call
    return fn(*args)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1340, in _run_fn
    target_list, status, run_metadata)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.AbortedError: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
         [[Node: gradients/SRGAN_g/Conv4_6/Conv2D_grad/Conv2DBackpropFilter = _MklConv2DBackpropFilter[T=DT_FLOAT, _kernel="MklOp", data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](SRGAN_g/pixelshufflerx2/1/6/Relu, gradients/SRGAN_g/Conv3_7/Conv2D_grad/Const_1, gradients/SRGAN_g/Conv4_6/add_grad/tuple/control_dependency, SRGAN_g/pixelshufflerx2/1/6/Relu:1, DMT/_250, DMT/_251)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 289, in <module>
    train()
  File "main.py", line 149, in train
    errM, _ = sess.run([mse_loss, g_optim_init], {t_image: b_imgs_96, t_target_image: b_imgs_384})
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 905, in run
    run_metadata_ptr)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1137, in _run
    feed_dict_tensor, options, run_metadata)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run
    options, run_metadata)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AbortedError: Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
         [[Node: gradients/SRGAN_g/Conv4_6/Conv2D_grad/Conv2DBackpropFilter = _MklConv2DBackpropFilter[T=DT_FLOAT, _kernel="MklOp", data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](SRGAN_g/pixelshufflerx2/1/6/Relu, gradients/SRGAN_g/Conv3_7/Conv2D_grad/Const_1, gradients/SRGAN_g/Conv4_6/add_grad/tuple/control_dependency, SRGAN_g/pixelshufflerx2/1/6/Relu:1, DMT/_250, DMT/_251)]]

Caused by op 'gradients/SRGAN_g/Conv4_6/Conv2D_grad/Conv2DBackpropFilter', defined at:
  File "main.py", line 289, in <module>
    train()
  File "main.py", line 89, in train
    g_optim_init = tf.train.AdamOptimizer(lr_v, beta1=beta1).minimize(mse_loss, var_list=g_vars)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 359, in minimize
    grad_loss=grad_loss)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 460, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 611, in gradients
    lambda: grad_fn(op, *out_grads))
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 377, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 611, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/ops/nn_grad.py", line 523, in _Conv2DGrad
    data_format=data_format)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 717, in conv2d_backprop_filter
    dilations=dilations, name=name)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
    op_def=op_def)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

...which was originally created as op 'SRGAN_g/Conv4_6/Conv2D', defined at:
  File "main.py", line 289, in <module>
    train()
  File "main.py", line 53, in train
    net_g = SRGAN_g(t_image, is_train=True, reuse=False)
  File "/global/home/hpc4142/srgan-master_TF/model.py", line 57, in SRGAN_g
    recursive_layers[i] = Conv2dReuse(recursive_layers[i], 256, (3, 3), (1, 1), act=None, padding='SAME', W_init=w_init, name='n256s1/2/%s' %i, group = "Conv4")
  File "/global/home/hpc4142/srgan-master_TF/nu_layers.py", line 461, in Conv2dReuse
    group = group)
  File "/global/home/hpc4142/srgan-master_TF/nu_layers.py", line 270, in __init__
    self.outputs = act( tf.nn.conv2d(self.inputs, W, strides=strides, padding=padding, use_cudnn_on_gpu=use_cudnn_on_gpu, data_format=data_format) + b )
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 631, in conv2d
    data_format=data_format, dilations=dilations, name=name)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op
    op_def=op_def)
  File "/global/home/hpc4142/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

AbortedError (see above for traceback): Operation received an exception:Status: 3, message: could not set native handle, in file ./tensorflow/core/kernels/mkl_conv_ops.h:494
         [[Node: gradients/SRGAN_g/Conv4_6/Conv2D_grad/Conv2DBackpropFilter = _MklConv2DBackpropFilter[T=DT_FLOAT, _kernel="MklOp", data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](SRGAN_g/pixelshufflerx2/1/6/Relu, gradients/SRGAN_g/Conv3_7/Conv2D_grad/Const_1, gradients/SRGAN_g/Conv4_6/add_grad/tuple/control_dependency, SRGAN_g/pixelshufflerx2/1/6/Relu:1, DMT/_250, DMT/_251)]]