Untitled

2024-05-01 13:24:06.597101: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-05-01 13:24:06.597152: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-05-01 13:24:06.598624: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-05-01 13:24:06.606296: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-05-01 13:24:07.720774: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-05-01 13:24:09.758749: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:24:09.812354: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:24:09.812669: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
GPUs available:  1
CPUs available:  1
Loading dataset...
Loaded: 2900 samples (112.971s)
Defining model...
2024-05-01 13:26:02.826919: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:26:02.827341: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:26:02.827619: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:26:03.225394: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:26:03.225736: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:26:03.225950: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Overriding orig_value setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2024-05-01 13:26:03.226042: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2024-05-01 13:26:03.226246: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13949 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5
Model: "SegNet"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Layer (type)              ┃ Output Shape           ┃        Param # ┃ Connected to           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ input (InputLayer)        │ (None, None, None, 3)  │              0 │ -                      │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv1_1 (Conv2DWithBN) │ (None, None, None, 64) │          2,048 │ input[0][0]            │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv1_2 (Conv2DWithBN) │ (None, None, None, 64) │         37,184 │ en_conv1_1[0][0]       │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_pool_with_argmax      │ [(None, None, None,    │              0 │ en_conv1_2[0][0]       │
│ (MaxPoolWithArgmax)       │ 64), (None, None,      │                │                        │
│                           │ None, 64)]             │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv2_1 (Conv2DWithBN) │ (None, None, None,     │         74,368 │ max_pool_with_argmax[… │
│                           │ 128)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv2_2 (Conv2DWithBN) │ (None, None, None,     │        148,096 │ en_conv2_1[0][0]       │
│                           │ 128)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_pool_with_argmax_1    │ [(None, None, None,    │              0 │ en_conv2_2[0][0]       │
│ (MaxPoolWithArgmax)       │ 128), (None, None,     │                │                        │
│                           │ None, 128)]            │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv3_1 (Conv2DWithBN) │ (None, None, None,     │        296,192 │ max_pool_with_argmax_… │
│                           │ 256)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv3_2 (Conv2DWithBN) │ (None, None, None,     │        591,104 │ en_conv3_1[0][0]       │
│                           │ 256)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv3_3 (Conv2DWithBN) │ (None, None, None,     │        591,104 │ en_conv3_2[0][0]       │
│                           │ 256)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_pool_with_argmax_2    │ [(None, None, None,    │              0 │ en_conv3_3[0][0]       │
│ (MaxPoolWithArgmax)       │ 256), (None, None,     │                │                        │
│                           │ None, 256)]            │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv4_1 (Conv2DWithBN) │ (None, None, None,     │      1,182,208 │ max_pool_with_argmax_… │
│                           │ 512)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv4_2 (Conv2DWithBN) │ (None, None, None,     │      2,361,856 │ en_conv4_1[0][0]       │
│                           │ 512)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ en_conv4_3 (Conv2DWithBN) │ (None, None, None,     │      2,361,856 │ en_conv4_2[0][0]       │
│                           │ 512)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_pool_with_argmax_3    │ [(None, None, None,    │              0 │ en_conv4_3[0][0]       │
│ (MaxPoolWithArgmax)       │ 512), (None, None,     │                │                        │
│                           │ None, 512)]            │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv1_1              │ (None, None, None,     │      2,361,856 │ max_pool_with_argmax_… │
│ (Conv2DTransposeWithBN)   │ 512)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_unpooling2d           │ (None, None, None,     │              0 │ de_deconv1_1[0][0],    │
│ (MaxUnpooling2D)          │ 512)                   │                │ max_pool_with_argmax_… │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv1_2              │ (None, None, None,     │      2,361,856 │ max_unpooling2d[0][0]  │
│ (Conv2DWithBN)            │ 512)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv1_3              │ (None, None, None,     │      2,361,856 │ de_deconv1_2[0][0]     │
│ (Conv2DWithBN)            │ 512)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv2_1              │ (None, None, None,     │      1,180,928 │ de_deconv1_3[0][0]     │
│ (Conv2DTransposeWithBN)   │ 256)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_unpooling2d_1         │ (None, None, None,     │              0 │ de_deconv2_1[0][0],    │
│ (MaxUnpooling2D)          │ 256)                   │                │ max_pool_with_argmax_… │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv2_2              │ (None, None, None,     │        591,104 │ max_unpooling2d_1[0][… │
│ (Conv2DWithBN)            │ 256)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv2_3              │ (None, None, None,     │        591,104 │ de_deconv2_2[0][0]     │
│ (Conv2DWithBN)            │ 256)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv3_1              │ (None, None, None,     │        295,552 │ de_deconv2_3[0][0]     │
│ (Conv2DTransposeWithBN)   │ 128)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_unpooling2d_2         │ (None, None, None,     │              0 │ de_deconv3_1[0][0],    │
│ (MaxUnpooling2D)          │ 128)                   │                │ max_pool_with_argmax_… │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv3_2              │ (None, None, None,     │        148,096 │ max_unpooling2d_2[0][… │
│ (Conv2DWithBN)            │ 128)                   │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv4_1              │ (None, None, None, 64) │         74,048 │ de_deconv3_2[0][0]     │
│ (Conv2DTransposeWithBN)   │                        │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ max_unpooling2d_3         │ (None, None, None, 64) │              0 │ de_deconv4_1[0][0],    │
│ (MaxUnpooling2D)          │                        │                │ max_pool_with_argmax[… │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ de_deconv4_2              │ (None, None, None, 64) │         37,184 │ max_unpooling2d_3[0][… │
│ (Conv2DWithBN)            │                        │                │                        │
├───────────────────────────┼────────────────────────┼────────────────┼────────────────────────┤
│ output (Conv2DTranspose)  │ (None, None, None, 10) │          5,770 │ de_deconv4_2[0][0]     │
└───────────────────────────┴────────────────────────┴────────────────┴────────────────────────┘
 Total params: 17,655,370 (67.35 MB)
 Trainable params: 17,644,618 (67.31 MB)
 Non-trainable params: 10,752 (42.00 KB)
----------------------------------------------------------------------------------------------------
Start training...
Epoch 1/30
2024-05-01 13:26:19.327948: W external/local_tsl/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 31457280 exceeds 10% of free system memory.
2024-05-01 13:26:19.502327: I external/local_xla/xla/service/service.cc:168] XLA service 0x7dac5c038470 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2024-05-01 13:26:19.502399: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2024-05-01 13:26:20.095428: W tensorflow/core/framework/op_kernel.cc:1839] OP_REQUIRES failed at xla_ops.cc:574 : INVALID_ARGUMENT: Detected unsupported operations when trying to compile graph __inference_one_step_on_data_18280[] on XLA_GPU_JIT: MaxPoolWithArgmax (No registered 'MaxPoolWithArgmax' OpKernel for XLA_GPU_JIT devices compatible with node {{node SegNet_1/max_pool_with_argmax_1/MaxPoolWithArgmax}}){{node SegNet_1/max_pool_with_argmax_1/MaxPoolWithArgmax}}
The op is created at:
File "content/drive/MyDrive/SegNet_SceneParse150/train.py", line 42, in <module>
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 314, in fit
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 117, in one_step_on_iterator
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 104, in one_step_on_data
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 51, in train_step
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/layers/layer.py", line 842, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/models/functional.py", line 199, in call
File "root/.local/lib/python3.10/site-packages/keras/src/ops/function.py", line 151, in _run_through_graph
File "root/.local/lib/python3.10/site-packages/keras/src/models/functional.py", line 589, in call
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/layers/layer.py", line 842, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler
File "content/drive/MyDrive/SegNet_SceneParse150/Layers/max_pool_with_argmax.py", line 32, in call
	tf2xla conversion failed while converting __inference_one_step_on_data_18280[]. Run with TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to obtain a dump of the compiled functions.
Traceback (most recent call last):
  File "/content/drive/MyDrive/SegNet_SceneParse150/train.py", line 42, in <module>
    train_history = model.fit(
  File "/root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:

Detected at node SegNet_1/max_pool_with_argmax_1/MaxPoolWithArgmax defined at (most recent call last):
<stack traces unavailable>
Detected at node SegNet_1/max_pool_with_argmax_1/MaxPoolWithArgmax defined at (most recent call last):
<stack traces unavailable>
Detected unsupported operations when trying to compile graph __inference_one_step_on_data_18280[] on XLA_GPU_JIT: MaxPoolWithArgmax (No registered 'MaxPoolWithArgmax' OpKernel for XLA_GPU_JIT devices compatible with node {{node SegNet_1/max_pool_with_argmax_1/MaxPoolWithArgmax}}){{node SegNet_1/max_pool_with_argmax_1/MaxPoolWithArgmax}}
The op is created at:
File "content/drive/MyDrive/SegNet_SceneParse150/train.py", line 42, in <module>
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 314, in fit
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 117, in one_step_on_iterator
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 104, in one_step_on_data
File "root/.local/lib/python3.10/site-packages/keras/src/backend/tensorflow/trainer.py", line 51, in train_step
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/layers/layer.py", line 842, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/models/functional.py", line 199, in call
File "root/.local/lib/python3.10/site-packages/keras/src/ops/function.py", line 151, in _run_through_graph
File "root/.local/lib/python3.10/site-packages/keras/src/models/functional.py", line 589, in call
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/layers/layer.py", line 842, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 117, in error_handler
File "root/.local/lib/python3.10/site-packages/keras/src/ops/operation.py", line 48, in __call__
File "root/.local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 156, in error_handler
File "content/drive/MyDrive/SegNet_SceneParse150/Layers/max_pool_with_argmax.py", line 32, in call
	tf2xla conversion failed while converting __inference_one_step_on_data_18280[]. Run with TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to obtain a dump of the compiled functions.
	 [[StatefulPartitionedCall]] [Op:__inference_one_step_on_iterator_18873]
2024-05-01 13:26:20.405371: W tensorflow/core/kernels/data/generator_dataset_op.cc:108] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]