API tools faq
paste
Guest User

ogbg schedule free log

a guest
Jun 5th, 2025
19
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 81.12 KB | Software | 0 0
raw download clone embed print report
  1. W0605 21:40:45.003000 20 site-packages/torch/distributed/run.py:793]
  2. W0605 21:40:45.003000 20 site-packages/torch/distributed/run.py:793] *****************************************
  3. W0605 21:40:45.003000 20 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
  4. W0605 21:40:45.003000 20 site-packages/torch/distributed/run.py:793] *****************************************
  5. 2025-06-05 21:40:46.306390: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  6. 2025-06-05 21:40:46.306390: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  7. 2025-06-05 21:40:46.306405: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  8. 2025-06-05 21:40:46.306404: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  9. 2025-06-05 21:40:46.306390: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  10. 2025-06-05 21:40:46.306390: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  11. 2025-06-05 21:40:46.306447: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  12. 2025-06-05 21:40:46.306899: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
  13. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  14. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  15. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  16. E0000 00:00:1749159646.327572 93 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  17. E0000 00:00:1749159646.327572 92 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  18. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  19. E0000 00:00:1749159646.327589 90 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  20. E0000 00:00:1749159646.327572 89 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  21. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  22. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  23. E0000 00:00:1749159646.327594 91 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  24. E0000 00:00:1749159646.327599 87 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  25. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  26. E0000 00:00:1749159646.327869 88 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  27. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
  28. E0000 00:00:1749159646.329187 94 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
  29. E0000 00:00:1749159646.333895 92 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  30. E0000 00:00:1749159646.333897 87 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  31. E0000 00:00:1749159646.333903 89 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  32. E0000 00:00:1749159646.333903 93 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  33. E0000 00:00:1749159646.333901 90 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  34. E0000 00:00:1749159646.334055 91 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  35. E0000 00:00:1749159646.334316 88 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  36. E0000 00:00:1749159646.335834 94 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
  37. ERROR:root:Unable to import wandb.
  38. Traceback (most recent call last):
  39. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  40. import wandb # pylint: disable=g-import-not-at-top
  41. ^^^^^^^^^^^^
  42. ModuleNotFoundError: No module named 'wandb'
  43. ERROR:root:Unable to import wandb.
  44. Traceback (most recent call last):
  45. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  46. import wandb # pylint: disable=g-import-not-at-top
  47. ^^^^^^^^^^^^
  48. ModuleNotFoundError: No module named 'wandb'
  49. ERROR:root:Unable to import wandb.
  50. Traceback (most recent call last):
  51. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  52. import wandb # pylint: disable=g-import-not-at-top
  53. ^^^^^^^^^^^^
  54. ModuleNotFoundError: No module named 'wandb'
  55. ERROR:root:Unable to import wandb.
  56. Traceback (most recent call last):
  57. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  58. import wandb # pylint: disable=g-import-not-at-top
  59. ^^^^^^^^^^^^
  60. ModuleNotFoundError: No module named 'wandb'
  61. ERROR:root:Unable to import wandb.
  62. Traceback (most recent call last):
  63. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  64. import wandb # pylint: disable=g-import-not-at-top
  65. ^^^^^^^^^^^^
  66. ModuleNotFoundError: No module named 'wandb'
  67. ERROR:root:Unable to import wandb.
  68. Traceback (most recent call last):
  69. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  70. import wandb # pylint: disable=g-import-not-at-top
  71. ^^^^^^^^^^^^
  72. ModuleNotFoundError: No module named 'wandb'
  73. ERROR:root:Unable to import wandb.
  74. Traceback (most recent call last):
  75. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  76. import wandb # pylint: disable=g-import-not-at-top
  77. ^^^^^^^^^^^^
  78. ModuleNotFoundError: No module named 'wandb'
  79. I0605 21:40:54.339662 140206541566592 logger_utils.py:59] Removing existing experiment directory /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch because --overwrite was set.
  80. ERROR:root:Unable to import wandb.
  81. Traceback (most recent call last):
  82. File "/algorithmic-efficiency/algoperf/logger_utils.py", line 27, in <module>
  83. import wandb # pylint: disable=g-import-not-at-top
  84. ^^^^^^^^^^^^
  85. ModuleNotFoundError: No module named 'wandb'
  86. [rank0]:[W605 21:40:55.527456479 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  87. [rank7]:[W605 21:40:55.735487210 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  88. [rank5]:[W605 21:40:55.774773963 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  89. [rank2]:[W605 21:40:55.783153949 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  90. [rank3]:[W605 21:40:55.799349970 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  91. [rank6]:[W605 21:40:56.061437091 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  92. [rank4]:[W605 21:40:56.099107785 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  93. [rank1]:[W605 21:40:56.168617336 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
  94. I0605 21:40:57.821369 140206541566592 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  95. I0605 21:40:57.821388 140007983239808 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  96. I0605 21:40:57.821390 140579512062592 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  97. I0605 21:40:57.821390 140231711031936 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  98. I0605 21:40:57.821389 140307471893120 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  99. I0605 21:40:57.821394 140052615066240 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  100. I0605 21:40:57.821389 140359587656320 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  101. I0605 21:40:57.821479 140578171036288 logger_utils.py:81] Creating experiment directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch.
  102. I0605 21:40:58.186334 140206541566592 submission_runner.py:665] Creating directory at /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch/trial_1.
  103. I0605 21:40:58.494310 140206541566592 submission_runner.py:218] Initializing dataset.
  104. I0605 21:40:58.494513 140206541566592 submission_runner.py:229] Initializing model.
  105. W0605 21:41:00.760917 140007983239808 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  106. W0605 21:41:00.760918 140359587656320 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  107. W0605 21:41:00.760933 140206541566592 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  108. W0605 21:41:00.760951 140231711031936 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  109. W0605 21:41:00.760964 140579512062592 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  110. W0605 21:41:00.760967 140307471893120 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  111. W0605 21:41:00.760986 140052615066240 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  112. W0605 21:41:00.761012 140578171036288 submission_runner.py:254] These workloads cannot be fully compiled under current PyTorch version. Proceeding without `torch.compile`.
  113. I0605 21:41:02.145434 140206541566592 submission_runner.py:272] Initializing optimizer.
  114. I0605 21:41:02.146213 140206541566592 submission_runner.py:279] Initializing metrics bundle.
  115. I0605 21:41:02.146409 140206541566592 submission_runner.py:301] Initializing checkpoint and logger.
  116. I0605 21:41:02.146344 140578171036288 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  117. I0605 21:41:02.146351 140231711031936 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  118. I0605 21:41:02.146354 140307471893120 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  119. I0605 21:41:02.146367 140579512062592 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  120. I0605 21:41:02.146378 140007983239808 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  121. I0605 21:41:02.146383 140052615066240 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  122. I0605 21:41:02.146521 140231711031936 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  123. I0605 21:41:02.146524 140307471893120 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  124. I0605 21:41:02.146534 140578171036288 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  125. I0605 21:41:02.146541 140579512062592 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  126. I0605 21:41:02.146442 140359587656320 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  127. I0605 21:41:02.146560 140052615066240 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  128. I0605 21:41:02.146568 140007983239808 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  129. I0605 21:41:02.146618 140206541566592 submission_runner.py:321] Saving meta data to /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch/trial_1/meta_data_0.json.
  130. I0605 21:41:02.146622 140359587656320 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  131. I0605 21:41:02.146804 140206541566592 logger_utils.py:262] Unable to record workload.train_mean information. Continuing without it.
  132. I0605 21:41:02.146863 140206541566592 logger_utils.py:262] Unable to record workload.train_stddev information. Continuing without it.
  133. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  134. To add an exception for this directory, call:
  135.  
  136. git config --global --add safe.directory /algorithmic-efficiency
  137. I0605 21:41:02.286514 140359587656320 logger_utils.py:225] Unable to record git information. Continuing without it.
  138. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  139. To add an exception for this directory, call:
  140.  
  141. git config --global --add safe.directory /algorithmic-efficiency
  142. I0605 21:41:02.304860 140579512062592 logger_utils.py:225] Unable to record git information. Continuing without it.
  143. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  144. To add an exception for this directory, call:
  145.  
  146. git config --global --add safe.directory /algorithmic-efficiency
  147. I0605 21:41:02.312843 140206541566592 logger_utils.py:225] Unable to record git information. Continuing without it.
  148. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  149. To add an exception for this directory, call:
  150.  
  151. git config --global --add safe.directory /algorithmic-efficiency
  152. I0605 21:41:02.326709 140052615066240 logger_utils.py:225] Unable to record git information. Continuing without it.
  153. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  154. To add an exception for this directory, call:
  155.  
  156. git config --global --add safe.directory /algorithmic-efficiency
  157. I0605 21:41:02.327448 140007983239808 logger_utils.py:225] Unable to record git information. Continuing without it.
  158. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  159. To add an exception for this directory, call:
  160.  
  161. git config --global --add safe.directory /algorithmic-efficiency
  162. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  163. To add an exception for this directory, call:
  164.  
  165. git config --global --add safe.directory /algorithmic-efficiency
  166. I0605 21:41:02.337476 140231711031936 logger_utils.py:225] Unable to record git information. Continuing without it.
  167. I0605 21:41:02.337925 140578171036288 logger_utils.py:225] Unable to record git information. Continuing without it.
  168. fatal: detected dubious ownership in repository at '/algorithmic-efficiency'
  169. To add an exception for this directory, call:
  170.  
  171. git config --global --add safe.directory /algorithmic-efficiency
  172. I0605 21:41:02.342284 140307471893120 logger_utils.py:225] Unable to record git information. Continuing without it.
  173. I0605 21:41:02.443264 140206541566592 submission_runner.py:325] Saving flags to /experiment_runs/submissions/rolling_leaderboard/self_tuning/schedule_free_adamw_v2/study_0/ogbg_pytorch/trial_1/flags_0.json.
  174. I0605 21:41:02.494452 140206541566592 submission_runner.py:337] Starting training loop.
  175. I0605 21:41:03.478737 140206541566592 dataset_info.py:690] Load dataset info from /data/ogbg/ogbg_molpcba/0.1.3
  176. I0605 21:41:03.488819 140206541566592 reader.py:261] Creating a tf.data.Dataset reading 8 files located in folders: /data/ogbg/ogbg_molpcba/0.1.3.
  177. WARNING:tensorflow:From /usr/local/lib/python3.11/site-packages/tensorflow_datasets/core/reader.py:101: CounterV2 (from tensorflow.python.data.experimental.ops.counter) is deprecated and will be removed in a future version.
  178. Instructions for updating:
  179. Use `tf.data.Dataset.counter(...)` instead.
  180. W0605 21:41:03.587561 140206541566592 deprecation.py:50] From /usr/local/lib/python3.11/site-packages/tensorflow_datasets/core/reader.py:101: CounterV2 (from tensorflow.python.data.experimental.ops.counter) is deprecated and will be removed in a future version.
  181. Instructions for updating:
  182. Use `tf.data.Dataset.counter(...)` instead.
  183. I0605 21:41:03.654927 140206541566592 logging_logger.py:49] Constructing tf.data.Dataset ogbg_molpcba for split train, from /data/ogbg/ogbg_molpcba/0.1.3
  184. I0605 21:41:14.403036 140206541566592 spec.py:321] Evaluating on the training split.
  185. I0605 21:41:14.406841 140206541566592 dataset_info.py:690] Load dataset info from /data/ogbg/ogbg_molpcba/0.1.3
  186. I0605 21:41:14.413961 140206541566592 reader.py:261] Creating a tf.data.Dataset reading 8 files located in folders: /data/ogbg/ogbg_molpcba/0.1.3.
  187. I0605 21:41:14.481190 140206541566592 logging_logger.py:49] Constructing tf.data.Dataset ogbg_molpcba for split train, from /data/ogbg/ogbg_molpcba/0.1.3
  188. I0605 21:41:47.592525 140206541566592 spec.py:333] Evaluating on the validation split.
  189. I0605 21:41:47.595510 140206541566592 dataset_info.py:690] Load dataset info from /data/ogbg/ogbg_molpcba/0.1.3
  190. I0605 21:41:47.600105 140206541566592 reader.py:261] Creating a tf.data.Dataset reading 1 files located in folders: /data/ogbg/ogbg_molpcba/0.1.3.
  191. I0605 21:41:47.666902 140206541566592 logging_logger.py:49] Constructing tf.data.Dataset ogbg_molpcba for split validation, from /data/ogbg/ogbg_molpcba/0.1.3
  192. I0605 21:42:07.706037 140206541566592 spec.py:349] Evaluating on the test split.
  193. I0605 21:42:07.709291 140206541566592 dataset_info.py:690] Load dataset info from /data/ogbg/ogbg_molpcba/0.1.3
  194. I0605 21:42:07.714253 140206541566592 reader.py:261] Creating a tf.data.Dataset reading 1 files located in folders: /data/ogbg/ogbg_molpcba/0.1.3.
  195. I0605 21:42:07.783147 140206541566592 logging_logger.py:49] Constructing tf.data.Dataset ogbg_molpcba for split test, from /data/ogbg/ogbg_molpcba/0.1.3
  196. I0605 21:42:28.449530 140206541566592 submission_runner.py:469] Time since start: 85.96s, Step: 1, {'train/accuracy': 0.5487086218758792, 'train/loss': 0.7567902562659737, 'train/mean_average_precision': 0.02280857642214985, 'validation/accuracy': 0.5499382158256794, 'validation/loss': 0.7547643230086376, 'validation/mean_average_precision': 0.026879201520133036, 'validation/num_examples': 43793, 'test/accuracy': 0.5476192181026106, 'test/loss': 0.7545978664804425, 'test/mean_average_precision': 0.028469111390666893, 'test/num_examples': 43793, 'score': 11.45370602607727, 'total_duration': 85.95512986183167, 'accumulated_submission_time': 11.45370602607727, 'accumulated_eval_time': 74.04636144638062, 'accumulated_logging_time': 0}
  197. I0605 21:42:28.460644 140155108943616 logging_writer.py:48] [1] accumulated_eval_time=74.0464, accumulated_logging_time=0, accumulated_submission_time=11.4537, global_step=1, preemption_count=0, score=11.4537, test/accuracy=0.547619, test/loss=0.754598, test/mean_average_precision=0.0284691, test/num_examples=43793, total_duration=85.9551, train/accuracy=0.548709, train/loss=0.75679, train/mean_average_precision=0.0228086, validation/accuracy=0.549938, validation/loss=0.754764, validation/mean_average_precision=0.0268792, validation/num_examples=43793
  198. I0605 21:46:29.063182 140206541566592 spec.py:321] Evaluating on the training split.
  199. I0605 21:46:59.233423 140206541566592 spec.py:333] Evaluating on the validation split.
  200. I0605 21:47:02.089110 140206541566592 spec.py:349] Evaluating on the test split.
  201. I0605 21:47:05.551532 140206541566592 submission_runner.py:469] Time since start: 363.06s, Step: 902, {'train/accuracy': 0.9868300669046549, 'train/loss': 0.049930963325571134, 'train/mean_average_precision': 0.07168529164764209, 'validation/accuracy': 0.984265332583155, 'validation/loss': 0.05948749504752348, 'validation/mean_average_precision': 0.0724888437460493, 'validation/num_examples': 43793, 'test/accuracy': 0.9832882584821239, 'test/loss': 0.0628461693817578, 'test/mean_average_precision': 0.07090503722563152, 'test/num_examples': 43793, 'score': 250.78779077529907, 'total_duration': 363.05691480636597, 'accumulated_submission_time': 250.78779077529907, 'accumulated_eval_time': 110.53444004058838, 'accumulated_logging_time': 0.055237770080566406}
  202. I0605 21:47:05.572340 140170150016768 logging_writer.py:48] [902] accumulated_eval_time=110.534, accumulated_logging_time=0.0552378, accumulated_submission_time=250.788, global_step=902, preemption_count=0, score=250.788, test/accuracy=0.983288, test/loss=0.0628462, test/mean_average_precision=0.070905, test/num_examples=43793, total_duration=363.057, train/accuracy=0.98683, train/loss=0.049931, train/mean_average_precision=0.0716853, validation/accuracy=0.984265, validation/loss=0.0594875, validation/mean_average_precision=0.0724888, validation/num_examples=43793
  203. I0605 21:51:06.097799 140206541566592 spec.py:321] Evaluating on the training split.
  204. I0605 21:51:36.808097 140206541566592 spec.py:333] Evaluating on the validation split.
  205. I0605 21:51:39.168776 140206541566592 spec.py:349] Evaluating on the test split.
  206. I0605 21:51:41.675676 140206541566592 submission_runner.py:469] Time since start: 639.18s, Step: 1815, {'train/accuracy': 0.987346426500519, 'train/loss': 0.04600702756738082, 'train/mean_average_precision': 0.11299452885755588, 'validation/accuracy': 0.9846688376375226, 'validation/loss': 0.055522824472459764, 'validation/mean_average_precision': 0.10790966652584111, 'validation/num_examples': 43793, 'test/accuracy': 0.9836365864866424, 'test/loss': 0.0586434098405399, 'test/mean_average_precision': 0.1092228993405109, 'test/num_examples': 43793, 'score': 490.03312635421753, 'total_duration': 639.1812260150909, 'accumulated_submission_time': 490.03312635421753, 'accumulated_eval_time': 146.11223602294922, 'accumulated_logging_time': 0.08680415153503418}
  207. I0605 21:51:41.687723 140170141624064 logging_writer.py:48] [1815] accumulated_eval_time=146.112, accumulated_logging_time=0.0868042, accumulated_submission_time=490.033, global_step=1815, preemption_count=0, score=490.033, test/accuracy=0.983637, test/loss=0.0586434, test/mean_average_precision=0.109223, test/num_examples=43793, total_duration=639.181, train/accuracy=0.987346, train/loss=0.046007, train/mean_average_precision=0.112995, validation/accuracy=0.984669, validation/loss=0.0555228, validation/mean_average_precision=0.10791, validation/num_examples=43793
  208. I0605 21:55:42.383457 140206541566592 spec.py:321] Evaluating on the training split.
  209. I0605 21:56:12.411188 140206541566592 spec.py:333] Evaluating on the validation split.
  210. I0605 21:56:14.871812 140206541566592 spec.py:349] Evaluating on the test split.
  211. I0605 21:56:17.354077 140206541566592 submission_runner.py:469] Time since start: 914.86s, Step: 2727, {'train/accuracy': 0.9877121796238771, 'train/loss': 0.043707295058043834, 'train/mean_average_precision': 0.1407305479948437, 'validation/accuracy': 0.9849598971183894, 'validation/loss': 0.05305284023513709, 'validation/mean_average_precision': 0.13507853115581597, 'validation/num_examples': 43793, 'test/accuracy': 0.9840038682519873, 'test/loss': 0.056025349233573093, 'test/mean_average_precision': 0.13564985068359142, 'test/num_examples': 43793, 'score': 729.4767143726349, 'total_duration': 914.8596456050873, 'accumulated_submission_time': 729.4767143726349, 'accumulated_eval_time': 181.0827398300171, 'accumulated_logging_time': 0.11113643646240234}
  212. I0605 21:56:17.365787 140170150016768 logging_writer.py:48] [2727] accumulated_eval_time=181.083, accumulated_logging_time=0.111136, accumulated_submission_time=729.477, global_step=2727, preemption_count=0, score=729.477, test/accuracy=0.984004, test/loss=0.0560253, test/mean_average_precision=0.13565, test/num_examples=43793, total_duration=914.86, train/accuracy=0.987712, train/loss=0.0437073, train/mean_average_precision=0.140731, validation/accuracy=0.98496, validation/loss=0.0530528, validation/mean_average_precision=0.135079, validation/num_examples=43793
  213. I0605 22:00:18.055022 140206541566592 spec.py:321] Evaluating on the training split.
  214. I0605 22:00:48.126878 140206541566592 spec.py:333] Evaluating on the validation split.
  215. I0605 22:00:50.644643 140206541566592 spec.py:349] Evaluating on the test split.
  216. I0605 22:00:53.112410 140206541566592 submission_runner.py:469] Time since start: 1190.62s, Step: 3641, {'train/accuracy': 0.9879753647382119, 'train/loss': 0.04198438626669612, 'train/mean_average_precision': 0.17400632474454017, 'validation/accuracy': 0.985191283316568, 'validation/loss': 0.0513741232188337, 'validation/mean_average_precision': 0.1572776220993675, 'validation/num_examples': 43793, 'test/accuracy': 0.9842704844876103, 'test/loss': 0.05418292522025323, 'test/mean_average_precision': 0.15736197600286395, 'test/num_examples': 43793, 'score': 968.9358668327332, 'total_duration': 1190.6180188655853, 'accumulated_submission_time': 968.9358668327332, 'accumulated_eval_time': 216.1401264667511, 'accumulated_logging_time': 0.1335761547088623}
  217. I0605 22:00:53.123961 140159630677760 logging_writer.py:48] [3641] accumulated_eval_time=216.14, accumulated_logging_time=0.133576, accumulated_submission_time=968.936, global_step=3641, preemption_count=0, score=968.936, test/accuracy=0.98427, test/loss=0.0541829, test/mean_average_precision=0.157362, test/num_examples=43793, total_duration=1190.62, train/accuracy=0.987975, train/loss=0.0419844, train/mean_average_precision=0.174006, validation/accuracy=0.985191, validation/loss=0.0513741, validation/mean_average_precision=0.157278, validation/num_examples=43793
  218. I0605 22:04:53.624129 140206541566592 spec.py:321] Evaluating on the training split.
  219. I0605 22:05:23.270371 140206541566592 spec.py:333] Evaluating on the validation split.
  220. I0605 22:05:25.595113 140206541566592 spec.py:349] Evaluating on the test split.
  221. I0605 22:05:28.032475 140206541566592 submission_runner.py:469] Time since start: 1465.54s, Step: 4544, {'train/accuracy': 0.988115111160736, 'train/loss': 0.04073314290036186, 'train/mean_average_precision': 0.20560963202541432, 'validation/accuracy': 0.9854182041670625, 'validation/loss': 0.050101660870848344, 'validation/mean_average_precision': 0.17307191954549914, 'validation/num_examples': 43793, 'test/accuracy': 0.9845282556348478, 'test/loss': 0.05279329207450597, 'test/mean_average_precision': 0.17287243665771912, 'test/num_examples': 43793, 'score': 1208.2064571380615, 'total_duration': 1465.5380177497864, 'accumulated_submission_time': 1208.2064571380615, 'accumulated_eval_time': 250.54839181900024, 'accumulated_logging_time': 0.1561717987060547}
  222. I0605 22:05:28.045099 140159513261824 logging_writer.py:48] [4544] accumulated_eval_time=250.548, accumulated_logging_time=0.156172, accumulated_submission_time=1208.21, global_step=4544, preemption_count=0, score=1208.21, test/accuracy=0.984528, test/loss=0.0527933, test/mean_average_precision=0.172872, test/num_examples=43793, total_duration=1465.54, train/accuracy=0.988115, train/loss=0.0407331, train/mean_average_precision=0.20561, validation/accuracy=0.985418, validation/loss=0.0501017, validation/mean_average_precision=0.173072, validation/num_examples=43793
  223. I0605 22:09:28.665737 140206541566592 spec.py:321] Evaluating on the training split.
  224. I0605 22:09:58.459216 140206541566592 spec.py:333] Evaluating on the validation split.
  225. I0605 22:10:00.908166 140206541566592 spec.py:349] Evaluating on the test split.
  226. I0605 22:10:03.408453 140206541566592 submission_runner.py:469] Time since start: 1740.91s, Step: 5458, {'train/accuracy': 0.9884641716336489, 'train/loss': 0.0394263024640572, 'train/mean_average_precision': 0.2214475080514221, 'validation/accuracy': 0.9856439071954612, 'validation/loss': 0.04900826911351482, 'validation/mean_average_precision': 0.18697227619744636, 'validation/num_examples': 43793, 'test/accuracy': 0.9847771816936997, 'test/loss': 0.05161484199513183, 'test/mean_average_precision': 0.1874613019717409, 'test/num_examples': 43793, 'score': 1447.595309972763, 'total_duration': 1740.9140267372131, 'accumulated_submission_time': 1447.595309972763, 'accumulated_eval_time': 285.2910010814667, 'accumulated_logging_time': 0.17986011505126953}
  227. I0605 22:10:03.420285 140159630677760 logging_writer.py:48] [5458] accumulated_eval_time=285.291, accumulated_logging_time=0.17986, accumulated_submission_time=1447.6, global_step=5458, preemption_count=0, score=1447.6, test/accuracy=0.984777, test/loss=0.0516148, test/mean_average_precision=0.187461, test/num_examples=43793, total_duration=1740.91, train/accuracy=0.988464, train/loss=0.0394263, train/mean_average_precision=0.221448, validation/accuracy=0.985644, validation/loss=0.0490083, validation/mean_average_precision=0.186972, validation/num_examples=43793
  228. I0605 22:14:04.008106 140206541566592 spec.py:321] Evaluating on the training split.
  229. I0605 22:14:34.221749 140206541566592 spec.py:333] Evaluating on the validation split.
  230. I0605 22:14:36.495209 140206541566592 spec.py:349] Evaluating on the test split.
  231. I0605 22:14:38.882772 140206541566592 submission_runner.py:469] Time since start: 2016.39s, Step: 6368, {'train/accuracy': 0.9888835496826034, 'train/loss': 0.03811229359292624, 'train/mean_average_precision': 0.2431977395941909, 'validation/accuracy': 0.9858525607145205, 'validation/loss': 0.04810503203075082, 'validation/mean_average_precision': 0.20216918724889832, 'validation/num_examples': 43793, 'test/accuracy': 0.9849907273990091, 'test/loss': 0.05067532118200707, 'test/mean_average_precision': 0.2010166675714519, 'test/num_examples': 43793, 'score': 1686.9271593093872, 'total_duration': 2016.3883547782898, 'accumulated_submission_time': 1686.9271593093872, 'accumulated_eval_time': 320.1658613681793, 'accumulated_logging_time': 0.2037489414215088}
  232. I0605 22:14:38.894670 140159513261824 logging_writer.py:48] [6368] accumulated_eval_time=320.166, accumulated_logging_time=0.203749, accumulated_submission_time=1686.93, global_step=6368, preemption_count=0, score=1686.93, test/accuracy=0.984991, test/loss=0.0506753, test/mean_average_precision=0.201017, test/num_examples=43793, total_duration=2016.39, train/accuracy=0.988884, train/loss=0.0381123, train/mean_average_precision=0.243198, validation/accuracy=0.985853, validation/loss=0.048105, validation/mean_average_precision=0.202169, validation/num_examples=43793
  233. I0605 22:18:39.505685 140206541566592 spec.py:321] Evaluating on the training split.
  234. I0605 22:19:09.481040 140206541566592 spec.py:333] Evaluating on the validation split.
  235. I0605 22:19:11.835303 140206541566592 spec.py:349] Evaluating on the test split.
  236. I0605 22:19:14.370511 140206541566592 submission_runner.py:469] Time since start: 2291.88s, Step: 7287, {'train/accuracy': 0.9891792048766057, 'train/loss': 0.03690697681165463, 'train/mean_average_precision': 0.2827011422888971, 'validation/accuracy': 0.9860733924545366, 'validation/loss': 0.04729029002027268, 'validation/mean_average_precision': 0.21619626092819227, 'validation/num_examples': 43793, 'test/accuracy': 0.9851566781049104, 'test/loss': 0.049899816211699186, 'test/mean_average_precision': 0.21196895358981632, 'test/num_examples': 43793, 'score': 1926.3120086193085, 'total_duration': 2291.87606215477, 'accumulated_submission_time': 1926.3120086193085, 'accumulated_eval_time': 355.0306644439697, 'accumulated_logging_time': 0.22652864456176758}
  237. I0605 22:19:14.382106 140159630677760 logging_writer.py:48] [7287] accumulated_eval_time=355.031, accumulated_logging_time=0.226529, accumulated_submission_time=1926.31, global_step=7287, preemption_count=0, score=1926.31, test/accuracy=0.985157, test/loss=0.0498998, test/mean_average_precision=0.211969, test/num_examples=43793, total_duration=2291.88, train/accuracy=0.989179, train/loss=0.036907, train/mean_average_precision=0.282701, validation/accuracy=0.986073, validation/loss=0.0472903, validation/mean_average_precision=0.216196, validation/num_examples=43793
  238. I0605 22:23:14.939334 140206541566592 spec.py:321] Evaluating on the training split.
  239. I0605 22:23:45.097196 140206541566592 spec.py:333] Evaluating on the validation split.
  240. I0605 22:23:47.481503 140206541566592 spec.py:349] Evaluating on the test split.
  241. I0605 22:23:50.038079 140206541566592 submission_runner.py:469] Time since start: 2567.54s, Step: 8209, {'train/accuracy': 0.9895124087729301, 'train/loss': 0.035788877355866056, 'train/mean_average_precision': 0.28230390216710727, 'validation/accuracy': 0.9862296796234818, 'validation/loss': 0.0466140784445895, 'validation/mean_average_precision': 0.22780603918726502, 'validation/num_examples': 43793, 'test/accuracy': 0.9853276831470319, 'test/loss': 0.04924081763997037, 'test/mean_average_precision': 0.22294419473239152, 'test/num_examples': 43793, 'score': 2165.6236696243286, 'total_duration': 2567.543551683426, 'accumulated_submission_time': 2165.6236696243286, 'accumulated_eval_time': 390.12936544418335, 'accumulated_logging_time': 0.24913430213928223}
  242. I0605 22:23:50.049961 140159513261824 logging_writer.py:48] [8209] accumulated_eval_time=390.129, accumulated_logging_time=0.249134, accumulated_submission_time=2165.62, global_step=8209, preemption_count=0, score=2165.62, test/accuracy=0.985328, test/loss=0.0492408, test/mean_average_precision=0.222944, test/num_examples=43793, total_duration=2567.54, train/accuracy=0.989512, train/loss=0.0357889, train/mean_average_precision=0.282304, validation/accuracy=0.98623, validation/loss=0.0466141, validation/mean_average_precision=0.227806, validation/num_examples=43793
  243. I0605 22:27:50.619583 140206541566592 spec.py:321] Evaluating on the training split.
  244. I0605 22:28:20.911055 140206541566592 spec.py:333] Evaluating on the validation split.
  245. I0605 22:28:23.371788 140206541566592 spec.py:349] Evaluating on the test split.
  246. I0605 22:28:25.904047 140206541566592 submission_runner.py:469] Time since start: 2843.41s, Step: 9125, {'train/accuracy': 0.9896322264420424, 'train/loss': 0.035185323509710015, 'train/mean_average_precision': 0.314140397545438, 'validation/accuracy': 0.9863563331214323, 'validation/loss': 0.04610260902146371, 'validation/mean_average_precision': 0.23634763131809192, 'validation/num_examples': 43793, 'test/accuracy': 0.9854822615964374, 'test/loss': 0.048707729954397254, 'test/mean_average_precision': 0.23116805238734756, 'test/num_examples': 43793, 'score': 2404.9517436027527, 'total_duration': 2843.409616470337, 'accumulated_submission_time': 2404.9517436027527, 'accumulated_eval_time': 425.41396856307983, 'accumulated_logging_time': 0.27196192741394043}
  247. I0605 22:28:25.916159 140159630677760 logging_writer.py:48] [9125] accumulated_eval_time=425.414, accumulated_logging_time=0.271962, accumulated_submission_time=2404.95, global_step=9125, preemption_count=0, score=2404.95, test/accuracy=0.985482, test/loss=0.0487077, test/mean_average_precision=0.231168, test/num_examples=43793, total_duration=2843.41, train/accuracy=0.989632, train/loss=0.0351853, train/mean_average_precision=0.31414, validation/accuracy=0.986356, validation/loss=0.0461026, validation/mean_average_precision=0.236348, validation/num_examples=43793
  248. I0605 22:32:26.490807 140206541566592 spec.py:321] Evaluating on the training split.
  249. I0605 22:32:56.897090 140206541566592 spec.py:333] Evaluating on the validation split.
  250. I0605 22:32:59.392972 140206541566592 spec.py:349] Evaluating on the test split.
  251. I0605 22:33:01.932400 140206541566592 submission_runner.py:469] Time since start: 3119.44s, Step: 10037, {'train/accuracy': 0.989865099883788, 'train/loss': 0.03417648492505538, 'train/mean_average_precision': 0.34122646946519486, 'validation/accuracy': 0.9864614717623591, 'validation/loss': 0.04571230655403436, 'validation/mean_average_precision': 0.2409976172140284, 'validation/num_examples': 43793, 'test/accuracy': 0.9856208346478117, 'test/loss': 0.048302830238745785, 'test/mean_average_precision': 0.23866705916049347, 'test/num_examples': 43793, 'score': 2644.287879705429, 'total_duration': 3119.437887430191, 'accumulated_submission_time': 2644.287879705429, 'accumulated_eval_time': 460.8554883003235, 'accumulated_logging_time': 0.2950148582458496}
  252. I0605 22:33:01.946323 140159513261824 logging_writer.py:48] [10037] accumulated_eval_time=460.855, accumulated_logging_time=0.295015, accumulated_submission_time=2644.29, global_step=10037, preemption_count=0, score=2644.29, test/accuracy=0.985621, test/loss=0.0483028, test/mean_average_precision=0.238667, test/num_examples=43793, total_duration=3119.44, train/accuracy=0.989865, train/loss=0.0341765, train/mean_average_precision=0.341226, validation/accuracy=0.986461, validation/loss=0.0457123, validation/mean_average_precision=0.240998, validation/num_examples=43793
  253. I0605 22:37:02.694463 140206541566592 spec.py:321] Evaluating on the training split.
  254. I0605 22:37:33.553227 140206541566592 spec.py:333] Evaluating on the validation split.
  255. I0605 22:37:35.943207 140206541566592 spec.py:349] Evaluating on the test split.
  256. I0605 22:37:38.364780 140206541566592 submission_runner.py:469] Time since start: 3395.87s, Step: 10937, {'train/accuracy': 0.9900276390405431, 'train/loss': 0.03363330110483381, 'train/mean_average_precision': 0.35886476044755355, 'validation/accuracy': 0.9865731054544628, 'validation/loss': 0.045422645665730566, 'validation/mean_average_precision': 0.24390431210719507, 'validation/num_examples': 43793, 'test/accuracy': 0.9857202365934785, 'test/loss': 0.04802692797655125, 'test/mean_average_precision': 0.24356935670676236, 'test/num_examples': 43793, 'score': 2883.7634630203247, 'total_duration': 3395.8703184127808, 'accumulated_submission_time': 2883.7634630203247, 'accumulated_eval_time': 496.5256667137146, 'accumulated_logging_time': 0.31999945640563965}
  257. I0605 22:37:38.376707 140159630677760 logging_writer.py:48] [10937] accumulated_eval_time=496.526, accumulated_logging_time=0.319999, accumulated_submission_time=2883.76, global_step=10937, preemption_count=0, score=2883.76, test/accuracy=0.98572, test/loss=0.0480269, test/mean_average_precision=0.243569, test/num_examples=43793, total_duration=3395.87, train/accuracy=0.990028, train/loss=0.0336333, train/mean_average_precision=0.358865, validation/accuracy=0.986573, validation/loss=0.0454226, validation/mean_average_precision=0.243904, validation/num_examples=43793
  258. I0605 22:41:39.093713 140206541566592 spec.py:321] Evaluating on the training split.
  259. I0605 22:42:09.275288 140206541566592 spec.py:333] Evaluating on the validation split.
  260. I0605 22:42:11.721524 140206541566592 spec.py:349] Evaluating on the test split.
  261. I0605 22:42:14.175299 140206541566592 submission_runner.py:469] Time since start: 3671.68s, Step: 11850, {'train/accuracy': 0.9904030800929371, 'train/loss': 0.032375967380599016, 'train/mean_average_precision': 0.3713264503189626, 'validation/accuracy': 0.9867078777663844, 'validation/loss': 0.04513779898039875, 'validation/mean_average_precision': 0.24741498888661106, 'validation/num_examples': 43793, 'test/accuracy': 0.9857989999995788, 'test/loss': 0.047761831990494476, 'test/mean_average_precision': 0.24795659329216024, 'test/num_examples': 43793, 'score': 3123.2585492134094, 'total_duration': 3671.6808309555054, 'accumulated_submission_time': 3123.2585492134094, 'accumulated_eval_time': 531.6070852279663, 'accumulated_logging_time': 0.34432554244995117}
  262. I0605 22:42:14.187629 140159513261824 logging_writer.py:48] [11850] accumulated_eval_time=531.607, accumulated_logging_time=0.344326, accumulated_submission_time=3123.26, global_step=11850, preemption_count=0, score=3123.26, test/accuracy=0.985799, test/loss=0.0477618, test/mean_average_precision=0.247957, test/num_examples=43793, total_duration=3671.68, train/accuracy=0.990403, train/loss=0.032376, train/mean_average_precision=0.371326, validation/accuracy=0.986708, validation/loss=0.0451378, validation/mean_average_precision=0.247415, validation/num_examples=43793
  263. I0605 22:46:14.740194 140206541566592 spec.py:321] Evaluating on the training split.
  264. I0605 22:46:45.261592 140206541566592 spec.py:333] Evaluating on the validation split.
  265. I0605 22:46:47.647310 140206541566592 spec.py:349] Evaluating on the test split.
  266. I0605 22:46:50.059598 140206541566592 submission_runner.py:469] Time since start: 3947.57s, Step: 12757, {'train/accuracy': 0.9906971846717189, 'train/loss': 0.03143831733234139, 'train/mean_average_precision': 0.3880912178424202, 'validation/accuracy': 0.9867643035234841, 'validation/loss': 0.044921508701947784, 'validation/mean_average_precision': 0.25112313571109884, 'validation/num_examples': 43793, 'test/accuracy': 0.9858912416356, 'test/loss': 0.04756825354572216, 'test/mean_average_precision': 0.2518388757864706, 'test/num_examples': 43793, 'score': 3362.600525856018, 'total_duration': 3947.565145254135, 'accumulated_submission_time': 3362.600525856018, 'accumulated_eval_time': 566.9263648986816, 'accumulated_logging_time': 0.36762285232543945}
  267. I0605 22:46:50.071693 140159630677760 logging_writer.py:48] [12757] accumulated_eval_time=566.926, accumulated_logging_time=0.367623, accumulated_submission_time=3362.6, global_step=12757, preemption_count=0, score=3362.6, test/accuracy=0.985891, test/loss=0.0475683, test/mean_average_precision=0.251839, test/num_examples=43793, total_duration=3947.57, train/accuracy=0.990697, train/loss=0.0314383, train/mean_average_precision=0.388091, validation/accuracy=0.986764, validation/loss=0.0449215, validation/mean_average_precision=0.251123, validation/num_examples=43793
  268. I0605 22:50:50.602522 140206541566592 spec.py:321] Evaluating on the training split.
  269. I0605 22:51:20.736195 140206541566592 spec.py:333] Evaluating on the validation split.
  270. I0605 22:51:23.172898 140206541566592 spec.py:349] Evaluating on the test split.
  271. I0605 22:51:25.623900 140206541566592 submission_runner.py:469] Time since start: 4223.13s, Step: 13667, {'train/accuracy': 0.9908064178581146, 'train/loss': 0.031106822679832308, 'train/mean_average_precision': 0.40308190199129657, 'validation/accuracy': 0.9868341253236362, 'validation/loss': 0.044751124607962774, 'validation/mean_average_precision': 0.2543841996417803, 'validation/num_examples': 43793, 'test/accuracy': 0.9859780077407159, 'test/loss': 0.04739616261105324, 'test/mean_average_precision': 0.2582237351994987, 'test/num_examples': 43793, 'score': 3601.9156806468964, 'total_duration': 4223.129507780075, 'accumulated_submission_time': 3601.9156806468964, 'accumulated_eval_time': 601.9476544857025, 'accumulated_logging_time': 0.39289426803588867}
  272. I0605 22:51:25.636026 140159513261824 logging_writer.py:48] [13667] accumulated_eval_time=601.948, accumulated_logging_time=0.392894, accumulated_submission_time=3601.92, global_step=13667, preemption_count=0, score=3601.92, test/accuracy=0.985978, test/loss=0.0473962, test/mean_average_precision=0.258224, test/num_examples=43793, total_duration=4223.13, train/accuracy=0.990806, train/loss=0.0311068, train/mean_average_precision=0.403082, validation/accuracy=0.986834, validation/loss=0.0447511, validation/mean_average_precision=0.254384, validation/num_examples=43793
  273. I0605 22:55:26.197944 140206541566592 spec.py:321] Evaluating on the training split.
  274. I0605 22:55:56.524410 140206541566592 spec.py:333] Evaluating on the validation split.
  275. I0605 22:55:58.973257 140206541566592 spec.py:349] Evaluating on the test split.
  276. I0605 22:56:01.423302 140206541566592 submission_runner.py:469] Time since start: 4498.93s, Step: 14582, {'train/accuracy': 0.9910017785967316, 'train/loss': 0.030296972690410852, 'train/mean_average_precision': 0.4192689310267741, 'validation/accuracy': 0.9868828382074633, 'validation/loss': 0.044635510794166146, 'validation/mean_average_precision': 0.25617875895009296, 'validation/num_examples': 43793, 'test/accuracy': 0.9860399233594151, 'test/loss': 0.04728132782888039, 'test/mean_average_precision': 0.2619333258901303, 'test/num_examples': 43793, 'score': 3841.241468667984, 'total_duration': 4498.92883515358, 'accumulated_submission_time': 3841.241468667984, 'accumulated_eval_time': 637.1728610992432, 'accumulated_logging_time': 0.41543030738830566}
  277. I0605 22:56:01.435550 140159630677760 logging_writer.py:48] [14582] accumulated_eval_time=637.173, accumulated_logging_time=0.41543, accumulated_submission_time=3841.24, global_step=14582, preemption_count=0, score=3841.24, test/accuracy=0.98604, test/loss=0.0472813, test/mean_average_precision=0.261933, test/num_examples=43793, total_duration=4498.93, train/accuracy=0.991002, train/loss=0.030297, train/mean_average_precision=0.419269, validation/accuracy=0.986883, validation/loss=0.0446355, validation/mean_average_precision=0.256179, validation/num_examples=43793
  278. I0605 23:00:02.011968 140206541566592 spec.py:321] Evaluating on the training split.
  279. I0605 23:00:32.410017 140206541566592 spec.py:333] Evaluating on the validation split.
  280. I0605 23:00:34.874189 140206541566592 spec.py:349] Evaluating on the test split.
  281. I0605 23:00:37.329072 140206541566592 submission_runner.py:469] Time since start: 4774.83s, Step: 15486, {'train/accuracy': 0.9910635079196393, 'train/loss': 0.0300505129987845, 'train/mean_average_precision': 0.41855093291242995, 'validation/accuracy': 0.9868690362237122, 'validation/loss': 0.04465052742860112, 'validation/mean_average_precision': 0.2567822049015745, 'validation/num_examples': 43793, 'test/accuracy': 0.9860651950405168, 'test/loss': 0.04731451994441072, 'test/mean_average_precision': 0.2591340860132876, 'test/num_examples': 43793, 'score': 4080.605883359909, 'total_duration': 4774.834667921066, 'accumulated_submission_time': 4080.605883359909, 'accumulated_eval_time': 672.4898743629456, 'accumulated_logging_time': 0.43826937675476074}
  282. I0605 23:00:37.341002 140159513261824 logging_writer.py:48] [15486] accumulated_eval_time=672.49, accumulated_logging_time=0.438269, accumulated_submission_time=4080.61, global_step=15486, preemption_count=0, score=4080.61, test/accuracy=0.986065, test/loss=0.0473145, test/mean_average_precision=0.259134, test/num_examples=43793, total_duration=4774.83, train/accuracy=0.991064, train/loss=0.0300505, train/mean_average_precision=0.418551, validation/accuracy=0.986869, validation/loss=0.0446505, validation/mean_average_precision=0.256782, validation/num_examples=43793
  283. I0605 23:04:37.992451 140206541566592 spec.py:321] Evaluating on the training split.
  284. I0605 23:05:07.960309 140206541566592 spec.py:333] Evaluating on the validation split.
  285. I0605 23:05:10.357159 140206541566592 spec.py:349] Evaluating on the test split.
  286. I0605 23:05:12.786864 140206541566592 submission_runner.py:469] Time since start: 5050.29s, Step: 16404, {'train/accuracy': 0.9911666266276546, 'train/loss': 0.029667278515892535, 'train/mean_average_precision': 0.44568383980824444, 'validation/accuracy': 0.9868410263155117, 'validation/loss': 0.044823600879714084, 'validation/mean_average_precision': 0.2601742198547765, 'validation/num_examples': 43793, 'test/accuracy': 0.9860550863680762, 'test/loss': 0.04748075034990749, 'test/mean_average_precision': 0.25852091458358784, 'test/num_examples': 43793, 'score': 4320.050471305847, 'total_duration': 5050.292399644852, 'accumulated_submission_time': 4320.050471305847, 'accumulated_eval_time': 707.2841956615448, 'accumulated_logging_time': 0.46062302589416504}
  287. I0605 23:05:12.799417 140159630677760 logging_writer.py:48] [16404] accumulated_eval_time=707.284, accumulated_logging_time=0.460623, accumulated_submission_time=4320.05, global_step=16404, preemption_count=0, score=4320.05, test/accuracy=0.986055, test/loss=0.0474808, test/mean_average_precision=0.258521, test/num_examples=43793, total_duration=5050.29, train/accuracy=0.991167, train/loss=0.0296673, train/mean_average_precision=0.445684, validation/accuracy=0.986841, validation/loss=0.0448236, validation/mean_average_precision=0.260174, validation/num_examples=43793
  288. I0605 23:09:13.352592 140206541566592 spec.py:321] Evaluating on the training split.
  289. I0605 23:09:43.109705 140206541566592 spec.py:333] Evaluating on the validation split.
  290. I0605 23:09:45.523474 140206541566592 spec.py:349] Evaluating on the test split.
  291. I0605 23:09:47.975932 140206541566592 submission_runner.py:469] Time since start: 5325.48s, Step: 17319, {'train/accuracy': 0.9914202092458955, 'train/loss': 0.028898587186446285, 'train/mean_average_precision': 0.4497590228960248, 'validation/accuracy': 0.9868698481051094, 'validation/loss': 0.04488496452484235, 'validation/mean_average_precision': 0.2607189226653791, 'validation/num_examples': 43793, 'test/accuracy': 0.986081621633233, 'test/loss': 0.04753425194665654, 'test/mean_average_precision': 0.2587980441248242, 'test/num_examples': 43793, 'score': 4559.381223917007, 'total_duration': 5325.481420755386, 'accumulated_submission_time': 4559.381223917007, 'accumulated_eval_time': 741.9073550701141, 'accumulated_logging_time': 0.4832913875579834}
  292. I0605 23:09:47.988408 140159513261824 logging_writer.py:48] [17319] accumulated_eval_time=741.907, accumulated_logging_time=0.483291, accumulated_submission_time=4559.38, global_step=17319, preemption_count=0, score=4559.38, test/accuracy=0.986082, test/loss=0.0475343, test/mean_average_precision=0.258798, test/num_examples=43793, total_duration=5325.48, train/accuracy=0.99142, train/loss=0.0288986, train/mean_average_precision=0.449759, validation/accuracy=0.98687, validation/loss=0.044885, validation/mean_average_precision=0.260719, validation/num_examples=43793
  293. I0605 23:13:48.533288 140206541566592 spec.py:321] Evaluating on the training split.
  294. I0605 23:14:18.556011 140206541566592 spec.py:333] Evaluating on the validation split.
  295. I0605 23:14:20.944454 140206541566592 spec.py:349] Evaluating on the test split.
  296. I0605 23:14:23.345668 140206541566592 submission_runner.py:469] Time since start: 5600.85s, Step: 18233, {'train/accuracy': 0.9913886505299813, 'train/loss': 0.028763286341455467, 'train/mean_average_precision': 0.4576594157542204, 'validation/accuracy': 0.9868657886981238, 'validation/loss': 0.045060521191322284, 'validation/mean_average_precision': 0.2605578340179095, 'validation/num_examples': 43793, 'test/accuracy': 0.986082885217288, 'test/loss': 0.047721321617311775, 'test/mean_average_precision': 0.25731959779494995, 'test/num_examples': 43793, 'score': 4798.7270612716675, 'total_duration': 5600.85126543045, 'accumulated_submission_time': 4798.7270612716675, 'accumulated_eval_time': 776.7196938991547, 'accumulated_logging_time': 0.507072925567627}
  297. I0605 23:14:23.358316 140159630677760 logging_writer.py:48] [18233] accumulated_eval_time=776.72, accumulated_logging_time=0.507073, accumulated_submission_time=4798.73, global_step=18233, preemption_count=0, score=4798.73, test/accuracy=0.986083, test/loss=0.0477213, test/mean_average_precision=0.25732, test/num_examples=43793, total_duration=5600.85, train/accuracy=0.991389, train/loss=0.0287633, train/mean_average_precision=0.457659, validation/accuracy=0.986866, validation/loss=0.0450605, validation/mean_average_precision=0.260558, validation/num_examples=43793
  298. I0605 23:18:23.981101 140206541566592 spec.py:321] Evaluating on the training split.
  299. I0605 23:18:53.648892 140206541566592 spec.py:333] Evaluating on the validation split.
  300. I0605 23:18:56.055882 140206541566592 spec.py:349] Evaluating on the test split.
  301. I0605 23:18:58.516973 140206541566592 submission_runner.py:469] Time since start: 5876.02s, Step: 19153, {'train/accuracy': 0.991357835827663, 'train/loss': 0.028554834337491907, 'train/mean_average_precision': 0.4698251917578294, 'validation/accuracy': 0.986884056029559, 'validation/loss': 0.04519286737328764, 'validation/mean_average_precision': 0.2620335385719013, 'validation/num_examples': 43793, 'test/accuracy': 0.9860719341554772, 'test/loss': 0.04785858172587892, 'test/mean_average_precision': 0.25568615931881655, 'test/num_examples': 43793, 'score': 5038.144383907318, 'total_duration': 5876.022454500198, 'accumulated_submission_time': 5038.144383907318, 'accumulated_eval_time': 811.255402803421, 'accumulated_logging_time': 0.5297536849975586}
  302. I0605 23:18:58.529653 140159513261824 logging_writer.py:48] [19153] accumulated_eval_time=811.255, accumulated_logging_time=0.529754, accumulated_submission_time=5038.14, global_step=19153, preemption_count=0, score=5038.14, test/accuracy=0.986072, test/loss=0.0478586, test/mean_average_precision=0.255686, test/num_examples=43793, total_duration=5876.02, train/accuracy=0.991358, train/loss=0.0285548, train/mean_average_precision=0.469825, validation/accuracy=0.986884, validation/loss=0.0451929, validation/mean_average_precision=0.262034, validation/num_examples=43793
  303. I0605 23:22:58.997235 140206541566592 spec.py:321] Evaluating on the training split.
  304. I0605 23:23:28.857002 140206541566592 spec.py:333] Evaluating on the validation split.
  305. I0605 23:23:31.290413 140206541566592 spec.py:349] Evaluating on the test split.
  306. I0605 23:23:33.751463 140206541566592 submission_runner.py:469] Time since start: 6151.26s, Step: 20074, {'train/accuracy': 0.9912297227201009, 'train/loss': 0.028678008335818624, 'train/mean_average_precision': 0.46541085594082504, 'validation/accuracy': 0.9868195114584881, 'validation/loss': 0.04558290027072185, 'validation/mean_average_precision': 0.2638449131435517, 'validation/num_examples': 43793, 'test/accuracy': 0.986039080970045, 'test/loss': 0.04831349831037752, 'test/mean_average_precision': 0.2569341219466754, 'test/num_examples': 43793, 'score': 5277.396889209747, 'total_duration': 6151.257014513016, 'accumulated_submission_time': 5277.396889209747, 'accumulated_eval_time': 846.0095317363739, 'accumulated_logging_time': 0.5527918338775635}
  307. I0605 23:23:33.764058 140159630677760 logging_writer.py:48] [20074] accumulated_eval_time=846.01, accumulated_logging_time=0.552792, accumulated_submission_time=5277.4, global_step=20074, preemption_count=0, score=5277.4, test/accuracy=0.986039, test/loss=0.0483135, test/mean_average_precision=0.256934, test/num_examples=43793, total_duration=6151.26, train/accuracy=0.99123, train/loss=0.028678, train/mean_average_precision=0.465411, validation/accuracy=0.98682, validation/loss=0.0455829, validation/mean_average_precision=0.263845, validation/num_examples=43793
  308. I0605 23:27:34.275405 140206541566592 spec.py:321] Evaluating on the training split.
  309. I0605 23:28:04.222022 140206541566592 spec.py:333] Evaluating on the validation split.
  310. I0605 23:28:06.638330 140206541566592 spec.py:349] Evaluating on the test split.
  311. I0605 23:28:09.096451 140206541566592 submission_runner.py:469] Time since start: 6426.60s, Step: 20983, {'train/accuracy': 0.9912423538656753, 'train/loss': 0.02858970553844494, 'train/mean_average_precision': 0.47229784768795147, 'validation/accuracy': 0.9868211352212823, 'validation/loss': 0.045647711240376164, 'validation/mean_average_precision': 0.26544775091409106, 'validation/num_examples': 43793, 'test/accuracy': 0.9860028582271326, 'test/loss': 0.04839539435194775, 'test/mean_average_precision': 0.25604010037332015, 'test/num_examples': 43793, 'score': 5516.7090446949005, 'total_duration': 6426.60202050209, 'accumulated_submission_time': 5516.7090446949005, 'accumulated_eval_time': 880.8305199146271, 'accumulated_logging_time': 0.5754847526550293}
  312. I0605 23:28:09.109385 140159513261824 logging_writer.py:48] [20983] accumulated_eval_time=880.831, accumulated_logging_time=0.575485, accumulated_submission_time=5516.71, global_step=20983, preemption_count=0, score=5516.71, test/accuracy=0.986003, test/loss=0.0483954, test/mean_average_precision=0.25604, test/num_examples=43793, total_duration=6426.6, train/accuracy=0.991242, train/loss=0.0285897, train/mean_average_precision=0.472298, validation/accuracy=0.986821, validation/loss=0.0456477, validation/mean_average_precision=0.265448, validation/num_examples=43793
  313. I0605 23:32:09.779481 140206541566592 spec.py:321] Evaluating on the training split.
  314. I0605 23:32:38.786880 140206541566592 spec.py:333] Evaluating on the validation split.
  315. I0605 23:32:41.265414 140206541566592 spec.py:349] Evaluating on the test split.
  316. I0605 23:32:43.746407 140206541566592 submission_runner.py:469] Time since start: 6701.25s, Step: 21925, {'train/accuracy': 0.9912574718746641, 'train/loss': 0.028489678525902465, 'train/mean_average_precision': 0.4687760065129034, 'validation/accuracy': 0.9867781055072351, 'validation/loss': 0.04560685711476025, 'validation/mean_average_precision': 0.26618073087099775, 'validation/num_examples': 43793, 'test/accuracy': 0.9859687414576453, 'test/loss': 0.04833030990136042, 'test/mean_average_precision': 0.2570975514932258, 'test/num_examples': 43793, 'score': 5756.187577486038, 'total_duration': 6701.251999616623, 'accumulated_submission_time': 5756.187577486038, 'accumulated_eval_time': 914.7973120212555, 'accumulated_logging_time': 0.5986073017120361}
  317. I0605 23:32:43.758740 140159630677760 logging_writer.py:48] [21925] accumulated_eval_time=914.797, accumulated_logging_time=0.598607, accumulated_submission_time=5756.19, global_step=21925, preemption_count=0, score=5756.19, test/accuracy=0.985969, test/loss=0.0483303, test/mean_average_precision=0.257098, test/num_examples=43793, total_duration=6701.25, train/accuracy=0.991257, train/loss=0.0284897, train/mean_average_precision=0.468776, validation/accuracy=0.986778, validation/loss=0.0456069, validation/mean_average_precision=0.266181, validation/num_examples=43793
  318. I0605 23:36:44.245711 140206541566592 spec.py:321] Evaluating on the training split.
  319. I0605 23:37:13.051341 140206541566592 spec.py:333] Evaluating on the validation split.
  320. I0605 23:37:15.539235 140206541566592 spec.py:349] Evaluating on the test split.
  321. I0605 23:37:18.014754 140206541566592 submission_runner.py:469] Time since start: 6975.52s, Step: 22873, {'train/accuracy': 0.9913276366054111, 'train/loss': 0.0283741058291139, 'train/mean_average_precision': 0.4724600856241391, 'validation/accuracy': 0.9867720163967567, 'validation/loss': 0.045515257230412755, 'validation/mean_average_precision': 0.2672145604043564, 'validation/num_examples': 43793, 'test/accuracy': 0.9859767441566608, 'test/loss': 0.04822133564835972, 'test/mean_average_precision': 0.2583805656681818, 'test/num_examples': 43793, 'score': 5995.44772028923, 'total_duration': 6975.5203449726105, 'accumulated_submission_time': 5995.44772028923, 'accumulated_eval_time': 948.5662040710449, 'accumulated_logging_time': 0.620842456817627}
  322. I0605 23:37:18.027403 140159513261824 logging_writer.py:48] [22873] accumulated_eval_time=948.566, accumulated_logging_time=0.620842, accumulated_submission_time=5995.45, global_step=22873, preemption_count=0, score=5995.45, test/accuracy=0.985977, test/loss=0.0482213, test/mean_average_precision=0.258381, test/num_examples=43793, total_duration=6975.52, train/accuracy=0.991328, train/loss=0.0283741, train/mean_average_precision=0.47246, validation/accuracy=0.986772, validation/loss=0.0455153, validation/mean_average_precision=0.267215, validation/num_examples=43793
  323. I0605 23:41:18.545394 140206541566592 spec.py:321] Evaluating on the training split.
  324. I0605 23:41:47.361901 140206541566592 spec.py:333] Evaluating on the validation split.
  325. I0605 23:41:49.801356 140206541566592 spec.py:349] Evaluating on the test split.
  326. I0605 23:41:52.282247 140206541566592 submission_runner.py:469] Time since start: 7249.79s, Step: 23818, {'train/accuracy': 0.9912558578620843, 'train/loss': 0.028476817710266417, 'train/mean_average_precision': 0.4740971192589781, 'validation/accuracy': 0.9867850064991106, 'validation/loss': 0.04538062128919459, 'validation/mean_average_precision': 0.2708101972295732, 'validation/num_examples': 43793, 'test/accuracy': 0.9860032794218176, 'test/loss': 0.04813336518969134, 'test/mean_average_precision': 0.26183146575940547, 'test/num_examples': 43793, 'score': 6234.726370334625, 'total_duration': 7249.787814855576, 'accumulated_submission_time': 6234.726370334625, 'accumulated_eval_time': 982.3029341697693, 'accumulated_logging_time': 0.6434557437896729}
  327. I0605 23:41:52.294861 140159630677760 logging_writer.py:48] [23818] accumulated_eval_time=982.303, accumulated_logging_time=0.643456, accumulated_submission_time=6234.73, global_step=23818, preemption_count=0, score=6234.73, test/accuracy=0.986003, test/loss=0.0481334, test/mean_average_precision=0.261831, test/num_examples=43793, total_duration=7249.79, train/accuracy=0.991256, train/loss=0.0284768, train/mean_average_precision=0.474097, validation/accuracy=0.986785, validation/loss=0.0453806, validation/mean_average_precision=0.27081, validation/num_examples=43793
  328. I0605 23:45:52.953383 140206541566592 spec.py:321] Evaluating on the training split.
  329. I0605 23:46:21.720630 140206541566592 spec.py:333] Evaluating on the validation split.
  330. I0605 23:46:24.095839 140206541566592 spec.py:349] Evaluating on the test split.
  331. I0605 23:46:26.534574 140206541566592 submission_runner.py:469] Time since start: 7524.04s, Step: 24766, {'train/accuracy': 0.9914003628966092, 'train/loss': 0.028109017267630043, 'train/mean_average_precision': 0.4738670667125567, 'validation/accuracy': 0.9868609174097411, 'validation/loss': 0.04486556182801592, 'validation/mean_average_precision': 0.2733954576253504, 'validation/num_examples': 43793, 'test/accuracy': 0.986067722208627, 'test/loss': 0.0475587503406412, 'test/mean_average_precision': 0.2659545676077682, 'test/num_examples': 43793, 'score': 6474.157246589661, 'total_duration': 7524.040155887604, 'accumulated_submission_time': 6474.157246589661, 'accumulated_eval_time': 1015.8840274810791, 'accumulated_logging_time': 0.6663665771484375}
  332. I0605 23:46:26.547271 140159513261824 logging_writer.py:48] [24766] accumulated_eval_time=1015.88, accumulated_logging_time=0.666367, accumulated_submission_time=6474.16, global_step=24766, preemption_count=0, score=6474.16, test/accuracy=0.986068, test/loss=0.0475588, test/mean_average_precision=0.265955, test/num_examples=43793, total_duration=7524.04, train/accuracy=0.9914, train/loss=0.028109, train/mean_average_precision=0.473867, validation/accuracy=0.986861, validation/loss=0.0448656, validation/mean_average_precision=0.273395, validation/num_examples=43793
  333. I0605 23:50:27.025815 140206541566592 spec.py:321] Evaluating on the training split.
  334. I0605 23:50:55.586430 140206541566592 spec.py:333] Evaluating on the validation split.
  335. I0605 23:50:57.999827 140206541566592 spec.py:349] Evaluating on the test split.
  336. I0605 23:51:00.437485 140206541566592 submission_runner.py:469] Time since start: 7797.94s, Step: 25715, {'train/accuracy': 0.9916488108508787, 'train/loss': 0.027627826667960143, 'train/mean_average_precision': 0.4829614983456372, 'validation/accuracy': 0.9869449471343428, 'validation/loss': 0.044460467896382826, 'validation/mean_average_precision': 0.2752513642193128, 'validation/num_examples': 43793, 'test/accuracy': 0.9861334285794915, 'test/loss': 0.04710141517728716, 'test/mean_average_precision': 0.2699399575535513, 'test/num_examples': 43793, 'score': 6713.432000637054, 'total_duration': 7797.943026542664, 'accumulated_submission_time': 6713.432000637054, 'accumulated_eval_time': 1049.295531988144, 'accumulated_logging_time': 0.6888301372528076}
  337. I0605 23:51:00.450123 140159630677760 logging_writer.py:48] [25715] accumulated_eval_time=1049.3, accumulated_logging_time=0.68883, accumulated_submission_time=6713.43, global_step=25715, preemption_count=0, score=6713.43, test/accuracy=0.986133, test/loss=0.0471014, test/mean_average_precision=0.26994, test/num_examples=43793, total_duration=7797.94, train/accuracy=0.991649, train/loss=0.0276278, train/mean_average_precision=0.482961, validation/accuracy=0.986945, validation/loss=0.0444605, validation/mean_average_precision=0.275251, validation/num_examples=43793
  338. I0605 23:55:01.043147 140206541566592 spec.py:321] Evaluating on the training split.
  339. I0605 23:55:29.876720 140206541566592 spec.py:333] Evaluating on the validation split.
  340. I0605 23:55:32.334053 140206541566592 spec.py:349] Evaluating on the test split.
  341. I0605 23:55:34.798547 140206541566592 submission_runner.py:469] Time since start: 8072.30s, Step: 26666, {'train/accuracy': 0.9917092048495199, 'train/loss': 0.027351078620699954, 'train/mean_average_precision': 0.49304085791378227, 'validation/accuracy': 0.9869904124925815, 'validation/loss': 0.04425504921726515, 'validation/mean_average_precision': 0.2765396781358895, 'validation/num_examples': 43793, 'test/accuracy': 0.9861725996851991, 'test/loss': 0.04690270013170758, 'test/mean_average_precision': 0.2742112554969621, 'test/num_examples': 43793, 'score': 6952.805119037628, 'total_duration': 8072.304066419601, 'accumulated_submission_time': 6952.805119037628, 'accumulated_eval_time': 1083.050758600235, 'accumulated_logging_time': 0.7113139629364014}
  342. I0605 23:55:34.811772 140159513261824 logging_writer.py:48] [26666] accumulated_eval_time=1083.05, accumulated_logging_time=0.711314, accumulated_submission_time=6952.81, global_step=26666, preemption_count=0, score=6952.81, test/accuracy=0.986173, test/loss=0.0469027, test/mean_average_precision=0.274211, test/num_examples=43793, total_duration=8072.3, train/accuracy=0.991709, train/loss=0.0273511, train/mean_average_precision=0.493041, validation/accuracy=0.98699, validation/loss=0.044255, validation/mean_average_precision=0.27654, validation/num_examples=43793
  343. I0605 23:59:35.379555 140206541566592 spec.py:321] Evaluating on the training split.
  344. I0606 00:00:04.488383 140206541566592 spec.py:333] Evaluating on the validation split.
  345. I0606 00:00:06.914477 140206541566592 spec.py:349] Evaluating on the test split.
  346. I0606 00:00:09.369379 140206541566592 submission_runner.py:469] Time since start: 8346.87s, Step: 27615, {'train/accuracy': 0.9919012873972195, 'train/loss': 0.026918822384694268, 'train/mean_average_precision': 0.4989590719944065, 'validation/accuracy': 0.987066323403212, 'validation/loss': 0.04398719495586207, 'validation/mean_average_precision': 0.2776168834168461, 'validation/num_examples': 43793, 'test/accuracy': 0.9862420968082288, 'test/loss': 0.04661332622075909, 'test/mean_average_precision': 0.27561151094208486, 'test/num_examples': 43793, 'score': 7192.131433010101, 'total_duration': 8346.874897241592, 'accumulated_submission_time': 7192.131433010101, 'accumulated_eval_time': 1117.040397644043, 'accumulated_logging_time': 0.7343757152557373}
  347. I0606 00:00:09.382542 140159630677760 logging_writer.py:48] [27615] accumulated_eval_time=1117.04, accumulated_logging_time=0.734376, accumulated_submission_time=7192.13, global_step=27615, preemption_count=0, score=7192.13, test/accuracy=0.986242, test/loss=0.0466133, test/mean_average_precision=0.275612, test/num_examples=43793, total_duration=8346.87, train/accuracy=0.991901, train/loss=0.0269188, train/mean_average_precision=0.498959, validation/accuracy=0.987066, validation/loss=0.0439872, validation/mean_average_precision=0.277617, validation/num_examples=43793
  348. I0606 00:04:10.056925 140206541566592 spec.py:321] Evaluating on the training split.
  349. I0606 00:04:39.060932 140206541566592 spec.py:333] Evaluating on the validation split.
  350. I0606 00:04:41.546401 140206541566592 spec.py:349] Evaluating on the test split.
  351. I0606 00:04:44.026495 140206541566592 submission_runner.py:469] Time since start: 8621.53s, Step: 28566, {'train/accuracy': 0.9919062207620605, 'train/loss': 0.02682081464579075, 'train/mean_average_precision': 0.5092326884244134, 'validation/accuracy': 0.9870025907135382, 'validation/loss': 0.04437587048908547, 'validation/mean_average_precision': 0.27600955737157384, 'validation/num_examples': 43793, 'test/accuracy': 0.986216403932442, 'test/loss': 0.04709180996411842, 'test/mean_average_precision': 0.27152752866874186, 'test/num_examples': 43793, 'score': 7431.579320430756, 'total_duration': 8621.53208398819, 'accumulated_submission_time': 7431.579320430756, 'accumulated_eval_time': 1151.0098872184753, 'accumulated_logging_time': 0.7585985660552979}
  352. I0606 00:04:44.039678 140159513261824 logging_writer.py:48] [28566] accumulated_eval_time=1151.01, accumulated_logging_time=0.758599, accumulated_submission_time=7431.58, global_step=28566, preemption_count=0, score=7431.58, test/accuracy=0.986216, test/loss=0.0470918, test/mean_average_precision=0.271528, test/num_examples=43793, total_duration=8621.53, train/accuracy=0.991906, train/loss=0.0268208, train/mean_average_precision=0.509233, validation/accuracy=0.987003, validation/loss=0.0443759, validation/mean_average_precision=0.27601, validation/num_examples=43793
  353. I0606 00:08:44.604856 140206541566592 spec.py:321] Evaluating on the training split.
  354. I0606 00:09:13.434299 140206541566592 spec.py:333] Evaluating on the validation split.
  355. I0606 00:09:15.818977 140206541566592 spec.py:349] Evaluating on the test split.
  356. I0606 00:09:18.295045 140206541566592 submission_runner.py:469] Time since start: 8895.80s, Step: 29516, {'train/accuracy': 0.9919414151798582, 'train/loss': 0.026532870695000238, 'train/mean_average_precision': 0.5193584712177852, 'validation/accuracy': 0.9870553630043509, 'validation/loss': 0.04413085990621146, 'validation/mean_average_precision': 0.27661220148412946, 'validation/num_examples': 43793, 'test/accuracy': 0.9862572598168898, 'test/loss': 0.046783432933591496, 'test/mean_average_precision': 0.2762866142336493, 'test/num_examples': 43793, 'score': 7670.951828718185, 'total_duration': 8895.800637483597, 'accumulated_submission_time': 7670.951828718185, 'accumulated_eval_time': 1184.7000379562378, 'accumulated_logging_time': 0.7817206382751465}
  357. I0606 00:09:18.308383 140159630677760 logging_writer.py:48] [29516] accumulated_eval_time=1184.7, accumulated_logging_time=0.781721, accumulated_submission_time=7670.95, global_step=29516, preemption_count=0, score=7670.95, test/accuracy=0.986257, test/loss=0.0467834, test/mean_average_precision=0.276287, test/num_examples=43793, total_duration=8895.8, train/accuracy=0.991941, train/loss=0.0265329, train/mean_average_precision=0.519358, validation/accuracy=0.987055, validation/loss=0.0441309, validation/mean_average_precision=0.276612, validation/num_examples=43793
  358. I0606 00:13:18.997440 140206541566592 spec.py:321] Evaluating on the training split.
  359. I0606 00:13:47.834757 140206541566592 spec.py:333] Evaluating on the validation split.
  360. I0606 00:13:50.251464 140206541566592 spec.py:349] Evaluating on the test split.
  361. I0606 00:13:52.702784 140206541566592 submission_runner.py:469] Time since start: 9170.21s, Step: 30468, {'train/accuracy': 0.9920943194673836, 'train/loss': 0.026274859228352602, 'train/mean_average_precision': 0.5109795240621366, 'validation/accuracy': 0.9871207194568189, 'validation/loss': 0.0439818733270169, 'validation/mean_average_precision': 0.278379634925744, 'validation/num_examples': 43793, 'test/accuracy': 0.9863061184003531, 'test/loss': 0.0465948397227865, 'test/mean_average_precision': 0.2780755648101628, 'test/num_examples': 43793, 'score': 7910.452227830887, 'total_duration': 9170.208403348923, 'accumulated_submission_time': 7910.452227830887, 'accumulated_eval_time': 1218.4053659439087, 'accumulated_logging_time': 0.8051965236663818}
  362. I0606 00:13:52.715734 140159513261824 logging_writer.py:48] [30468] accumulated_eval_time=1218.41, accumulated_logging_time=0.805197, accumulated_submission_time=7910.45, global_step=30468, preemption_count=0, score=7910.45, test/accuracy=0.986306, test/loss=0.0465948, test/mean_average_precision=0.278076, test/num_examples=43793, total_duration=9170.21, train/accuracy=0.992094, train/loss=0.0262749, train/mean_average_precision=0.51098, validation/accuracy=0.987121, validation/loss=0.0439819, validation/mean_average_precision=0.27838, validation/num_examples=43793
  363. I0606 00:17:53.309845 140206541566592 spec.py:321] Evaluating on the training split.
  364. I0606 00:18:21.881352 140206541566592 spec.py:333] Evaluating on the validation split.
  365. I0606 00:18:24.613676 140206541566592 spec.py:349] Evaluating on the test split.
  366. I0606 00:18:27.066525 140206541566592 submission_runner.py:469] Time since start: 9444.57s, Step: 31417, {'train/accuracy': 0.9923217284414019, 'train/loss': 0.025744564290858925, 'train/mean_average_precision': 0.5161818174180625, 'validation/accuracy': 0.9871393927289526, 'validation/loss': 0.04391963754366907, 'validation/mean_average_precision': 0.2803604170745514, 'validation/num_examples': 43793, 'test/accuracy': 0.9863318112761399, 'test/loss': 0.0465312821029324, 'test/mean_average_precision': 0.279317550777637, 'test/num_examples': 43793, 'score': 8149.863886833191, 'total_duration': 9444.572100400925, 'accumulated_submission_time': 8149.863886833191, 'accumulated_eval_time': 1252.1619246006012, 'accumulated_logging_time': 0.8281047344207764}
  367. I0606 00:18:27.080496 140159630677760 logging_writer.py:48] [31417] accumulated_eval_time=1252.16, accumulated_logging_time=0.828105, accumulated_submission_time=8149.86, global_step=31417, preemption_count=0, score=8149.86, test/accuracy=0.986332, test/loss=0.0465313, test/mean_average_precision=0.279318, test/num_examples=43793, total_duration=9444.57, train/accuracy=0.992322, train/loss=0.0257446, train/mean_average_precision=0.516182, validation/accuracy=0.987139, validation/loss=0.0439196, validation/mean_average_precision=0.28036, validation/num_examples=43793
  368. I0606 00:22:27.713407 140206541566592 spec.py:321] Evaluating on the training split.
  369. I0606 00:22:56.284168 140206541566592 spec.py:333] Evaluating on the validation split.
  370. I0606 00:22:58.680942 140206541566592 spec.py:349] Evaluating on the test split.
  371. I0606 00:23:01.125143 140206541566592 submission_runner.py:469] Time since start: 9718.63s, Step: 32369, {'train/accuracy': 0.9925041523335721, 'train/loss': 0.02526251579264309, 'train/mean_average_precision': 0.5290929132372566, 'validation/accuracy': 0.9871617194673733, 'validation/loss': 0.043908049205289894, 'validation/mean_average_precision': 0.27999327508594885, 'validation/num_examples': 43793, 'test/accuracy': 0.9863124363206286, 'test/loss': 0.04650394064482379, 'test/mean_average_precision': 0.2789805393009878, 'test/num_examples': 43793, 'score': 8389.307892799377, 'total_duration': 9718.630764961243, 'accumulated_submission_time': 8389.307892799377, 'accumulated_eval_time': 1285.5735428333282, 'accumulated_logging_time': 0.8523833751678467}
  372. I0606 00:23:01.138399 140159513261824 logging_writer.py:48] [32369] accumulated_eval_time=1285.57, accumulated_logging_time=0.852383, accumulated_submission_time=8389.31, global_step=32369, preemption_count=0, score=8389.31, test/accuracy=0.986312, test/loss=0.0465039, test/mean_average_precision=0.278981, test/num_examples=43793, total_duration=9718.63, train/accuracy=0.992504, train/loss=0.0252625, train/mean_average_precision=0.529093, validation/accuracy=0.987162, validation/loss=0.043908, validation/mean_average_precision=0.279993, validation/num_examples=43793
  373. I0606 00:27:01.693059 140206541566592 spec.py:321] Evaluating on the training split.
  374. I0606 00:27:30.473422 140206541566592 spec.py:333] Evaluating on the validation split.
  375. I0606 00:27:32.900552 140206541566592 spec.py:349] Evaluating on the test split.
  376. I0606 00:27:35.346549 140206541566592 submission_runner.py:469] Time since start: 9992.85s, Step: 33323, {'train/accuracy': 0.992682824267133, 'train/loss': 0.02471512092360922, 'train/mean_average_precision': 0.5532402035936022, 'validation/accuracy': 0.9871523828313065, 'validation/loss': 0.04392542854144695, 'validation/mean_average_precision': 0.28022212757069037, 'validation/num_examples': 43793, 'test/accuracy': 0.9863065395950381, 'test/loss': 0.04651004796775671, 'test/mean_average_precision': 0.278549655915466, 'test/num_examples': 43793, 'score': 8628.653096437454, 'total_duration': 9992.852115869522, 'accumulated_submission_time': 8628.653096437454, 'accumulated_eval_time': 1319.2269492149353, 'accumulated_logging_time': 0.8755114078521729}
  377. I0606 00:27:35.359569 140159630677760 logging_writer.py:48] [33323] accumulated_eval_time=1319.23, accumulated_logging_time=0.875511, accumulated_submission_time=8628.65, global_step=33323, preemption_count=0, score=8628.65, test/accuracy=0.986307, test/loss=0.04651, test/mean_average_precision=0.27855, test/num_examples=43793, total_duration=9992.85, train/accuracy=0.992683, train/loss=0.0247151, train/mean_average_precision=0.55324, validation/accuracy=0.987152, validation/loss=0.0439254, validation/mean_average_precision=0.280222, validation/num_examples=43793
  378. I0606 00:31:35.901987 140206541566592 spec.py:321] Evaluating on the training split.
  379. I0606 00:32:04.821742 140206541566592 spec.py:333] Evaluating on the validation split.
  380. I0606 00:32:07.219993 140206541566592 spec.py:349] Evaluating on the test split.
  381. I0606 00:32:09.740759 140206541566592 submission_runner.py:469] Time since start: 10267.25s, Step: 34281, {'train/accuracy': 0.9927955406077692, 'train/loss': 0.02447629044787512, 'train/mean_average_precision': 0.5529152665906756, 'validation/accuracy': 0.9871710561034401, 'validation/loss': 0.04394914118678387, 'validation/mean_average_precision': 0.2813206899829609, 'validation/num_examples': 43793, 'test/accuracy': 0.9863280205239746, 'test/loss': 0.04652034420345556, 'test/mean_average_precision': 0.27793161007741, 'test/num_examples': 43793, 'score': 8867.994435071945, 'total_duration': 10267.246317863464, 'accumulated_submission_time': 8867.994435071945, 'accumulated_eval_time': 1353.0657291412354, 'accumulated_logging_time': 0.9003341197967529}
  382. I0606 00:32:09.754453 140159513261824 logging_writer.py:48] [34281] accumulated_eval_time=1353.07, accumulated_logging_time=0.900334, accumulated_submission_time=8867.99, global_step=34281, preemption_count=0, score=8867.99, test/accuracy=0.986328, test/loss=0.0465203, test/mean_average_precision=0.277932, test/num_examples=43793, total_duration=10267.2, train/accuracy=0.992796, train/loss=0.0244763, train/mean_average_precision=0.552915, validation/accuracy=0.987171, validation/loss=0.0439491, validation/mean_average_precision=0.281321, validation/num_examples=43793
  383. I0606 00:32:10.252077 140159630677760 logging_writer.py:48] [34281] global_step=34281, preemption_count=0, score=8867.99
  384. I0606 00:32:10.418975 140206541566592 submission_runner.py:750] Final ogbg score: 8867.994435071945
  385.  
Tags: pytorch
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!