message: "================================================================================
[New LWP 234]
[New LWP 248]
[New LWP 242]
[New LWP 245]
[New LWP 243]
[New LWP 244]
[New LWP 249]
[New LWP 247]
[New LWP 246]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python -m trainer.test --job-dir gs://my_test_bucket_keras/s_27_100630'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fe9352c44d8 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
--------------------------------------------------------------------------------
bt full:
#0  0x00007fe9352c44d8 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00000000004b7d85 in ?? ()
No symbol table info available.
#2  0x000000000049bb0c in PyDict_SetItem ()
No symbol table info available.
#3  0x000000000051a513 in _PyModule_Clear ()
No symbol table info available.
#4  0x0000000000519c35 in PyImport_Cleanup ()
No symbol table info available.
#5  0x0000000000513c24 in Py_Finalize ()
No symbol table info available.
#6  0x0000000000498099 in Py_Main ()
No symbol table info available.
#7  0x00007fe93526cb45 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#8  0x0000000000497b8b in _start ()
No symbol table info available.
--------------------------------------------------------------------------------
thread apply all bt:

Thread 9 (Thread 0x7fe917ee2700 (LWP 246)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 8 (Thread 0x7fe9176e1700 (LWP 247)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7fe9166df700 (LWP 249)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 6 (Thread 0x7fe931700700 (LWP 244)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 5 (Thread 0x7fe930eff700 (LWP 243)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 4 (Thread 0x7fe9186e3700 (LWP 245)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 3 (Thread 0x7fe9306fe700 (LWP 242)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 2 (Thread 0x7fe916ee0700 (LWP 248)):
#0  0x00007fe935f2504f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe92502dc7c in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007fe927879877 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tensorflow::thread::EigenEnvironment::Task*) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#3  0x00007fe92787a0f4 in std::_Function_handler<void (), Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::NonBlockingThreadPoolTempl(int, tensorflow::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#4  0x00007fe9278791f0 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so
#5  0x00007fe925031970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007fe935f21064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007fe93533362d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Thread 1 (Thread 0x7fe936349700 (LWP 234)):
#0  0x00007fe9352c44d8 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00000000004b7d85 in ?? ()
#2  0x000000000049bb0c in PyDict_SetItem ()
#3  0x000000000051a513 in _PyModule_Clear ()
#4  0x0000000000519c35 in PyImport_Cleanup ()
#5  0x0000000000513c24 in Py_Finalize ()
#6  0x0000000000498099 in Py_Main ()
#7  0x00007fe93526cb45 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x0000000000497b8b in _start ()

Called "gsutil -q mb -p linen-nebula-165905-64c26 gs://linen-nebula-165905-64c26-coredumps". Stdout was <>

Called "gsutil -q mb -p linen-nebula-165905-64c26 gs://linen-nebula-165905-64c26-coredumps". Stderr was <ServiceException: 409 Bucket linen-nebula-165905-64c26-coredumps already exists. >

Module raised an exception for failing to call a subprocess Command '['python', '-m', u'trainer.test', '--job-dir', u'gs://my_test_bucket_keras/s_27_100630']' returned non-zero exit status -11.

The replica master 0 exited with a non-zero status of 245. Termination reason: Error.