Loading models....
 50%|████████████████████▌                    | 8/16 [00:24<00:24,  3.12s/it][2019-04-19 09:50:15,523] ERROR in app: Exception on /api/detect_entities [POST]
Traceback (most recent call last):
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4096,4096] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
	 [[{{node save/RestoreV2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/wilq/.local/lib/python3.5/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/pocs/overwritedev/py/app.py", line 37, in detect_entities
    feature_models = load_all_models(models_path, weights_path, features_list, batch_size)
  File "/opt/pocs/overwritedev/py/tools.py", line 68, in load_all_models
    saver.restore(sess, model_data)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4096,4096] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
	 [[node save/RestoreV2 (defined at /opt/pocs/overwritedev/py/tools.py:66) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Caused by op 'save/RestoreV2', defined at:
  File "/usr/local/bin/gunicorn", line 11, in <module>
    sys.exit(run())
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 61, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/app/base.py", line 223, in run
    super(Application, self).run()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 203, in run
    self.manage_workers()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 545, in manage_workers
    self.spawn_workers()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 616, in spawn_workers
    self.spawn_worker()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.run()
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 124, in run
    self.run_for_one(timeout)
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 68, in run_for_one
    self.accept(listener)
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 30, in accept
    self.handle(listener, client, addr)
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/wilq/.local/lib/python3.5/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/wilq/.local/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/pocs/overwritedev/py/app.py", line 37, in detect_entities
    feature_models = load_all_models(models_path, weights_path, features_list, batch_size)
  File "/opt/pocs/overwritedev/py/tools.py", line 66, in load_all_models
    saver = tf.train.Saver()
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 832, in __init__
    self.build()
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 844, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 881, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 513, in _build_internal
    restore_sequentially, reshape)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps
    restore_sequentially)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2
    name=name)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/home/wilq/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4096,4096] and type float on /job:localhost/replica:0/task:0/device:CPU:0 by allocator cpu
	 [[node save/RestoreV2 (defined at /opt/pocs/overwritedev/py/tools.py:66) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


[2019-04-19 09:50:25 +0000] [29788] [INFO] Handling signal: winch
[2019-04-19 09:50:25 +0000] [29788] [INFO] Handling signal: winch
[2019-04-19 09:50:26 +0000] [29788] [INFO] Handling signal: winch
[2019-04-19 09:50:26 +0000] [29788] [INFO] Handling signal: winch
[2019-04-19 09:50:26 +0000] [29788] [INFO] Handling signal: winch
[2019-04-19 09:50:26 +0000] [29788] [INFO] Handling signal: winch
[2019-04-19 09:50:26 +0000] [29788] [INFO] Handling signal: winch
[2019-04-19 09:50:26 +0000] [29788] [INFO] Handling signal: winch