Untitled

/home/michalmartyniak/repos/sim2real/sim2real/utils/os.py:180: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  LOGGER.warn("nvidia-smi not present")
2020-02-28 13:13:34,712 MPI-0 11048 main WARNING  sim2real.utils.os nvidia-smi not present
2020-02-28 13:13:45,109 MPI-0 11048 main INFO     exp.ppo2.ppo2 Collecting trajectories for update 1
2020-02-28 13:13:45,116 MPI-0 11048 unroll-0 DEBUG    sim2real.carla.env [127.0.0.1:42105|vehicle-265] reset
2020-02-28 13:13:45,418 MPI-0 11048 unroll-0 DEBUG    sim2real.carla.server Set: WeatherParameters(cloudyness=0.000000, precipitation=0.000000, precipitation_deposits=0.000000, wind_intensity=0.350000, sun_azimuth_angle=0.000000, sun_altitude_angle=75.000000) on server 127.0.0.1:42105
2020-02-28 13:13:46,015 MPI-0 11048 unroll-0 DEBUG    sim2real.storage using storage: /mnt/ml-team/rl/carla/logs
2020-02-28 13:13:46,017 MPI-0 11048 unroll-0 INFO     sim2real.birdview Loading cache from /mnt/ml-team/rl/carla/logs/Town01__px_per_meter=4__opendrive_hash=217c7b9106f1a6abdf8c23f38b06da618f570858.npy
2020-02-28 13:13:46,777 MPI-0 11048 unroll-0 INFO     sim2real.birdview Loaded static layers from cache file: /mnt/ml-team/rl/carla/logs/Town01__px_per_meter=4__opendrive_hash=217c7b9106f1a6abdf8c23f38b06da618f570858.npy
2020-02-28 13:13:46,842 MPI-0 11048 unroll-0 DEBUG    sim2real.carla.env [127.0.0.1:42105|vehicle-269] Speed-up for 10 steps
2020-02-28 13:13:49,020 MPI-0 11048 unroll-0 ERROR    sim2real.runner Error occurred in unroll thread
Traceback (most recent call last):
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable model_1/car_metrics_embedded_01/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/model_1/car_metrics_embedded_01/kernel/N10tensorflow3VarE does not exist.
     [[{{node model_1/car_metrics_embedded_01/MatMul/ReadVariableOp}} = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/car_metrics_embedded_01/kernel)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/michalmartyniak/repos/sim2real/sim2real/runner.py", line 100, in run
    policy_result = self._policy_runner.step([ob])
  File "/home/michalmartyniak/repos/sim2real/sim2real/carla_policy_runner.py", line 104, in step
    return self._any_step(obs, self._sampled_action_steering, self._sampled_action_neglogprobs)
  File "/home/michalmartyniak/repos/sim2real/sim2real/carla_policy_runner.py", line 119, in _any_step
    ], feed_dict)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable model_1/car_metrics_embedded_01/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/model_1/car_metrics_embedded_01/kernel/N10tensorflow3VarE does not exist.
     [[node model_1/car_metrics_embedded_01/MatMul/ReadVariableOp (defined at /home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/policy.py:85)  = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/car_metrics_embedded_01/kernel)]]

Caused by op 'model_1/car_metrics_embedded_01/MatMul/ReadVariableOp', defined at:
  File "/home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/train.py", line 351, in <module>
    main()
  File "/home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/train.py", line 332, in main
    tf_sess=tf_sess,
  File "/home/michalmartyniak/repos/sim2real/exp/ppo2/ppo2.py", line 87, in learn
    policy_runner = GraphPolicyRunner(policy_graph=policy, session=tf_sess)
  File "/home/michalmartyniak/repos/sim2real/sim2real/carla_policy_runner.py", line 59, in __init__
    self._policy_output: CarlaPolicyOutput = policy_graph.forward(self._policy_input)
  File "/home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/policy.py", line 85, in forward
    _ = Dense(units=16, activation=leaky_relu, name="car_metrics_embedded_01")(input.car_metrics)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 757, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py", line 970, in call
    outputs = gen_math_ops.mat_mul(inputs, self.kernel)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4560, in mat_mul
    name=name)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1146, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1258, in _dense_var_to_tensor
    return var._dense_var_to_tensor(dtype=dtype, name=name, as_ref=as_ref)  # pylint: disable=protected-access
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1213, in _dense_var_to_tensor
    return self.value()
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 644, in value
    return self._read_variable_op()
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 727, in _read_variable_op
    self._dtype)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 508, in read_variable_op
    "ReadVariableOp", resource=resource, dtype=dtype, name=name)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

FailedPreconditionError (see above for traceback): Error while reading resource variable model_1/car_metrics_embedded_01/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/model_1/car_metrics_embedded_01/kernel/N10tensorflow3VarE does not exist.
     [[node model_1/car_metrics_embedded_01/MatMul/ReadVariableOp (defined at /home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/policy.py:85)  = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/car_metrics_embedded_01/kernel)]]

Exception in thread unroll-0:
Traceback (most recent call last):
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable model_1/car_metrics_embedded_01/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/model_1/car_metrics_embedded_01/kernel/N10tensorflow3VarE does not exist.
     [[{{node model_1/car_metrics_embedded_01/MatMul/ReadVariableOp}} = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/car_metrics_embedded_01/kernel)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/michalmartyniak/repos/sim2real/sim2real/runner.py", line 100, in run
    policy_result = self._policy_runner.step([ob])
  File "/home/michalmartyniak/repos/sim2real/sim2real/carla_policy_runner.py", line 104, in step
    return self._any_step(obs, self._sampled_action_steering, self._sampled_action_neglogprobs)
  File "/home/michalmartyniak/repos/sim2real/sim2real/carla_policy_runner.py", line 119, in _any_step
    ], feed_dict)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable model_1/car_metrics_embedded_01/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/model_1/car_metrics_embedded_01/kernel/N10tensorflow3VarE does not exist.
     [[node model_1/car_metrics_embedded_01/MatMul/ReadVariableOp (defined at /home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/policy.py:85)  = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/car_metrics_embedded_01/kernel)]]

Caused by op 'model_1/car_metrics_embedded_01/MatMul/ReadVariableOp', defined at:
  File "/home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/train.py", line 351, in <module>
    main()
  File "/home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/train.py", line 332, in main
    tf_sess=tf_sess,
  File "/home/michalmartyniak/repos/sim2real/exp/ppo2/ppo2.py", line 87, in learn
    policy_runner = GraphPolicyRunner(policy_graph=policy, session=tf_sess)
  File "/home/michalmartyniak/repos/sim2real/sim2real/carla_policy_runner.py", line 59, in __init__
    self._policy_output: CarlaPolicyOutput = policy_graph.forward(self._policy_input)
  File "/home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/policy.py", line 85, in forward
    _ = Dense(units=16, activation=leaky_relu, name="car_metrics_embedded_01")(input.car_metrics)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 757, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py", line 970, in call
    outputs = gen_math_ops.mat_mul(inputs, self.kernel)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4560, in mat_mul
    name=name)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1146, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1258, in _dense_var_to_tensor
    return var._dense_var_to_tensor(dtype=dtype, name=name, as_ref=as_ref)  # pylint: disable=protected-access
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1213, in _dense_var_to_tensor
    return self.value()
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 644, in value
    return self._read_variable_op()
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 727, in _read_variable_op
    self._dtype)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 508, in read_variable_op
    "ReadVariableOp", resource=resource, dtype=dtype, name=name)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/michalmartyniak/miniconda3/envs/sim2real-env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

FailedPreconditionError (see above for traceback): Error while reading resource variable model_1/car_metrics_embedded_01/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/model_1/car_metrics_embedded_01/kernel/N10tensorflow3VarE does not exist.
     [[node model_1/car_metrics_embedded_01/MatMul/ReadVariableOp (defined at /home/michalmartyniak/repos/sim2real/sim2real/rl/privileged_agent/policy.py:85)  = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](model_1/car_metrics_embedded_01/kernel)]]