Using gpu device 3: Tesla K40m (CNMeM is disabled, cuDNN 5004)
/home/anton-fr/.local/lib/python3.5/site-packages/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module.
  "downsample module has been moved to the theano.tensor.signal.pool module.")
model loaded, there are 51 sets of params
compiling optimizers...
finished, took 48.406128 sec
optimizers in use: adam_0.01, sgd_0.005, sgd_0.0025, sgd_0.00125, sgd_0.000625, sgd_0.000313, sgd_0.000156, sgd_0.000078, sgd_0.000039, sgd_0.00002
loading corpus from train.sample
init complete

running test with initial params...
Traceback (most recent call last):
  File "theano/scan_module/scan_perform.pyx", line 397, in theano.scan_module.scan_perform.perform (/place/home/anton-fr/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/scan_perform/mod.cpp:4193)
ValueError: GpuReshape: cannot reshape input of shape (1, 400, 600) to shape (1, 100, 10, 600).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/scan_module/scan_op.py", line 951, in rval
    r = p(n, [x[0] for x in i], o)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/scan_module/scan_op.py", line 940, in <lambda>
    self, node)
  File "theano/scan_module/scan_perform.pyx", line 405, in theano.scan_module.scan_perform.perform (/place/home/anton-fr/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/scan_perform/mod.cpp:4316)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "theano/scan_module/scan_perform.pyx", line 397, in theano.scan_module.scan_perform.perform (/place/home/anton-fr/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/scan_perform/mod.cpp:4193)
ValueError: GpuReshape: cannot reshape input of shape (1, 400, 600) to shape (1, 100, 10, 600).
Apply node that caused the error: GpuReshape{4}(GpuBatchedDot{stream_threshold=650}.0, TensorConstant{[  1 100  10 600]})
Toposort index: 60
Inputs types: [CudaNdarrayType(float32, (True, False, False)), TensorType(int64, vector)]
Inputs shapes: [(1, 400, 600), (4,)]
Inputs strides: [(0, 600, 1), (8,)]
Inputs values: ['not shown', array([  1, 100,  10, 600])]
Outputs clients: [[GpuElemwise{ScalarSigmoid}[(0, 0)](GpuReshape{4}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/anton-fr/nnet/nnet/run/srl/run.py", line 310, in <module>
    SRLRunner().run()
  File "/opt/anton-fr/nnet/nnet/run/runner.py", line 160, in run
    dbg_print_rate=a.dbg_print_rate
  File "/opt/anton-fr/nnet/nnet/training.py", line 15, in train
    tester.run(model)
  File "/opt/anton-fr/nnet/nnet/testing.py", line 151, in run
    loss, errors, errors_w = self.compute_error(model)
  File "/opt/anton-fr/nnet/nnet/testing.py", line 173, in compute_error
    loss, e, e_w = self.error_computer.compute(model, batch)
  File "/opt/anton-fr/nnet/nnet/run/srl/run.py", line 137, in compute
    predictions = model.predict(*model_input[:-1])
  File "/opt/anton-fr/nnet/nnet/models/srl.py", line 191, in predict
    return self.compute_probs(*args)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/compile/function_module.py", line 871, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/scan_module/scan_op.py", line 951, in rval
    r = p(n, [x[0] for x in i], o)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/scan_module/scan_op.py", line 940, in <lambda>
    self, node)
  File "theano/scan_module/scan_perform.pyx", line 405, in theano.scan_module.scan_perform.perform (/place/home/anton-fr/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/scan_perform/mod.cpp:4316)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/anton-fr/.local/lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "theano/scan_module/scan_perform.pyx", line 397, in theano.scan_module.scan_perform.perform (/place/home/anton-fr/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/scan_perform/mod.cpp:4193)
ValueError: GpuReshape: cannot reshape input of shape (1, 400, 600) to shape (1, 100, 10, 600).
Apply node that caused the error: GpuReshape{4}(GpuBatchedDot{stream_threshold=650}.0, TensorConstant{[  1 100  10 600]})
Toposort index: 60
Inputs types: [CudaNdarrayType(float32, (True, False, False)), TensorType(int64, vector)]
Inputs shapes: [(1, 400, 600), (4,)]
Inputs strides: [(0, 600, 1), (8,)]
Inputs values: ['not shown', array([  1, 100,  10, 600])]
Outputs clients: [[GpuElemwise{ScalarSigmoid}[(0, 0)](GpuReshape{4}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Apply node that caused the error: forall_inplace,gpu,scan_fn}(Elemwise{Composite{minimum(minimum(minimum(i0, i1), i2), i3)}}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuSubtensor{int64:int64:int8}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuJoin.0, GpuReshape{3}.0, GpuReshape{3}.0, GpuReshape{3}.0, GpuReshape{3}.0)
Toposort index: 256
Inputs types: [TensorType(int64, scalar), CudaNdarrayType(float32, 3D), CudaNdarrayType(float32, (False, False, True)), CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 3D), CudaNdarrayType(float32, 3D), CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, (True, False, False)), CudaNdarrayType(float32, (True, False, False)), CudaNdarrayType(float32, (True, False, False)), CudaNdarrayType(float32, (True, False, False))]
Inputs shapes: [(), (38, 40, 2400), (38, 40, 1), (38, 40, 1, 200), (38, 40, 1, 200), (21, 40, 600), (38, 40, 600), (600, 2400), (1, 600, 600), (1, 600, 1800), (1, 600, 600), (1, 600, 1800)]
Inputs strides: [(), (96000, 2400, 1), (1, 38, 0), (200, 40000, 0, 1), (200, 40000, 0, 1), (24000, 600, 1), (24000, 600, 1), (2400, 1), (0, 600, 1), (0, 1800, 1), (0, 600, 1), (0, 1800, 1)]
Inputs values: [array(38), 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown']
Outputs clients: [[], [GpuSubtensor{int64:int64:int8}(forall_inplace,gpu,scan_fn}.1, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
<TensorType(float32, matrix)>
<TensorType(float32, matrix)>
<TensorType(float32, matrix)>
<TensorType(float32, matrix)>
<TensorType(float32, matrix)>
<TensorType(float32, matrix)>

 ERROR  Deleting  build/framenet_batch_100__1_300_600_500_500_500_1_/out/_best.bin , command failed 2016-09-27 16:36:17

 ERROR  Build error: command failed