API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Apr 23rd, 2018
78
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 301.58 KB | None | 0 0
raw download clone embed print report
  1. /usr/bin/python3.5 /home/denis/beta1/main.py
  2. WARNING 16:17:23.313 22842 cleaner.py ------------ HACK!!! read file /dev/video0 fps directly
  3. INFO 16:17:23.486 22842 database_service.py ------------ Register collections: {'Camera': 'cameras', 'TelegramChat': 'telegram', 'TreadedVideo': 'treaded_videos', 'PresenceInfo': 'presence_info', 'Room': 'rooms', 'Human': 'humans', 'HumanStateInfo': 'human_state_info', 'Anomaly': 'anomaly'}
  4. INFO 16:17:23.491 22842 aiservice.py ------------ Init shared image segment of 200 images
  5. DEBUG 16:17:36.653 22842 selector_events.py ------------ Using selector: EpollSelector
  6. INFO 16:17:36.655 22842 command.py ------------ Loading module core.video_process_worker ...
  7. INFO 16:17:36.655 22864 command.py ------------ Loading module core.tracker ...
  8. INFO 16:17:36.657 22842 command.py ------------ Loading module core.video_process_worker ...OK
  9. INFO 16:17:36.659 22865 command.py ------------ Loading module core.cnn ...
  10. INFO 16:17:36.661 22842 aiservice.py ------------ Starting...
  11. INFO 16:17:36.662 22866 command.py ------------ Loading module core.show_video ...
  12. INFO 16:17:36.663 22842 aiservice.py ------------ Starting...OK
  13. DEBUG 16:17:36.663 22842 shared_array.py ------------ Shared memory free 200, used 0
  14. INFO 16:17:36.664 22842 aiservice.py ce3625921a85 Put batch for camera test request to queue, spend 0.0009264945983886719 sec
  15. INFO 16:17:36.664 22842 aiservice.py ------------ Start processing batches
  16. INFO 16:17:36.665 22842 aiservice.py ------------ Start processing batches
  17. INFO 16:17:36.665 22842 aiservice.py ------------ Start processing batches
  18. INFO 16:17:36.665 22867 aiservice.py ------------ Dispatch images...
  19. INFO 16:17:36.666 22867 aiservice.py ce3625921a85 Start collecting batch for camera test, spend 0.0022940635681152344 sec
  20. INFO 16:17:36.666 22867 batch_reader.py ------------ Create stream to test, args {'started': 1524489456.6661441, 'development': True, 'camera_name': 'test', 'qid': 'ce3625921a85', 'duration': 6, 'restart': True, 'fps': 4}
  21. DEBUG 16:17:36.866 22842 shared_array.py ------------ Shared memory free 176, used 24
  22. INFO 16:17:36.866 22842 aiservice.py 4467122a4283 Put batch for camera test request to queue, spend 0.0007071495056152344 sec
  23. [TensorRT] INFO: Added linear block of size 75497472
  24. [TensorRT] INFO: Added linear block of size 75497472
  25. [TensorRT] INFO: Added linear block of size 9437184
  26. [TensorRT] INFO: Added linear block of size 2359296
  27. [TensorRT] INFO: Added linear block of size 2359296
  28. DEBUG 16:17:37.276 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489457270
  29. DEBUG 16:17:37.505 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489457502
  30. DEBUG 16:17:37.738 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489457734
  31. /usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  32. from ._conv import register_converters as _register_converters
  33. /usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  34. from ._conv import register_converters as _register_converters
  35. [TensorRT] INFO: Verifying engine construction was successful
  36. [TensorRT] INFO: Allocating GPU buffers
  37. INFO 16:17:37.931 22865 command.py ------------ Loading module core.cnn ...OK
  38. DEBUG 16:17:37.969 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489457966
  39. Using TensorFlow backend.
  40. Using TensorFlow backend.
  41. DEBUG 16:17:38.078 22866 geos.py ------------ Found GEOS DLL: <CDLL '/usr/local/lib/python3.5/dist-packages/shapely/.libs/libgeos_c-bd8d3f16.so.1.10.2', handle 4e844f0 at 0x7f99b7045a20>, using it.
  42. DEBUG 16:17:38.088 22866 geos.py ------------ Trying `CDLL(libc.so.6)`
  43. DEBUG 16:17:38.088 22866 geos.py ------------ Library path: 'libc.so.6'
  44. DEBUG 16:17:38.089 22866 geos.py ------------ DLL: <CDLL 'libc.so.6', handle 7f9b7fbb94e8 at 0x7f99b7045e48>
  45. DEBUG 16:17:38.091 22864 geos.py ------------ Found GEOS DLL: <CDLL '/usr/local/lib/python3.5/dist-packages/shapely/.libs/libgeos_c-bd8d3f16.so.1.10.2', handle 4e80ac0 at 0x7f99b7030358>, using it.
  46. DEBUG 16:17:38.101 22864 geos.py ------------ Trying `CDLL(libc.so.6)`
  47. DEBUG 16:17:38.102 22864 geos.py ------------ Library path: 'libc.so.6'
  48. DEBUG 16:17:38.102 22864 geos.py ------------ DLL: <CDLL 'libc.so.6', handle 7f9b7fbb94e8 at 0x7f99b7030780>
  49. INFO 16:17:38.103 22866 command.py ------------ Loading module core.show_video ...OK
  50. DEBUG 16:17:38.201 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489458198
  51. 2018-04-23 16:17:38.239860: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
  52. 2018-04-23 16:17:38.324507: W tensorflow/stream_executor/cuda/cuda_driver.cc:527] A non-primary context 0x1ccf060 for device 0 exists before initializing the StreamExecutor. The primary context is now 0x1ca12e0. We haven't verified StreamExecutor works with that.
  53. 2018-04-23 16:17:38.324710: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  54. 2018-04-23 16:17:38.324940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties:
  55. name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.733
  56. pciBusID: 0000:01:00.0
  57. totalMemory: 5.93GiB freeMemory: 4.80GiB
  58. 2018-04-23 16:17:38.324952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0
  59. DEBUG 16:17:38.433 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489458430
  60. 2018-04-23 16:17:38.645731: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4539 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
  61. DEBUG 16:17:38.665 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489458662
  62. DEBUG 16:17:38.897 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489458894
  63. DEBUG 16:17:39.129 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489459126
  64. DEBUG 16:17:39.361 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489459358
  65. DEBUG 16:17:39.593 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489459590
  66. INFO 16:17:39.733 22864 command.py ------------ Loading module core.tracker ...OK
  67. DEBUG 16:17:39.828 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489459822
  68. DEBUG 16:17:40.061 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489460054
  69. DEBUG 16:17:40.293 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489460286
  70. DEBUG 16:17:40.525 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489460518
  71. DEBUG 16:17:40.759 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489460750
  72. DEBUG 16:17:40.989 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489460982
  73. DEBUG 16:17:41.221 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489461214
  74. DEBUG 16:17:41.453 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489461446
  75. DEBUG 16:17:41.684 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489461678
  76. DEBUG 16:17:41.918 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489461910
  77. DEBUG 16:17:42.150 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489462142
  78. DEBUG 16:17:42.381 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489462374
  79. DEBUG 16:17:42.613 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489462606
  80. DEBUG 16:17:42.615 22867 aiservice.py ce3625921a85 Batch for camera test collected, spend 5.94929575920105 sec
  81. INFO 16:17:42.617 22867 aiservice.py 4467122a4283 Start collecting batch for camera test, spend 5.7513415813446045 sec
  82. DEBUG 16:17:42.706 22842 shared_array.py ------------ Shared memory free 152, used 48
  83. INFO 16:17:42.707 22842 aiservice.py f03e3345b059 Put batch for camera test request to queue, spend 0.0025587081909179688 sec
  84. INFO 16:17:42.716 22842 aiservice.py ce3625921a85 Start processing batch for camera test, spend 6.05005955696106 sec
  85. DEBUG 16:17:42.717 22842 foreman.py ce3625921a85 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  86. DEBUG 16:17:42.844 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489462838
  87. DEBUG 16:17:42.891 22842 foreman.py ce3625921a85 Running=0 finished=1
  88. DEBUG 16:17:42.891 22842 foreman.py ce3625921a85 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  89. DEBUG 16:17:42.894 22865 command.py ce3625921a85 Start core.cnn
  90. 24
  91. [TensorRT] INFO: Detecting input data format
  92. [TensorRT] INFO: Dectected data format LCHW
  93. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  94. [TensorRT] INFO: Verifying batches are the expected data type
  95. [TensorRT] INFO: Executing inference
  96. [TensorRT] INFO: Number of Batches: 1
  97. ERROR 16:17:42.980 22865 command.py ce3625921a85 Worker core.cnn failed
  98. Traceback (most recent call last):
  99. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  100. result = worker.do(batch)
  101. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  102. batch.payload.get("image_idx")
  103. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  104. frames = self.process_frames(images, image_indexes)
  105. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  106. estimated = self.pose_estimator.estimate(images, self.scale_level)
  107. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  108. result = self.process(imgs, scale=scale)
  109. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  110. result.append(self._process(batch, **kwargs))
  111. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  112. l1s, l2s = self.engine.infer(items)
  113. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  114. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  115. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  116. DEBUG 16:17:42.986 22842 foreman.py ce3625921a85 Running=0 finished=1
  117. CRITICAL 16:17:42.986 22842 foreman.py ce3625921a85 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  118. CRITICAL 16:17:42.986 22842 foreman.py ce3625921a85 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  119. ERROR 16:17:42.987 22842 conclusion.py ------------ No Telegram's chats connected
  120. INFO 16:17:42.987 22842 aiservice.py ce3625921a85 Batch for camera test processed, spend 0.2707405090332031 sec
  121. DEBUG 16:17:42.987 22842 shared_array.py ------------ Shared memory free 152, used 48
  122. DEBUG 16:17:43.077 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489463070
  123. DEBUG 16:17:43.310 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489463302
  124. DEBUG 16:17:43.542 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489463534
  125. DEBUG 16:17:43.775 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489463766
  126. DEBUG 16:17:44.006 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489463998
  127. DEBUG 16:17:44.238 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489464230
  128. DEBUG 16:17:44.470 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489464462
  129. DEBUG 16:17:44.701 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489464694
  130. DEBUG 16:17:44.932 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489464926
  131. DEBUG 16:17:45.165 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489465158
  132. DEBUG 16:17:45.398 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489465390
  133. DEBUG 16:17:45.629 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489465622
  134. DEBUG 16:17:45.861 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489465854
  135. DEBUG 16:17:46.089 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489466086
  136. DEBUG 16:17:46.326 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489466318
  137. DEBUG 16:17:46.557 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489466550
  138. DEBUG 16:17:46.789 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489466782
  139. DEBUG 16:17:47.018 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489467014
  140. DEBUG 16:17:47.253 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489467246
  141. DEBUG 16:17:47.484 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489467478
  142. DEBUG 16:17:47.719 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489467710
  143. DEBUG 16:17:47.949 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489467942
  144. DEBUG 16:17:48.180 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489468174
  145. DEBUG 16:17:48.182 22867 aiservice.py 4467122a4283 Batch for camera test collected, spend 5.564433574676514 sec
  146. INFO 16:17:48.183 22867 aiservice.py f03e3345b059 Start collecting batch for camera test, spend 5.4780824184417725 sec
  147. INFO 16:17:48.235 22842 aiservice.py 4467122a4283 Start processing batch for camera test, spend 5.617734909057617 sec
  148. DEBUG 16:17:48.236 22842 foreman.py 4467122a4283 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  149. DEBUG 16:17:48.249 22842 foreman.py 4467122a4283 Running=0 finished=1
  150. DEBUG 16:17:48.249 22842 foreman.py 4467122a4283 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  151. DEBUG 16:17:48.251 22865 command.py 4467122a4283 Start core.cnn
  152. 24
  153. [TensorRT] INFO: Detecting input data format
  154. [TensorRT] INFO: Dectected data format LCHW
  155. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  156. [TensorRT] INFO: Verifying batches are the expected data type
  157. [TensorRT] INFO: Executing inference
  158. [TensorRT] INFO: Number of Batches: 1
  159. ERROR 16:17:48.329 22865 command.py 4467122a4283 Worker core.cnn failed
  160. Traceback (most recent call last):
  161. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  162. result = worker.do(batch)
  163. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  164. batch.payload.get("image_idx")
  165. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  166. frames = self.process_frames(images, image_indexes)
  167. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  168. estimated = self.pose_estimator.estimate(images, self.scale_level)
  169. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  170. result = self.process(imgs, scale=scale)
  171. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  172. result.append(self._process(batch, **kwargs))
  173. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  174. l1s, l2s = self.engine.infer(items)
  175. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  176. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  177. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  178. DEBUG 16:17:48.333 22842 foreman.py 4467122a4283 Running=0 finished=1
  179. CRITICAL 16:17:48.333 22842 foreman.py 4467122a4283 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5c0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  180. CRITICAL 16:17:48.333 22842 foreman.py 4467122a4283 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  181. ERROR 16:17:48.334 22842 conclusion.py ------------ No Telegram's chats connected
  182. INFO 16:17:48.334 22842 aiservice.py 4467122a4283 Batch for camera test processed, spend 0.09837770462036133 sec
  183. DEBUG 16:17:48.334 22842 shared_array.py ------------ Shared memory free 176, used 24
  184. DEBUG 16:17:48.357 22842 shared_array.py ------------ Shared memory free 176, used 24
  185. INFO 16:17:48.357 22842 aiservice.py befff4119a77 Put batch for camera test request to queue, spend 0.00043392181396484375 sec
  186. DEBUG 16:17:48.409 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489468406
  187. DEBUG 16:17:48.637 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489468638
  188. DEBUG 16:17:48.875 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489468870
  189. DEBUG 16:17:49.101 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489469102
  190. DEBUG 16:17:49.338 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489469334
  191. DEBUG 16:17:49.569 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489469566
  192. DEBUG 16:17:49.801 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489469798
  193. DEBUG 16:17:50.034 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489470030
  194. DEBUG 16:17:50.265 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489470262
  195. DEBUG 16:17:50.497 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489470494
  196. DEBUG 16:17:50.726 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489470726
  197. DEBUG 16:17:50.961 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489470958
  198. DEBUG 16:17:51.193 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489471190
  199. DEBUG 16:17:51.421 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489471422
  200. DEBUG 16:17:51.653 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489471654
  201. DEBUG 16:17:51.885 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489471886
  202. DEBUG 16:17:52.117 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489472118
  203. DEBUG 16:17:52.349 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489472350
  204. DEBUG 16:17:52.581 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489472582
  205. DEBUG 16:17:52.813 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489472814
  206. DEBUG 16:17:53.045 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489473046
  207. DEBUG 16:17:53.276 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489473278
  208. DEBUG 16:17:53.509 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489473510
  209. DEBUG 16:17:53.740 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489473742
  210. DEBUG 16:17:53.741 22867 aiservice.py f03e3345b059 Batch for camera test collected, spend 5.557866334915161 sec
  211. INFO 16:17:53.741 22867 aiservice.py befff4119a77 Start collecting batch for camera test, spend 5.384763479232788 sec
  212. INFO 16:17:53.783 22842 aiservice.py f03e3345b059 Start processing batch for camera test, spend 5.600019931793213 sec
  213. DEBUG 16:17:53.783 22842 foreman.py f03e3345b059 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  214. DEBUG 16:17:53.787 22842 foreman.py f03e3345b059 Running=0 finished=1
  215. DEBUG 16:17:53.787 22842 foreman.py f03e3345b059 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  216. DEBUG 16:17:53.788 22842 shared_array.py ------------ Shared memory free 152, used 48
  217. INFO 16:17:53.788 22842 aiservice.py 1f489afa0590 Put batch for camera test request to queue, spend 0.0006787776947021484 sec
  218. DEBUG 16:17:53.789 22865 command.py f03e3345b059 Start core.cnn
  219. 24
  220. [TensorRT] INFO: Detecting input data format
  221. [TensorRT] INFO: Dectected data format LCHW
  222. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  223. [TensorRT] INFO: Verifying batches are the expected data type
  224. [TensorRT] INFO: Executing inference
  225. [TensorRT] INFO: Number of Batches: 1
  226. ERROR 16:17:53.836 22865 command.py f03e3345b059 Worker core.cnn failed
  227. Traceback (most recent call last):
  228. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  229. result = worker.do(batch)
  230. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  231. batch.payload.get("image_idx")
  232. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  233. frames = self.process_frames(images, image_indexes)
  234. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  235. estimated = self.pose_estimator.estimate(images, self.scale_level)
  236. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  237. result = self.process(imgs, scale=scale)
  238. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  239. result.append(self._process(batch, **kwargs))
  240. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  241. l1s, l2s = self.engine.infer(items)
  242. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  243. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  244. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  245. DEBUG 16:17:53.839 22842 foreman.py f03e3345b059 Running=0 finished=1
  246. CRITICAL 16:17:53.839 22842 foreman.py f03e3345b059 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f668> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  247. CRITICAL 16:17:53.839 22842 foreman.py f03e3345b059 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  248. ERROR 16:17:53.840 22842 conclusion.py ------------ No Telegram's chats connected
  249. INFO 16:17:53.840 22842 aiservice.py f03e3345b059 Batch for camera test processed, spend 0.05674171447753906 sec
  250. DEBUG 16:17:53.840 22842 shared_array.py ------------ Shared memory free 152, used 48
  251. DEBUG 16:17:53.973 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489473974
  252. DEBUG 16:17:54.209 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489474206
  253. DEBUG 16:17:54.441 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489474438
  254. DEBUG 16:17:54.673 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489474670
  255. DEBUG 16:17:54.905 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489474898
  256. DEBUG 16:17:55.137 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489475130
  257. DEBUG 16:17:55.369 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489475362
  258. DEBUG 16:17:55.601 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489475595
  259. DEBUG 16:17:55.833 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489475826
  260. DEBUG 16:17:56.064 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489476058
  261. DEBUG 16:17:56.299 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489476291
  262. DEBUG 16:17:56.530 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489476523
  263. DEBUG 16:17:56.761 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489476755
  264. DEBUG 16:17:56.993 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489476987
  265. DEBUG 16:17:57.225 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489477219
  266. DEBUG 16:17:57.457 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489477450
  267. DEBUG 16:17:57.689 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489477682
  268. DEBUG 16:17:57.921 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489477915
  269. DEBUG 16:17:58.153 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489478147
  270. DEBUG 16:17:58.386 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489478379
  271. DEBUG 16:17:58.617 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489478611
  272. DEBUG 16:17:58.849 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489478843
  273. DEBUG 16:17:59.081 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489479075
  274. DEBUG 16:17:59.312 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489479307
  275. DEBUG 16:17:59.313 22867 aiservice.py befff4119a77 Batch for camera test collected, spend 5.571845054626465 sec
  276. INFO 16:17:59.314 22867 aiservice.py 1f489afa0590 Start collecting batch for camera test, spend 5.527013778686523 sec
  277. INFO 16:17:59.348 22842 aiservice.py befff4119a77 Start processing batch for camera test, spend 5.606193780899048 sec
  278. DEBUG 16:17:59.348 22842 foreman.py befff4119a77 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  279. DEBUG 16:17:59.361 22842 foreman.py befff4119a77 Running=0 finished=1
  280. DEBUG 16:17:59.361 22842 foreman.py befff4119a77 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  281. DEBUG 16:17:59.363 22865 command.py befff4119a77 Start core.cnn
  282. 24
  283. [TensorRT] INFO: Detecting input data format
  284. [TensorRT] INFO: Dectected data format LCHW
  285. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  286. [TensorRT] INFO: Verifying batches are the expected data type
  287. [TensorRT] INFO: Executing inference
  288. [TensorRT] INFO: Number of Batches: 1
  289. ERROR 16:17:59.414 22865 command.py befff4119a77 Worker core.cnn failed
  290. Traceback (most recent call last):
  291. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  292. result = worker.do(batch)
  293. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  294. batch.payload.get("image_idx")
  295. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  296. frames = self.process_frames(images, image_indexes)
  297. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  298. estimated = self.pose_estimator.estimate(images, self.scale_level)
  299. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  300. result = self.process(imgs, scale=scale)
  301. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  302. result.append(self._process(batch, **kwargs))
  303. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  304. l1s, l2s = self.engine.infer(items)
  305. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  306. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  307. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  308. DEBUG 16:17:59.424 22842 foreman.py befff4119a77 Running=0 finished=1
  309. CRITICAL 16:17:59.424 22842 foreman.py befff4119a77 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b62271668> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  310. CRITICAL 16:17:59.424 22842 foreman.py befff4119a77 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  311. ERROR 16:17:59.425 22842 conclusion.py ------------ No Telegram's chats connected
  312. INFO 16:17:59.425 22842 aiservice.py befff4119a77 Batch for camera test processed, spend 0.0767056941986084 sec
  313. DEBUG 16:17:59.425 22842 shared_array.py ------------ Shared memory free 176, used 24
  314. DEBUG 16:17:59.429 22842 shared_array.py ------------ Shared memory free 176, used 24
  315. INFO 16:17:59.430 22842 aiservice.py 1c39f24ccde0 Put batch for camera test request to queue, spend 0.00040149688720703125 sec
  316. DEBUG 16:17:59.545 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489479539
  317. DEBUG 16:17:59.777 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489479771
  318. DEBUG 16:18:00.009 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489480003
  319. DEBUG 16:18:00.240 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489480235
  320. DEBUG 16:18:00.473 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489480467
  321. DEBUG 16:18:00.706 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489480699
  322. DEBUG 16:18:00.937 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489480931
  323. DEBUG 16:18:01.170 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489481163
  324. DEBUG 16:18:01.401 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489481395
  325. DEBUG 16:18:01.634 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489481627
  326. DEBUG 16:18:01.865 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489481859
  327. DEBUG 16:18:02.097 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489482091
  328. DEBUG 16:18:02.325 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489482323
  329. DEBUG 16:18:02.561 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489482555
  330. DEBUG 16:18:02.793 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489482787
  331. DEBUG 16:18:03.024 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489483019
  332. DEBUG 16:18:03.257 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489483251
  333. DEBUG 16:18:03.491 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489483483
  334. DEBUG 16:18:03.721 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489483715
  335. DEBUG 16:18:03.953 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489483947
  336. DEBUG 16:18:04.186 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489484179
  337. DEBUG 16:18:04.417 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489484411
  338. DEBUG 16:18:04.650 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489484643
  339. DEBUG 16:18:04.881 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489484875
  340. DEBUG 16:18:04.882 22867 aiservice.py 1f489afa0590 Batch for camera test collected, spend 5.567297697067261 sec
  341. INFO 16:18:04.883 22867 aiservice.py 1c39f24ccde0 Start collecting batch for camera test, spend 5.453832387924194 sec
  342. INFO 16:18:04.911 22842 aiservice.py 1f489afa0590 Start processing batch for camera test, spend 5.596738338470459 sec
  343. DEBUG 16:18:04.912 22842 foreman.py 1f489afa0590 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  344. DEBUG 16:18:04.926 22842 foreman.py 1f489afa0590 Running=0 finished=1
  345. DEBUG 16:18:04.926 22842 foreman.py 1f489afa0590 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  346. DEBUG 16:18:04.928 22865 command.py 1f489afa0590 Start core.cnn
  347. 24
  348. [TensorRT] INFO: Detecting input data format
  349. [TensorRT] INFO: Dectected data format LCHW
  350. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  351. [TensorRT] INFO: Verifying batches are the expected data type
  352. [TensorRT] INFO: Executing inference
  353. [TensorRT] INFO: Number of Batches: 1
  354. ERROR 16:18:05.006 22865 command.py 1f489afa0590 Worker core.cnn failed
  355. Traceback (most recent call last):
  356. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  357. result = worker.do(batch)
  358. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  359. batch.payload.get("image_idx")
  360. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  361. frames = self.process_frames(images, image_indexes)
  362. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  363. estimated = self.pose_estimator.estimate(images, self.scale_level)
  364. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  365. result = self.process(imgs, scale=scale)
  366. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  367. result.append(self._process(batch, **kwargs))
  368. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  369. l1s, l2s = self.engine.infer(items)
  370. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  371. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  372. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  373. DEBUG 16:18:05.011 22842 foreman.py 1f489afa0590 Running=0 finished=1
  374. CRITICAL 16:18:05.011 22842 foreman.py 1f489afa0590 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f4a8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  375. CRITICAL 16:18:05.011 22842 foreman.py 1f489afa0590 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  376. ERROR 16:18:05.012 22842 conclusion.py ------------ No Telegram's chats connected
  377. INFO 16:18:05.012 22842 aiservice.py 1f489afa0590 Batch for camera test processed, spend 0.1003570556640625 sec
  378. DEBUG 16:18:05.012 22842 shared_array.py ------------ Shared memory free 176, used 24
  379. DEBUG 16:18:05.076 22842 shared_array.py ------------ Shared memory free 176, used 24
  380. INFO 16:18:05.077 22842 aiservice.py 232cff25b6e6 Put batch for camera test request to queue, spend 0.0011816024780273438 sec
  381. DEBUG 16:18:05.112 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489485107
  382. DEBUG 16:18:05.345 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489485339
  383. DEBUG 16:18:05.578 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489485571
  384. DEBUG 16:18:05.810 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489485803
  385. DEBUG 16:18:06.043 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489486035
  386. DEBUG 16:18:06.275 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489486267
  387. DEBUG 16:18:06.505 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489486499
  388. DEBUG 16:18:06.738 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489486731
  389. DEBUG 16:18:06.967 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489486963
  390. DEBUG 16:18:07.201 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489487195
  391. DEBUG 16:18:07.433 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489487427
  392. DEBUG 16:18:07.665 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489487659
  393. DEBUG 16:18:07.897 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489487891
  394. DEBUG 16:18:08.129 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489488123
  395. DEBUG 16:18:08.361 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489488355
  396. DEBUG 16:18:08.593 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489488587
  397. DEBUG 16:18:08.825 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489488819
  398. DEBUG 16:18:09.058 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489489051
  399. DEBUG 16:18:09.285 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489489283
  400. DEBUG 16:18:09.517 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489489515
  401. DEBUG 16:18:09.749 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489489747
  402. DEBUG 16:18:09.987 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489489979
  403. DEBUG 16:18:10.217 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489490211
  404. DEBUG 16:18:10.449 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489490443
  405. DEBUG 16:18:10.450 22867 aiservice.py 1c39f24ccde0 Batch for camera test collected, spend 5.566927433013916 sec
  406. INFO 16:18:10.451 22867 aiservice.py 232cff25b6e6 Start collecting batch for camera test, spend 5.375347137451172 sec
  407. INFO 16:18:10.458 22842 aiservice.py 1c39f24ccde0 Start processing batch for camera test, spend 5.5742881298065186 sec
  408. DEBUG 16:18:10.458 22842 foreman.py 1c39f24ccde0 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  409. 24
  410. DEBUG 16:18:10.465 22842 foreman.py 1c39f24ccde0 Running=0 finished=1
  411. DEBUG 16:18:10.465 22842 foreman.py 1c39f24ccde0 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  412. DEBUG 16:18:10.465 22865 command.py 1c39f24ccde0 Start core.cnn
  413. [TensorRT] INFO: Detecting input data format
  414. [TensorRT] INFO: Dectected data format LCHW
  415. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  416. [TensorRT] INFO: Verifying batches are the expected data type
  417. [TensorRT] INFO: Executing inference
  418. [TensorRT] INFO: Number of Batches: 1
  419. ERROR 16:18:10.512 22865 command.py 1c39f24ccde0 Worker core.cnn failed
  420. Traceback (most recent call last):
  421. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  422. result = worker.do(batch)
  423. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  424. batch.payload.get("image_idx")
  425. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  426. frames = self.process_frames(images, image_indexes)
  427. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  428. estimated = self.pose_estimator.estimate(images, self.scale_level)
  429. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  430. result = self.process(imgs, scale=scale)
  431. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  432. result.append(self._process(batch, **kwargs))
  433. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  434. l1s, l2s = self.engine.infer(items)
  435. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  436. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  437. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  438. DEBUG 16:18:10.516 22842 shared_array.py ------------ Shared memory free 152, used 48
  439. INFO 16:18:10.517 22842 aiservice.py 53e12f3678be Put batch for camera test request to queue, spend 0.0007183551788330078 sec
  440. DEBUG 16:18:10.520 22842 foreman.py 1c39f24ccde0 Running=0 finished=1
  441. CRITICAL 16:18:10.520 22842 foreman.py 1c39f24ccde0 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f630> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  442. CRITICAL 16:18:10.520 22842 foreman.py 1c39f24ccde0 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  443. ERROR 16:18:10.520 22842 conclusion.py ------------ No Telegram's chats connected
  444. INFO 16:18:10.520 22842 aiservice.py 1c39f24ccde0 Batch for camera test processed, spend 0.06234860420227051 sec
  445. DEBUG 16:18:10.521 22842 shared_array.py ------------ Shared memory free 152, used 48
  446. DEBUG 16:18:10.678 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489490675
  447. DEBUG 16:18:10.913 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489490907
  448. DEBUG 16:18:11.145 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489491139
  449. DEBUG 16:18:11.377 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489491371
  450. DEBUG 16:18:11.605 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489491603
  451. DEBUG 16:18:11.842 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489491835
  452. DEBUG 16:18:12.073 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489492067
  453. DEBUG 16:18:12.306 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489492299
  454. DEBUG 16:18:12.537 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489492531
  455. DEBUG 16:18:12.769 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489492763
  456. DEBUG 16:18:13.003 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489492995
  457. DEBUG 16:18:13.233 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489493227
  458. DEBUG 16:18:13.465 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489493459
  459. DEBUG 16:18:13.697 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489493691
  460. DEBUG 16:18:13.929 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489493923
  461. DEBUG 16:18:14.161 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489494155
  462. DEBUG 16:18:14.394 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489494387
  463. DEBUG 16:18:14.625 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489494619
  464. DEBUG 16:18:14.859 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489494851
  465. DEBUG 16:18:15.090 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489495083
  466. DEBUG 16:18:15.321 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489495315
  467. DEBUG 16:18:15.553 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489495547
  468. DEBUG 16:18:15.787 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489495779
  469. DEBUG 16:18:16.018 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489496011
  470. DEBUG 16:18:16.019 22867 aiservice.py 232cff25b6e6 Batch for camera test collected, spend 5.568118333816528 sec
  471. INFO 16:18:16.021 22867 aiservice.py 53e12f3678be Start collecting batch for camera test, spend 5.504440784454346 sec
  472. INFO 16:18:16.071 22842 aiservice.py 232cff25b6e6 Start processing batch for camera test, spend 5.619923114776611 sec
  473. DEBUG 16:18:16.072 22842 foreman.py 232cff25b6e6 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  474. DEBUG 16:18:16.087 22842 foreman.py 232cff25b6e6 Running=0 finished=1
  475. DEBUG 16:18:16.087 22842 foreman.py 232cff25b6e6 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  476. DEBUG 16:18:16.089 22865 command.py 232cff25b6e6 Start core.cnn
  477. 24
  478. [TensorRT] INFO: Detecting input data format
  479. [TensorRT] INFO: Dectected data format LCHW
  480. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  481. DEBUG 16:18:16.161 22842 shared_array.py ------------ Shared memory free 152, used 48
  482. INFO 16:18:16.161 22842 aiservice.py 1ecb5c69a916 Put batch for camera test request to queue, spend 0.0006725788116455078 sec
  483. [TensorRT] INFO: Verifying batches are the expected data type
  484. [TensorRT] INFO: Executing inference
  485. [TensorRT] INFO: Number of Batches: 1
  486. ERROR 16:18:16.165 22865 command.py 232cff25b6e6 Worker core.cnn failed
  487. Traceback (most recent call last):
  488. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  489. result = worker.do(batch)
  490. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  491. batch.payload.get("image_idx")
  492. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  493. frames = self.process_frames(images, image_indexes)
  494. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  495. estimated = self.pose_estimator.estimate(images, self.scale_level)
  496. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  497. result = self.process(imgs, scale=scale)
  498. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  499. result.append(self._process(batch, **kwargs))
  500. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  501. l1s, l2s = self.engine.infer(items)
  502. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  503. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  504. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  505. DEBUG 16:18:16.172 22842 foreman.py 232cff25b6e6 Running=0 finished=1
  506. CRITICAL 16:18:16.172 22842 foreman.py 232cff25b6e6 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f748> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  507. CRITICAL 16:18:16.172 22842 foreman.py 232cff25b6e6 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  508. ERROR 16:18:16.173 22842 conclusion.py ------------ No Telegram's chats connected
  509. INFO 16:18:16.173 22842 aiservice.py 232cff25b6e6 Batch for camera test processed, spend 0.10134625434875488 sec
  510. DEBUG 16:18:16.173 22842 shared_array.py ------------ Shared memory free 152, used 48
  511. DEBUG 16:18:16.248 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489496243
  512. DEBUG 16:18:16.478 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489496475
  513. DEBUG 16:18:16.713 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489496707
  514. DEBUG 16:18:16.947 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489496939
  515. DEBUG 16:18:17.178 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489497171
  516. DEBUG 16:18:17.409 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489497403
  517. DEBUG 16:18:17.641 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489497635
  518. DEBUG 16:18:17.870 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489497867
  519. DEBUG 16:18:18.101 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489498099
  520. DEBUG 16:18:18.334 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489498331
  521. DEBUG 16:18:18.565 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489498563
  522. DEBUG 16:18:18.793 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489498795
  523. DEBUG 16:18:19.025 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489499027
  524. DEBUG 16:18:19.261 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489499259
  525. DEBUG 16:18:19.491 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489499491
  526. DEBUG 16:18:19.725 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489499723
  527. DEBUG 16:18:19.957 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489499955
  528. DEBUG 16:18:20.190 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489500187
  529. DEBUG 16:18:20.421 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489500419
  530. DEBUG 16:18:20.652 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489500651
  531. DEBUG 16:18:20.886 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489500883
  532. DEBUG 16:18:21.117 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489501115
  533. DEBUG 16:18:21.349 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489501347
  534. DEBUG 16:18:21.577 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489501579
  535. DEBUG 16:18:21.577 22867 aiservice.py 53e12f3678be Batch for camera test collected, spend 5.55617356300354 sec
  536. INFO 16:18:21.577 22867 aiservice.py 1ecb5c69a916 Start collecting batch for camera test, spend 5.416975736618042 sec
  537. DEBUG 16:18:21.605 22842 shared_array.py ------------ Shared memory free 152, used 48
  538. INFO 16:18:21.605 22842 aiservice.py 795de217e0c5 Put batch for camera test request to queue, spend 0.0007016658782958984 sec
  539. INFO 16:18:21.633 22842 aiservice.py 53e12f3678be Start processing batch for camera test, spend 5.6117095947265625 sec
  540. DEBUG 16:18:21.633 22842 foreman.py 53e12f3678be Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  541. 24
  542. DEBUG 16:18:21.637 22842 foreman.py 53e12f3678be Running=0 finished=1
  543. DEBUG 16:18:21.637 22842 foreman.py 53e12f3678be Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  544. DEBUG 16:18:21.637 22865 command.py 53e12f3678be Start core.cnn
  545. [TensorRT] INFO: Detecting input data format
  546. [TensorRT] INFO: Dectected data format LCHW
  547. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  548. [TensorRT] INFO: Verifying batches are the expected data type
  549. [TensorRT] INFO: Executing inference
  550. [TensorRT] INFO: Number of Batches: 1
  551. ERROR 16:18:21.687 22865 command.py 53e12f3678be Worker core.cnn failed
  552. Traceback (most recent call last):
  553. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  554. result = worker.do(batch)
  555. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  556. batch.payload.get("image_idx")
  557. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  558. frames = self.process_frames(images, image_indexes)
  559. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  560. estimated = self.pose_estimator.estimate(images, self.scale_level)
  561. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  562. result = self.process(imgs, scale=scale)
  563. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  564. result.append(self._process(batch, **kwargs))
  565. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  566. l1s, l2s = self.engine.infer(items)
  567. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  568. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  569. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  570. DEBUG 16:18:21.693 22842 foreman.py 53e12f3678be Running=0 finished=1
  571. CRITICAL 16:18:21.693 22842 foreman.py 53e12f3678be Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f438> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  572. CRITICAL 16:18:21.693 22842 foreman.py 53e12f3678be Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  573. ERROR 16:18:21.693 22842 conclusion.py ------------ No Telegram's chats connected
  574. INFO 16:18:21.693 22842 aiservice.py 53e12f3678be Batch for camera test processed, spend 0.06073164939880371 sec
  575. DEBUG 16:18:21.694 22842 shared_array.py ------------ Shared memory free 152, used 48
  576. DEBUG 16:18:21.810 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489501811
  577. DEBUG 16:18:22.041 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489502043
  578. DEBUG 16:18:22.277 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489502275
  579. DEBUG 16:18:22.509 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489502507
  580. DEBUG 16:18:22.741 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489502739
  581. DEBUG 16:18:22.973 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489502971
  582. DEBUG 16:18:23.205 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489503203
  583. DEBUG 16:18:23.437 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489503435
  584. DEBUG 16:18:23.669 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489503667
  585. DEBUG 16:18:23.901 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489503899
  586. DEBUG 16:18:24.133 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489504131
  587. DEBUG 16:18:24.361 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489504359
  588. DEBUG 16:18:24.594 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489504591
  589. DEBUG 16:18:24.830 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489504823
  590. DEBUG 16:18:25.062 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489505055
  591. DEBUG 16:18:25.293 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489505287
  592. DEBUG 16:18:25.521 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489505519
  593. DEBUG 16:18:25.757 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489505751
  594. DEBUG 16:18:25.990 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489505983
  595. DEBUG 16:18:26.221 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489506215
  596. DEBUG 16:18:26.454 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489506447
  597. DEBUG 16:18:26.685 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489506679
  598. DEBUG 16:18:26.917 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489506911
  599. DEBUG 16:18:27.150 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489507143
  600. DEBUG 16:18:27.152 22867 aiservice.py 1ecb5c69a916 Batch for camera test collected, spend 5.575019598007202 sec
  601. INFO 16:18:27.153 22867 aiservice.py 795de217e0c5 Start collecting batch for camera test, spend 5.548831224441528 sec
  602. INFO 16:18:27.203 22842 aiservice.py 1ecb5c69a916 Start processing batch for camera test, spend 5.626060724258423 sec
  603. DEBUG 16:18:27.204 22842 foreman.py 1ecb5c69a916 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  604. DEBUG 16:18:27.217 22842 foreman.py 1ecb5c69a916 Running=0 finished=1
  605. DEBUG 16:18:27.217 22842 foreman.py 1ecb5c69a916 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  606. DEBUG 16:18:27.219 22865 command.py 1ecb5c69a916 Start core.cnn
  607. 24
  608. DEBUG 16:18:27.246 22842 shared_array.py ------------ Shared memory free 152, used 48
  609. INFO 16:18:27.247 22842 aiservice.py 7c365adaad95 Put batch for camera test request to queue, spend 0.0018880367279052734 sec
  610. [TensorRT] INFO: Detecting input data format
  611. [TensorRT] INFO: Dectected data format LCHW
  612. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  613. [TensorRT] INFO: Verifying batches are the expected data type
  614. [TensorRT] INFO: Executing inference
  615. [TensorRT] INFO: Number of Batches: 1
  616. ERROR 16:18:27.287 22865 command.py 1ecb5c69a916 Worker core.cnn failed
  617. Traceback (most recent call last):
  618. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  619. result = worker.do(batch)
  620. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  621. batch.payload.get("image_idx")
  622. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  623. frames = self.process_frames(images, image_indexes)
  624. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  625. estimated = self.pose_estimator.estimate(images, self.scale_level)
  626. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  627. result = self.process(imgs, scale=scale)
  628. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  629. result.append(self._process(batch, **kwargs))
  630. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  631. l1s, l2s = self.engine.infer(items)
  632. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  633. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  634. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  635. DEBUG 16:18:27.291 22842 foreman.py 1ecb5c69a916 Running=0 finished=1
  636. CRITICAL 16:18:27.291 22842 foreman.py 1ecb5c69a916 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5f8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  637. CRITICAL 16:18:27.291 22842 foreman.py 1ecb5c69a916 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  638. ERROR 16:18:27.292 22842 conclusion.py ------------ No Telegram's chats connected
  639. INFO 16:18:27.292 22842 aiservice.py 1ecb5c69a916 Batch for camera test processed, spend 0.08831429481506348 sec
  640. DEBUG 16:18:27.292 22842 shared_array.py ------------ Shared memory free 152, used 48
  641. DEBUG 16:18:27.383 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489507375
  642. DEBUG 16:18:27.613 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489507607
  643. DEBUG 16:18:27.845 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489507839
  644. DEBUG 16:18:28.077 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489508071
  645. DEBUG 16:18:28.309 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489508303
  646. DEBUG 16:18:28.537 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489508535
  647. DEBUG 16:18:28.773 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489508767
  648. DEBUG 16:18:29.007 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489508999
  649. DEBUG 16:18:29.237 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489509231
  650. DEBUG 16:18:29.470 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489509463
  651. DEBUG 16:18:29.701 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489509695
  652. DEBUG 16:18:29.933 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489509927
  653. DEBUG 16:18:30.161 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489510159
  654. DEBUG 16:18:30.397 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489510391
  655. DEBUG 16:18:30.630 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489510623
  656. DEBUG 16:18:30.861 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489510855
  657. DEBUG 16:18:31.094 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489511087
  658. DEBUG 16:18:31.325 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489511319
  659. DEBUG 16:18:31.557 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489511551
  660. DEBUG 16:18:31.789 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489511783
  661. DEBUG 16:18:32.021 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489512015
  662. DEBUG 16:18:32.249 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489512247
  663. DEBUG 16:18:32.483 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489512479
  664. DEBUG 16:18:32.718 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489512711
  665. DEBUG 16:18:32.720 22867 aiservice.py 795de217e0c5 Batch for camera test collected, spend 5.566285848617554 sec
  666. INFO 16:18:32.720 22867 aiservice.py 7c365adaad95 Start collecting batch for camera test, spend 5.47514009475708 sec
  667. INFO 16:18:32.752 22842 aiservice.py 795de217e0c5 Start processing batch for camera test, spend 5.59834361076355 sec
  668. DEBUG 16:18:32.753 22842 foreman.py 795de217e0c5 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  669. DEBUG 16:18:32.766 22842 foreman.py 795de217e0c5 Running=0 finished=1
  670. DEBUG 16:18:32.767 22842 foreman.py 795de217e0c5 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  671. DEBUG 16:18:32.769 22865 command.py 795de217e0c5 Start core.cnn
  672. 24
  673. [TensorRT] INFO: Detecting input data format
  674. [TensorRT] INFO: Dectected data format LCHW
  675. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  676. [TensorRT] INFO: Verifying batches are the expected data type
  677. [TensorRT] INFO: Executing inference
  678. [TensorRT] INFO: Number of Batches: 1
  679. ERROR 16:18:32.837 22865 command.py 795de217e0c5 Worker core.cnn failed
  680. Traceback (most recent call last):
  681. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  682. result = worker.do(batch)
  683. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  684. batch.payload.get("image_idx")
  685. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  686. frames = self.process_frames(images, image_indexes)
  687. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  688. estimated = self.pose_estimator.estimate(images, self.scale_level)
  689. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  690. result = self.process(imgs, scale=scale)
  691. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  692. result.append(self._process(batch, **kwargs))
  693. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  694. l1s, l2s = self.engine.infer(items)
  695. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  696. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  697. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  698. DEBUG 16:18:32.840 22842 foreman.py 795de217e0c5 Running=0 finished=1
  699. CRITICAL 16:18:32.840 22842 foreman.py 795de217e0c5 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f710> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  700. CRITICAL 16:18:32.840 22842 foreman.py 795de217e0c5 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  701. ERROR 16:18:32.841 22842 conclusion.py ------------ No Telegram's chats connected
  702. INFO 16:18:32.841 22842 aiservice.py 795de217e0c5 Batch for camera test processed, spend 0.08893442153930664 sec
  703. DEBUG 16:18:32.841 22842 shared_array.py ------------ Shared memory free 176, used 24
  704. DEBUG 16:18:32.900 22842 shared_array.py ------------ Shared memory free 176, used 24
  705. INFO 16:18:32.900 22842 aiservice.py 6f7e607d618c Put batch for camera test request to queue, spend 0.0008256435394287109 sec
  706. DEBUG 16:18:32.950 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489512943
  707. DEBUG 16:18:33.181 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489513175
  708. DEBUG 16:18:33.414 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489513407
  709. DEBUG 16:18:33.641 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489513639
  710. DEBUG 16:18:33.877 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489513871
  711. DEBUG 16:18:34.110 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489514103
  712. DEBUG 16:18:34.341 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489514335
  713. DEBUG 16:18:34.575 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489514567
  714. DEBUG 16:18:34.801 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489514799
  715. DEBUG 16:18:35.037 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489515031
  716. DEBUG 16:18:35.269 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489515263
  717. DEBUG 16:18:35.502 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489515495
  718. DEBUG 16:18:35.734 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489515727
  719. DEBUG 16:18:35.961 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489515959
  720. DEBUG 16:18:36.194 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489516191
  721. DEBUG 16:18:36.425 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489516423
  722. DEBUG 16:18:36.657 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489516655
  723. DEBUG 16:18:36.889 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489516887
  724. DEBUG 16:18:37.125 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489517119
  725. DEBUG 16:18:37.357 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489517351
  726. DEBUG 16:18:37.589 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489517583
  727. DEBUG 16:18:37.817 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489517815
  728. DEBUG 16:18:38.054 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489518047
  729. DEBUG 16:18:38.285 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489518279
  730. DEBUG 16:18:38.287 22867 aiservice.py 7c365adaad95 Batch for camera test collected, spend 5.566610813140869 sec
  731. INFO 16:18:38.288 22867 aiservice.py 6f7e607d618c Start collecting batch for camera test, spend 5.388692140579224 sec
  732. INFO 16:18:38.295 22842 aiservice.py 7c365adaad95 Start processing batch for camera test, spend 5.574621915817261 sec
  733. DEBUG 16:18:38.296 22842 foreman.py 7c365adaad95 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  734. DEBUG 16:18:38.309 22842 foreman.py 7c365adaad95 Running=0 finished=1
  735. DEBUG 16:18:38.309 22842 foreman.py 7c365adaad95 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  736. DEBUG 16:18:38.311 22865 command.py 7c365adaad95 Start core.cnn
  737. 24
  738. DEBUG 16:18:38.348 22842 shared_array.py ------------ Shared memory free 152, used 48
  739. INFO 16:18:38.349 22842 aiservice.py 27c10e4a6d06 Put batch for camera test request to queue, spend 0.0012497901916503906 sec
  740. [TensorRT] INFO: Detecting input data format
  741. [TensorRT] INFO: Dectected data format LCHW
  742. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  743. [TensorRT] INFO: Verifying batches are the expected data type
  744. [TensorRT] INFO: Executing inference
  745. [TensorRT] INFO: Number of Batches: 1
  746. ERROR 16:18:38.387 22865 command.py 7c365adaad95 Worker core.cnn failed
  747. Traceback (most recent call last):
  748. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  749. result = worker.do(batch)
  750. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  751. batch.payload.get("image_idx")
  752. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  753. frames = self.process_frames(images, image_indexes)
  754. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  755. estimated = self.pose_estimator.estimate(images, self.scale_level)
  756. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  757. result = self.process(imgs, scale=scale)
  758. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  759. result.append(self._process(batch, **kwargs))
  760. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  761. l1s, l2s = self.engine.infer(items)
  762. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  763. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  764. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  765. DEBUG 16:18:38.395 22842 foreman.py 7c365adaad95 Running=0 finished=1
  766. CRITICAL 16:18:38.395 22842 foreman.py 7c365adaad95 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5c0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  767. CRITICAL 16:18:38.395 22842 foreman.py 7c365adaad95 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  768. ERROR 16:18:38.396 22842 conclusion.py ------------ No Telegram's chats connected
  769. INFO 16:18:38.396 22842 aiservice.py 7c365adaad95 Batch for camera test processed, spend 0.10014724731445312 sec
  770. DEBUG 16:18:38.396 22842 shared_array.py ------------ Shared memory free 152, used 48
  771. DEBUG 16:18:38.517 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489518511
  772. DEBUG 16:18:38.749 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489518743
  773. DEBUG 16:18:38.981 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489518975
  774. DEBUG 16:18:39.213 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489519207
  775. DEBUG 16:18:39.445 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489519439
  776. DEBUG 16:18:39.677 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489519671
  777. DEBUG 16:18:39.910 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489519903
  778. DEBUG 16:18:40.138 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489520135
  779. DEBUG 16:18:40.369 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489520367
  780. DEBUG 16:18:40.606 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489520599
  781. DEBUG 16:18:40.837 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489520831
  782. DEBUG 16:18:41.069 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489521063
  783. DEBUG 16:18:41.299 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489521295
  784. DEBUG 16:18:41.536 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489521527
  785. DEBUG 16:18:41.765 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489521759
  786. DEBUG 16:18:41.998 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489521991
  787. DEBUG 16:18:42.229 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489522223
  788. DEBUG 16:18:42.462 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489522455
  789. DEBUG 16:18:42.693 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489522687
  790. DEBUG 16:18:42.926 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489522919
  791. DEBUG 16:18:43.158 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489523151
  792. DEBUG 16:18:43.390 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489523383
  793. DEBUG 16:18:43.622 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489523615
  794. DEBUG 16:18:43.853 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489523847
  795. DEBUG 16:18:43.854 22867 aiservice.py 6f7e607d618c Batch for camera test collected, spend 5.566078424453735 sec
  796. INFO 16:18:43.855 22867 aiservice.py 27c10e4a6d06 Start collecting batch for camera test, spend 5.507776498794556 sec
  797. INFO 16:18:43.858 22842 aiservice.py 6f7e607d618c Start processing batch for camera test, spend 5.5701584815979 sec
  798. DEBUG 16:18:43.859 22842 foreman.py 6f7e607d618c Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  799. DEBUG 16:18:43.872 22842 foreman.py 6f7e607d618c Running=0 finished=1
  800. DEBUG 16:18:43.872 22842 foreman.py 6f7e607d618c Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  801. DEBUG 16:18:43.873 22865 command.py 6f7e607d618c Start core.cnn
  802. 24
  803. [TensorRT] INFO: Detecting input data format
  804. [TensorRT] INFO: Dectected data format LCHW
  805. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  806. [TensorRT] INFO: Verifying batches are the expected data type
  807. [TensorRT] INFO: Executing inference
  808. [TensorRT] INFO: Number of Batches: 1
  809. ERROR 16:18:43.940 22865 command.py 6f7e607d618c Worker core.cnn failed
  810. Traceback (most recent call last):
  811. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  812. result = worker.do(batch)
  813. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  814. batch.payload.get("image_idx")
  815. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  816. frames = self.process_frames(images, image_indexes)
  817. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  818. estimated = self.pose_estimator.estimate(images, self.scale_level)
  819. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  820. result = self.process(imgs, scale=scale)
  821. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  822. result.append(self._process(batch, **kwargs))
  823. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  824. l1s, l2s = self.engine.infer(items)
  825. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  826. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  827. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  828. DEBUG 16:18:43.945 22842 foreman.py 6f7e607d618c Running=0 finished=1
  829. CRITICAL 16:18:43.946 22842 foreman.py 6f7e607d618c Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f4e0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  830. CRITICAL 16:18:43.946 22842 foreman.py 6f7e607d618c Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  831. ERROR 16:18:43.946 22842 conclusion.py ------------ No Telegram's chats connected
  832. INFO 16:18:43.946 22842 aiservice.py 6f7e607d618c Batch for camera test processed, spend 0.08782482147216797 sec
  833. DEBUG 16:18:43.947 22842 shared_array.py ------------ Shared memory free 176, used 24
  834. DEBUG 16:18:44.000 22842 shared_array.py ------------ Shared memory free 176, used 24
  835. INFO 16:18:44.001 22842 aiservice.py e731f6cbae6a Put batch for camera test request to queue, spend 0.0012288093566894531 sec
  836. DEBUG 16:18:44.085 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489524079
  837. DEBUG 16:18:44.319 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489524311
  838. DEBUG 16:18:44.549 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489524543
  839. DEBUG 16:18:44.782 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489524775
  840. DEBUG 16:18:45.013 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489525007
  841. DEBUG 16:18:45.246 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489525239
  842. DEBUG 16:18:45.478 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489525471
  843. DEBUG 16:18:45.709 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489525703
  844. DEBUG 16:18:45.942 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489525935
  845. DEBUG 16:18:46.174 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489526167
  846. DEBUG 16:18:46.406 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489526399
  847. DEBUG 16:18:46.637 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489526631
  848. DEBUG 16:18:46.871 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489526863
  849. DEBUG 16:18:47.098 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489527095
  850. DEBUG 16:18:47.329 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489527327
  851. DEBUG 16:18:47.562 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489527559
  852. DEBUG 16:18:47.794 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489527791
  853. DEBUG 16:18:48.025 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489528023
  854. DEBUG 16:18:48.258 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489528255
  855. DEBUG 16:18:48.490 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489528487
  856. DEBUG 16:18:48.723 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489528719
  857. DEBUG 16:18:48.954 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489528951
  858. DEBUG 16:18:49.186 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489529183
  859. DEBUG 16:18:49.417 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489529415
  860. DEBUG 16:18:49.418 22867 aiservice.py 27c10e4a6d06 Batch for camera test collected, spend 5.562716960906982 sec
  861. INFO 16:18:49.419 22867 aiservice.py e731f6cbae6a Start collecting batch for camera test, spend 5.419525623321533 sec
  862. DEBUG 16:18:49.442 22842 shared_array.py ------------ Shared memory free 152, used 48
  863. INFO 16:18:49.443 22842 aiservice.py e2c6eb8520f5 Put batch for camera test request to queue, spend 0.002534151077270508 sec
  864. INFO 16:18:49.446 22842 aiservice.py 27c10e4a6d06 Start processing batch for camera test, spend 5.590879440307617 sec
  865. DEBUG 16:18:49.447 22842 foreman.py 27c10e4a6d06 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  866. DEBUG 16:18:49.460 22842 foreman.py 27c10e4a6d06 Running=0 finished=1
  867. DEBUG 16:18:49.461 22842 foreman.py 27c10e4a6d06 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  868. 24
  869. DEBUG 16:18:49.462 22865 command.py 27c10e4a6d06 Start core.cnn
  870. [TensorRT] INFO: Detecting input data format
  871. [TensorRT] INFO: Dectected data format LCHW
  872. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  873. [TensorRT] INFO: Verifying batches are the expected data type
  874. [TensorRT] INFO: Executing inference
  875. [TensorRT] INFO: Number of Batches: 1
  876. ERROR 16:18:49.539 22865 command.py 27c10e4a6d06 Worker core.cnn failed
  877. Traceback (most recent call last):
  878. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  879. result = worker.do(batch)
  880. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  881. batch.payload.get("image_idx")
  882. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  883. frames = self.process_frames(images, image_indexes)
  884. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  885. estimated = self.pose_estimator.estimate(images, self.scale_level)
  886. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  887. result = self.process(imgs, scale=scale)
  888. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  889. result.append(self._process(batch, **kwargs))
  890. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  891. l1s, l2s = self.engine.infer(items)
  892. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  893. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  894. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  895. DEBUG 16:18:49.545 22842 foreman.py 27c10e4a6d06 Running=0 finished=1
  896. CRITICAL 16:18:49.545 22842 foreman.py 27c10e4a6d06 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f668> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  897. CRITICAL 16:18:49.545 22842 foreman.py 27c10e4a6d06 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  898. ERROR 16:18:49.546 22842 conclusion.py ------------ No Telegram's chats connected
  899. INFO 16:18:49.546 22842 aiservice.py 27c10e4a6d06 Batch for camera test processed, spend 0.09958648681640625 sec
  900. DEBUG 16:18:49.546 22842 shared_array.py ------------ Shared memory free 152, used 48
  901. DEBUG 16:18:49.645 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489529647
  902. DEBUG 16:18:49.882 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489529879
  903. DEBUG 16:18:50.113 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489530111
  904. DEBUG 16:18:50.348 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489530343
  905. DEBUG 16:18:50.578 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489530575
  906. DEBUG 16:18:50.810 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489530807
  907. DEBUG 16:18:51.042 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489531039
  908. DEBUG 16:18:51.273 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489531271
  909. DEBUG 16:18:51.505 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489531503
  910. DEBUG 16:18:51.738 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489531735
  911. DEBUG 16:18:51.969 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489531967
  912. DEBUG 16:18:52.202 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489532199
  913. DEBUG 16:18:52.433 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489532431
  914. DEBUG 16:18:52.667 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489532663
  915. DEBUG 16:18:52.899 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489532895
  916. DEBUG 16:18:53.129 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489533127
  917. DEBUG 16:18:53.362 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489533359
  918. DEBUG 16:18:53.590 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489533591
  919. DEBUG 16:18:53.821 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489533819
  920. DEBUG 16:18:54.057 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489534051
  921. DEBUG 16:18:54.286 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489534283
  922. DEBUG 16:18:54.518 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489534515
  923. DEBUG 16:18:54.755 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489534747
  924. DEBUG 16:18:54.981 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489534979
  925. DEBUG 16:18:54.981 22867 aiservice.py e731f6cbae6a Batch for camera test collected, spend 5.562138319015503 sec
  926. INFO 16:18:54.982 22867 aiservice.py e2c6eb8520f5 Start collecting batch for camera test, spend 5.541039228439331 sec
  927. INFO 16:18:54.992 22842 aiservice.py e731f6cbae6a Start processing batch for camera test, spend 5.572863578796387 sec
  928. DEBUG 16:18:54.992 22842 foreman.py e731f6cbae6a Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  929. 24
  930. DEBUG 16:18:54.997 22842 foreman.py e731f6cbae6a Running=0 finished=1
  931. DEBUG 16:18:54.997 22842 foreman.py e731f6cbae6a Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  932. DEBUG 16:18:54.997 22865 command.py e731f6cbae6a Start core.cnn
  933. [TensorRT] INFO: Detecting input data format
  934. [TensorRT] INFO: Dectected data format LCHW
  935. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  936. [TensorRT] INFO: Verifying batches are the expected data type
  937. [TensorRT] INFO: Executing inference
  938. [TensorRT] INFO: Number of Batches: 1
  939. ERROR 16:18:55.059 22865 command.py e731f6cbae6a Worker core.cnn failed
  940. Traceback (most recent call last):
  941. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  942. result = worker.do(batch)
  943. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  944. batch.payload.get("image_idx")
  945. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  946. frames = self.process_frames(images, image_indexes)
  947. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  948. estimated = self.pose_estimator.estimate(images, self.scale_level)
  949. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  950. result = self.process(imgs, scale=scale)
  951. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  952. result.append(self._process(batch, **kwargs))
  953. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  954. l1s, l2s = self.engine.infer(items)
  955. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  956. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  957. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  958. DEBUG 16:18:55.063 22842 foreman.py e731f6cbae6a Running=0 finished=1
  959. CRITICAL 16:18:55.063 22842 foreman.py e731f6cbae6a Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f6d8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  960. CRITICAL 16:18:55.063 22842 foreman.py e731f6cbae6a Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  961. ERROR 16:18:55.063 22842 conclusion.py ------------ No Telegram's chats connected
  962. INFO 16:18:55.063 22842 aiservice.py e731f6cbae6a Batch for camera test processed, spend 0.07116818428039551 sec
  963. DEBUG 16:18:55.064 22842 shared_array.py ------------ Shared memory free 176, used 24
  964. DEBUG 16:18:55.089 22842 shared_array.py ------------ Shared memory free 176, used 24
  965. INFO 16:18:55.089 22842 aiservice.py 5a83a06edcfb Put batch for camera test request to queue, spend 0.00043082237243652344 sec
  966. DEBUG 16:18:55.218 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489535211
  967. DEBUG 16:18:55.445 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489535443
  968. DEBUG 16:18:55.681 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489535675
  969. DEBUG 16:18:55.913 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489535907
  970. DEBUG 16:18:56.145 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489536139
  971. DEBUG 16:18:56.378 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489536371
  972. DEBUG 16:18:56.609 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489536603
  973. DEBUG 16:18:56.841 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489536835
  974. DEBUG 16:18:57.074 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489537067
  975. DEBUG 16:18:57.306 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489537299
  976. DEBUG 16:18:57.537 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489537531
  977. DEBUG 16:18:57.765 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489537763
  978. DEBUG 16:18:58.001 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489537995
  979. DEBUG 16:18:58.229 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489538227
  980. DEBUG 16:18:58.466 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489538459
  981. DEBUG 16:18:58.697 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489538691
  982. DEBUG 16:18:58.930 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489538923
  983. DEBUG 16:18:59.163 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489539155
  984. DEBUG 16:18:59.393 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489539387
  985. DEBUG 16:18:59.626 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489539619
  986. DEBUG 16:18:59.853 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489539851
  987. DEBUG 16:19:00.086 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489540083
  988. DEBUG 16:19:00.320 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489540315
  989. DEBUG 16:19:00.553 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489540547
  990. DEBUG 16:19:00.554 22867 aiservice.py e2c6eb8520f5 Batch for camera test collected, spend 5.572315692901611 sec
  991. INFO 16:19:00.555 22867 aiservice.py 5a83a06edcfb Start collecting batch for camera test, spend 5.466270208358765 sec
  992. INFO 16:19:00.581 22842 aiservice.py e2c6eb8520f5 Start processing batch for camera test, spend 5.598905324935913 sec
  993. DEBUG 16:19:00.581 22842 foreman.py e2c6eb8520f5 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  994. 24
  995. DEBUG 16:19:00.594 22842 foreman.py e2c6eb8520f5 Running=0 finished=1
  996. DEBUG 16:19:00.594 22842 foreman.py e2c6eb8520f5 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  997. DEBUG 16:19:00.596 22865 command.py e2c6eb8520f5 Start core.cnn
  998. [TensorRT] INFO: Detecting input data format
  999. [TensorRT] INFO: Dectected data format LCHW
  1000. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1001. ERROR 16:19:00.665 22865 command.py e2c6eb8520f5 Worker core.cnn failed
  1002. Traceback (most recent call last):
  1003. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1004. result = worker.do(batch)
  1005. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1006. batch.payload.get("image_idx")
  1007. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1008. frames = self.process_frames(images, image_indexes)
  1009. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1010. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1011. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1012. result = self.process(imgs, scale=scale)
  1013. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1014. result.append(self._process(batch, **kwargs))
  1015. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1016. l1s, l2s = self.engine.infer(items)
  1017. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1018. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1019. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1020. [TensorRT] INFO: Verifying batches are the expected data type
  1021. [TensorRT] INFO: Executing inference
  1022. [TensorRT] INFO: Number of Batches: 1
  1023. DEBUG 16:19:00.667 22842 foreman.py e2c6eb8520f5 Running=0 finished=1
  1024. CRITICAL 16:19:00.667 22842 foreman.py e2c6eb8520f5 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f4a8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1025. CRITICAL 16:19:00.667 22842 foreman.py e2c6eb8520f5 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1026. ERROR 16:19:00.668 22842 conclusion.py ------------ No Telegram's chats connected
  1027. INFO 16:19:00.668 22842 aiservice.py e2c6eb8520f5 Batch for camera test processed, spend 0.08694124221801758 sec
  1028. DEBUG 16:19:00.668 22842 shared_array.py ------------ Shared memory free 176, used 24
  1029. DEBUG 16:19:00.727 22842 shared_array.py ------------ Shared memory free 176, used 24
  1030. INFO 16:19:00.727 22842 aiservice.py 4d46a0c33416 Put batch for camera test request to queue, spend 0.0009133815765380859 sec
  1031. DEBUG 16:19:00.785 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489540779
  1032. DEBUG 16:19:01.013 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489541011
  1033. DEBUG 16:19:01.249 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489541243
  1034. DEBUG 16:19:01.480 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489541475
  1035. DEBUG 16:19:01.714 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489541707
  1036. DEBUG 16:19:01.945 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489541939
  1037. DEBUG 16:19:02.178 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489542171
  1038. DEBUG 16:19:02.408 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489542403
  1039. DEBUG 16:19:02.642 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489542635
  1040. DEBUG 16:19:02.870 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489542867
  1041. DEBUG 16:19:03.105 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489543099
  1042. DEBUG 16:19:03.338 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489543331
  1043. DEBUG 16:19:03.569 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489543563
  1044. DEBUG 16:19:03.797 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489543795
  1045. DEBUG 16:19:04.033 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489544027
  1046. DEBUG 16:19:04.266 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489544259
  1047. DEBUG 16:19:04.494 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489544491
  1048. DEBUG 16:19:04.726 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489544723
  1049. DEBUG 16:19:04.962 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489544955
  1050. DEBUG 16:19:05.194 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489545187
  1051. DEBUG 16:19:05.425 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489545419
  1052. DEBUG 16:19:05.657 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489545651
  1053. DEBUG 16:19:05.885 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489545883
  1054. DEBUG 16:19:06.122 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489546115
  1055. DEBUG 16:19:06.124 22867 aiservice.py 5a83a06edcfb Batch for camera test collected, spend 5.568626880645752 sec
  1056. INFO 16:19:06.124 22867 aiservice.py 4d46a0c33416 Start collecting batch for camera test, spend 5.397520065307617 sec
  1057. INFO 16:19:06.166 22842 aiservice.py 5a83a06edcfb Start processing batch for camera test, spend 5.610841751098633 sec
  1058. DEBUG 16:19:06.167 22842 foreman.py 5a83a06edcfb Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1059. DEBUG 16:19:06.181 22842 foreman.py 5a83a06edcfb Running=0 finished=1
  1060. DEBUG 16:19:06.181 22842 foreman.py 5a83a06edcfb Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1061. DEBUG 16:19:06.184 22842 shared_array.py ------------ Shared memory free 152, used 48
  1062. INFO 16:19:06.185 22842 aiservice.py 519d88d5e0d4 Put batch for camera test request to queue, spend 0.0026466846466064453 sec
  1063. 24
  1064. DEBUG 16:19:06.188 22865 command.py 5a83a06edcfb Start core.cnn
  1065. [TensorRT] INFO: Detecting input data format
  1066. [TensorRT] INFO: Dectected data format LCHW
  1067. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1068. [TensorRT] INFO: Verifying batches are the expected data type
  1069. [TensorRT] INFO: Executing inference
  1070. [TensorRT] INFO: Number of Batches: 1
  1071. ERROR 16:19:06.268 22865 command.py 5a83a06edcfb Worker core.cnn failed
  1072. Traceback (most recent call last):
  1073. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1074. result = worker.do(batch)
  1075. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1076. batch.payload.get("image_idx")
  1077. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1078. frames = self.process_frames(images, image_indexes)
  1079. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1080. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1081. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1082. result = self.process(imgs, scale=scale)
  1083. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1084. result.append(self._process(batch, **kwargs))
  1085. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1086. l1s, l2s = self.engine.infer(items)
  1087. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1088. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1089. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1090. DEBUG 16:19:06.275 22842 foreman.py 5a83a06edcfb Running=0 finished=1
  1091. CRITICAL 16:19:06.275 22842 foreman.py 5a83a06edcfb Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f630> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1092. CRITICAL 16:19:06.276 22842 foreman.py 5a83a06edcfb Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1093. ERROR 16:19:06.276 22842 conclusion.py ------------ No Telegram's chats connected
  1094. INFO 16:19:06.276 22842 aiservice.py 5a83a06edcfb Batch for camera test processed, spend 0.1101541519165039 sec
  1095. DEBUG 16:19:06.277 22842 shared_array.py ------------ Shared memory free 152, used 48
  1096. DEBUG 16:19:06.353 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489546347
  1097. DEBUG 16:19:06.581 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489546579
  1098. DEBUG 16:19:06.817 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489546811
  1099. DEBUG 16:19:07.050 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489547043
  1100. DEBUG 16:19:07.281 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489547275
  1101. DEBUG 16:19:07.514 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489547507
  1102. DEBUG 16:19:07.742 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489547739
  1103. DEBUG 16:19:07.977 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489547971
  1104. DEBUG 16:19:08.212 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489548203
  1105. DEBUG 16:19:08.437 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489548435
  1106. DEBUG 16:19:08.675 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489548667
  1107. DEBUG 16:19:08.905 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489548899
  1108. DEBUG 16:19:09.134 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489549131
  1109. DEBUG 16:19:09.369 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489549363
  1110. DEBUG 16:19:09.602 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489549595
  1111. DEBUG 16:19:09.834 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489549827
  1112. DEBUG 16:19:10.066 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489550059
  1113. DEBUG 16:19:10.297 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489550291
  1114. DEBUG 16:19:10.529 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489550523
  1115. DEBUG 16:19:10.762 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489550755
  1116. DEBUG 16:19:10.994 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489550987
  1117. DEBUG 16:19:11.225 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489551219
  1118. DEBUG 16:19:11.458 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489551451
  1119. DEBUG 16:19:11.690 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489551683
  1120. DEBUG 16:19:11.693 22867 aiservice.py 4d46a0c33416 Batch for camera test collected, spend 5.568313360214233 sec
  1121. INFO 16:19:11.693 22867 aiservice.py 519d88d5e0d4 Start collecting batch for camera test, spend 5.511116981506348 sec
  1122. INFO 16:19:11.728 22842 aiservice.py 4d46a0c33416 Start processing batch for camera test, spend 5.603930950164795 sec
  1123. DEBUG 16:19:11.729 22842 foreman.py 4d46a0c33416 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1124. DEBUG 16:19:11.744 22842 foreman.py 4d46a0c33416 Running=0 finished=1
  1125. 24
  1126. DEBUG 16:19:11.744 22842 foreman.py 4d46a0c33416 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1127. DEBUG 16:19:11.746 22865 command.py 4d46a0c33416 Start core.cnn
  1128. [TensorRT] INFO: Detecting input data format
  1129. [TensorRT] INFO: Dectected data format LCHW
  1130. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1131. ERROR 16:19:11.824 22865 command.py 4d46a0c33416 Worker core.cnn failed
  1132. Traceback (most recent call last):
  1133. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1134. result = worker.do(batch)
  1135. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1136. batch.payload.get("image_idx")
  1137. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1138. frames = self.process_frames(images, image_indexes)
  1139. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1140. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1141. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1142. result = self.process(imgs, scale=scale)
  1143. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1144. result.append(self._process(batch, **kwargs))
  1145. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1146. l1s, l2s = self.engine.infer(items)
  1147. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1148. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1149. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1150. [TensorRT] INFO: Verifying batches are the expected data type
  1151. [TensorRT] INFO: Executing inference
  1152. [TensorRT] INFO: Number of Batches: 1
  1153. DEBUG 16:19:11.829 22842 shared_array.py ------------ Shared memory free 152, used 48
  1154. INFO 16:19:11.829 22842 aiservice.py 4372cd30e49d Put batch for camera test request to queue, spend 0.0005295276641845703 sec
  1155. DEBUG 16:19:11.830 22842 foreman.py 4d46a0c33416 Running=0 finished=1
  1156. CRITICAL 16:19:11.830 22842 foreman.py 4d46a0c33416 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f748> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1157. CRITICAL 16:19:11.830 22842 foreman.py 4d46a0c33416 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1158. ERROR 16:19:11.830 22842 conclusion.py ------------ No Telegram's chats connected
  1159. INFO 16:19:11.830 22842 aiservice.py 4d46a0c33416 Batch for camera test processed, spend 0.1019737720489502 sec
  1160. DEBUG 16:19:11.831 22842 shared_array.py ------------ Shared memory free 152, used 48
  1161. DEBUG 16:19:11.918 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489551915
  1162. DEBUG 16:19:12.155 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489552147
  1163. DEBUG 16:19:12.386 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489552379
  1164. DEBUG 16:19:12.618 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489552611
  1165. DEBUG 16:19:12.850 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489552843
  1166. DEBUG 16:19:13.082 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489553075
  1167. DEBUG 16:19:13.314 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489553307
  1168. DEBUG 16:19:13.545 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489553539
  1169. DEBUG 16:19:13.778 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489553771
  1170. DEBUG 16:19:14.010 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489554003
  1171. DEBUG 16:19:14.238 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489554235
  1172. DEBUG 16:19:14.475 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489554467
  1173. DEBUG 16:19:14.707 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489554699
  1174. DEBUG 16:19:14.937 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489554931
  1175. DEBUG 16:19:15.166 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489555163
  1176. DEBUG 16:19:15.402 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489555395
  1177. DEBUG 16:19:15.635 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489555627
  1178. DEBUG 16:19:15.864 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489555859
  1179. DEBUG 16:19:16.094 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489556091
  1180. DEBUG 16:19:16.326 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489556323
  1181. DEBUG 16:19:16.557 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489556555
  1182. DEBUG 16:19:16.790 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489556787
  1183. DEBUG 16:19:17.021 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489557019
  1184. DEBUG 16:19:17.252 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489557251
  1185. DEBUG 16:19:17.253 22867 aiservice.py 519d88d5e0d4 Batch for camera test collected, spend 5.558942079544067 sec
  1186. INFO 16:19:17.254 22867 aiservice.py 4372cd30e49d Start collecting batch for camera test, spend 5.425291538238525 sec
  1187. DEBUG 16:19:17.273 22842 shared_array.py ------------ Shared memory free 152, used 48
  1188. INFO 16:19:17.274 22842 aiservice.py 0ef380873e2b Put batch for camera test request to queue, spend 0.0008943080902099609 sec
  1189. INFO 16:19:17.293 22842 aiservice.py 519d88d5e0d4 Start processing batch for camera test, spend 5.598935842514038 sec
  1190. DEBUG 16:19:17.294 22842 foreman.py 519d88d5e0d4 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1191. DEBUG 16:19:17.297 22842 foreman.py 519d88d5e0d4 Running=0 finished=1
  1192. DEBUG 16:19:17.298 22842 foreman.py 519d88d5e0d4 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1193. DEBUG 16:19:17.298 22865 command.py 519d88d5e0d4 Start core.cnn
  1194. 24
  1195. [TensorRT] INFO: Detecting input data format
  1196. [TensorRT] INFO: Dectected data format LCHW
  1197. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1198. [TensorRT] INFO: Verifying batches are the expected data type
  1199. [TensorRT] INFO: Executing inference
  1200. [TensorRT] INFO: Number of Batches: 1
  1201. ERROR 16:19:17.348 22865 command.py 519d88d5e0d4 Worker core.cnn failed
  1202. Traceback (most recent call last):
  1203. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1204. result = worker.do(batch)
  1205. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1206. batch.payload.get("image_idx")
  1207. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1208. frames = self.process_frames(images, image_indexes)
  1209. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1210. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1211. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1212. result = self.process(imgs, scale=scale)
  1213. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1214. result.append(self._process(batch, **kwargs))
  1215. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1216. l1s, l2s = self.engine.infer(items)
  1217. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1218. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1219. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1220. DEBUG 16:19:17.353 22842 foreman.py 519d88d5e0d4 Running=0 finished=1
  1221. CRITICAL 16:19:17.353 22842 foreman.py 519d88d5e0d4 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f438> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1222. CRITICAL 16:19:17.353 22842 foreman.py 519d88d5e0d4 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1223. ERROR 16:19:17.353 22842 conclusion.py ------------ No Telegram's chats connected
  1224. INFO 16:19:17.353 22842 aiservice.py 519d88d5e0d4 Batch for camera test processed, spend 0.059983253479003906 sec
  1225. DEBUG 16:19:17.354 22842 shared_array.py ------------ Shared memory free 152, used 48
  1226. DEBUG 16:19:17.485 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489557483
  1227. DEBUG 16:19:17.718 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489557715
  1228. DEBUG 16:19:17.950 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489557947
  1229. DEBUG 16:19:18.182 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489558179
  1230. DEBUG 16:19:18.414 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489558411
  1231. DEBUG 16:19:18.645 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489558643
  1232. DEBUG 16:19:18.878 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489558875
  1233. DEBUG 16:19:19.110 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489559107
  1234. DEBUG 16:19:19.343 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489559339
  1235. DEBUG 16:19:19.570 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489559571
  1236. DEBUG 16:19:19.802 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489559803
  1237. DEBUG 16:19:20.034 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489560035
  1238. DEBUG 16:19:20.266 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489560267
  1239. DEBUG 16:19:20.498 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489560499
  1240. DEBUG 16:19:20.734 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489560731
  1241. DEBUG 16:19:20.965 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489560963
  1242. DEBUG 16:19:21.198 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489561195
  1243. DEBUG 16:19:21.430 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489561427
  1244. DEBUG 16:19:21.658 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489561659
  1245. DEBUG 16:19:21.895 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489561891
  1246. DEBUG 16:19:22.125 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489562123
  1247. DEBUG 16:19:22.359 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489562355
  1248. DEBUG 16:19:22.589 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489562587
  1249. DEBUG 16:19:22.823 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489562815
  1250. DEBUG 16:19:22.824 22867 aiservice.py 4372cd30e49d Batch for camera test collected, spend 5.569787979125977 sec
  1251. INFO 16:19:22.825 22867 aiservice.py 0ef380873e2b Start collecting batch for camera test, spend 5.5518457889556885 sec
  1252. INFO 16:19:22.857 22842 aiservice.py 4372cd30e49d Start processing batch for camera test, spend 5.603182554244995 sec
  1253. DEBUG 16:19:22.858 22842 foreman.py 4372cd30e49d Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1254. DEBUG 16:19:22.874 22842 foreman.py 4372cd30e49d Running=0 finished=1
  1255. DEBUG 16:19:22.874 22842 foreman.py 4372cd30e49d Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1256. DEBUG 16:19:22.876 22865 command.py 4372cd30e49d Start core.cnn
  1257. 24
  1258. DEBUG 16:19:22.910 22842 shared_array.py ------------ Shared memory free 152, used 48
  1259. INFO 16:19:22.911 22842 aiservice.py 1ce75f0dcaba Put batch for camera test request to queue, spend 0.0015232563018798828 sec
  1260. [TensorRT] INFO: Detecting input data format
  1261. [TensorRT] INFO: Dectected data format LCHW
  1262. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1263. [TensorRT] INFO: Verifying batches are the expected data type
  1264. [TensorRT] INFO: Executing inference
  1265. [TensorRT] INFO: Number of Batches: 1
  1266. ERROR 16:19:22.957 22865 command.py 4372cd30e49d Worker core.cnn failed
  1267. Traceback (most recent call last):
  1268. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1269. result = worker.do(batch)
  1270. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1271. batch.payload.get("image_idx")
  1272. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1273. frames = self.process_frames(images, image_indexes)
  1274. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1275. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1276. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1277. result = self.process(imgs, scale=scale)
  1278. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1279. result.append(self._process(batch, **kwargs))
  1280. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1281. l1s, l2s = self.engine.infer(items)
  1282. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1283. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1284. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1285. DEBUG 16:19:22.959 22842 foreman.py 4372cd30e49d Running=0 finished=1
  1286. CRITICAL 16:19:22.959 22842 foreman.py 4372cd30e49d Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5f8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1287. CRITICAL 16:19:22.959 22842 foreman.py 4372cd30e49d Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1288. ERROR 16:19:22.959 22842 conclusion.py ------------ No Telegram's chats connected
  1289. INFO 16:19:22.959 22842 aiservice.py 4372cd30e49d Batch for camera test processed, spend 0.10164523124694824 sec
  1290. DEBUG 16:19:22.960 22842 shared_array.py ------------ Shared memory free 152, used 48
  1291. DEBUG 16:19:23.054 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489563047
  1292. DEBUG 16:19:23.287 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489563279
  1293. DEBUG 16:19:23.518 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489563511
  1294. DEBUG 16:19:23.750 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489563743
  1295. DEBUG 16:19:23.982 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489563975
  1296. DEBUG 16:19:24.214 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489564207
  1297. DEBUG 16:19:24.446 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489564439
  1298. DEBUG 16:19:24.678 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489564671
  1299. DEBUG 16:19:24.911 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489564903
  1300. DEBUG 16:19:25.142 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489565135
  1301. DEBUG 16:19:25.374 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489565367
  1302. DEBUG 16:19:25.606 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489565599
  1303. DEBUG 16:19:25.837 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489565831
  1304. DEBUG 16:19:26.070 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489566063
  1305. DEBUG 16:19:26.302 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489566295
  1306. DEBUG 16:19:26.534 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489566527
  1307. DEBUG 16:19:26.766 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489566759
  1308. DEBUG 16:19:26.998 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489566991
  1309. DEBUG 16:19:27.230 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489567223
  1310. DEBUG 16:19:27.462 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489567455
  1311. DEBUG 16:19:27.694 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489567687
  1312. DEBUG 16:19:27.925 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489567919
  1313. DEBUG 16:19:28.155 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489568151
  1314. DEBUG 16:19:28.386 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489568383
  1315. DEBUG 16:19:28.387 22867 aiservice.py 0ef380873e2b Batch for camera test collected, spend 5.5624167919158936 sec
  1316. INFO 16:19:28.388 22867 aiservice.py 1ce75f0dcaba Start collecting batch for camera test, spend 5.4780097007751465 sec
  1317. INFO 16:19:28.420 22842 aiservice.py 0ef380873e2b Start processing batch for camera test, spend 5.595560312271118 sec
  1318. DEBUG 16:19:28.421 22842 foreman.py 0ef380873e2b Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1319. DEBUG 16:19:28.424 22842 foreman.py 0ef380873e2b Running=0 finished=1
  1320. DEBUG 16:19:28.424 22842 foreman.py 0ef380873e2b Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1321. DEBUG 16:19:28.425 22865 command.py 0ef380873e2b Start core.cnn
  1322. 24
  1323. [TensorRT] INFO: Detecting input data format
  1324. [TensorRT] INFO: Dectected data format LCHW
  1325. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1326. [TensorRT] INFO: Verifying batches are the expected data type
  1327. [TensorRT] INFO: Executing inference
  1328. [TensorRT] INFO: Number of Batches: 1
  1329. ERROR 16:19:28.478 22865 command.py 0ef380873e2b Worker core.cnn failed
  1330. Traceback (most recent call last):
  1331. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1332. result = worker.do(batch)
  1333. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1334. batch.payload.get("image_idx")
  1335. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1336. frames = self.process_frames(images, image_indexes)
  1337. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1338. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1339. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1340. result = self.process(imgs, scale=scale)
  1341. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1342. result.append(self._process(batch, **kwargs))
  1343. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1344. l1s, l2s = self.engine.infer(items)
  1345. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1346. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1347. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1348. DEBUG 16:19:28.484 22842 foreman.py 0ef380873e2b Running=0 finished=1
  1349. CRITICAL 16:19:28.485 22842 foreman.py 0ef380873e2b Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f710> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1350. CRITICAL 16:19:28.485 22842 foreman.py 0ef380873e2b Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1351. ERROR 16:19:28.485 22842 conclusion.py ------------ No Telegram's chats connected
  1352. INFO 16:19:28.485 22842 aiservice.py 0ef380873e2b Batch for camera test processed, spend 0.06476974487304688 sec
  1353. DEBUG 16:19:28.486 22842 shared_array.py ------------ Shared memory free 176, used 24
  1354. DEBUG 16:19:28.558 22842 shared_array.py ------------ Shared memory free 176, used 24
  1355. INFO 16:19:28.559 22842 aiservice.py 51b76487f5c8 Put batch for camera test request to queue, spend 0.0010690689086914062 sec
  1356. DEBUG 16:19:28.621 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489568615
  1357. DEBUG 16:19:28.855 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489568847
  1358. DEBUG 16:19:29.083 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489569079
  1359. DEBUG 16:19:29.320 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489569311
  1360. DEBUG 16:19:29.550 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489569543
  1361. DEBUG 16:19:29.782 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489569775
  1362. DEBUG 16:19:30.015 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489570007
  1363. DEBUG 16:19:30.247 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489570239
  1364. DEBUG 16:19:30.478 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489570471
  1365. DEBUG 16:19:30.710 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489570703
  1366. DEBUG 16:19:30.941 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489570935
  1367. DEBUG 16:19:31.174 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489571167
  1368. DEBUG 16:19:31.407 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489571399
  1369. DEBUG 16:19:31.639 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489571631
  1370. DEBUG 16:19:31.870 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489571863
  1371. DEBUG 16:19:32.102 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489572095
  1372. DEBUG 16:19:32.334 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489572327
  1373. DEBUG 16:19:32.566 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489572559
  1374. DEBUG 16:19:32.798 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489572791
  1375. DEBUG 16:19:33.030 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489573023
  1376. DEBUG 16:19:33.262 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489573255
  1377. DEBUG 16:19:33.490 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489573487
  1378. DEBUG 16:19:33.725 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489573719
  1379. DEBUG 16:19:33.958 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489573951
  1380. DEBUG 16:19:33.959 22867 aiservice.py 1ce75f0dcaba Batch for camera test collected, spend 5.570969820022583 sec
  1381. INFO 16:19:33.960 22867 aiservice.py 51b76487f5c8 Start collecting batch for camera test, spend 5.4020116329193115 sec
  1382. DEBUG 16:19:34.006 22842 shared_array.py ------------ Shared memory free 152, used 48
  1383. INFO 16:19:34.007 22842 aiservice.py 34ed58fcadd2 Put batch for camera test request to queue, spend 0.002313375473022461 sec
  1384. INFO 16:19:34.011 22842 aiservice.py 1ce75f0dcaba Start processing batch for camera test, spend 5.623271703720093 sec
  1385. DEBUG 16:19:34.012 22842 foreman.py 1ce75f0dcaba Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1386. DEBUG 16:19:34.024 22842 foreman.py 1ce75f0dcaba Running=0 finished=1
  1387. DEBUG 16:19:34.025 22842 foreman.py 1ce75f0dcaba Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1388. 24
  1389. DEBUG 16:19:34.026 22865 command.py 1ce75f0dcaba Start core.cnn
  1390. [TensorRT] INFO: Detecting input data format
  1391. [TensorRT] INFO: Dectected data format LCHW
  1392. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1393. [TensorRT] INFO: Verifying batches are the expected data type
  1394. [TensorRT] INFO: Executing inference
  1395. [TensorRT] INFO: Number of Batches: 1
  1396. ERROR 16:19:34.089 22865 command.py 1ce75f0dcaba Worker core.cnn failed
  1397. Traceback (most recent call last):
  1398. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1399. result = worker.do(batch)
  1400. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1401. batch.payload.get("image_idx")
  1402. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1403. frames = self.process_frames(images, image_indexes)
  1404. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1405. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1406. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1407. result = self.process(imgs, scale=scale)
  1408. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1409. result.append(self._process(batch, **kwargs))
  1410. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1411. l1s, l2s = self.engine.infer(items)
  1412. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1413. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1414. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1415. DEBUG 16:19:34.098 22842 foreman.py 1ce75f0dcaba Running=0 finished=1
  1416. CRITICAL 16:19:34.098 22842 foreman.py 1ce75f0dcaba Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5c0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1417. CRITICAL 16:19:34.098 22842 foreman.py 1ce75f0dcaba Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1418. ERROR 16:19:34.098 22842 conclusion.py ------------ No Telegram's chats connected
  1419. INFO 16:19:34.098 22842 aiservice.py 1ce75f0dcaba Batch for camera test processed, spend 0.08723282814025879 sec
  1420. DEBUG 16:19:34.099 22842 shared_array.py ------------ Shared memory free 152, used 48
  1421. DEBUG 16:19:34.190 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489574183
  1422. DEBUG 16:19:34.422 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489574415
  1423. DEBUG 16:19:34.653 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489574647
  1424. DEBUG 16:19:34.886 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489574879
  1425. DEBUG 16:19:35.118 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489575111
  1426. DEBUG 16:19:35.350 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489575343
  1427. DEBUG 16:19:35.582 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489575575
  1428. DEBUG 16:19:35.813 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489575807
  1429. DEBUG 16:19:36.046 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489576039
  1430. DEBUG 16:19:36.278 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489576271
  1431. DEBUG 16:19:36.510 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489576503
  1432. DEBUG 16:19:36.743 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489576735
  1433. DEBUG 16:19:36.975 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489576967
  1434. DEBUG 16:19:37.206 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489577199
  1435. DEBUG 16:19:37.437 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489577431
  1436. DEBUG 16:19:37.667 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489577663
  1437. DEBUG 16:19:37.901 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489577895
  1438. DEBUG 16:19:38.135 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489578127
  1439. DEBUG 16:19:38.362 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489578359
  1440. DEBUG 16:19:38.597 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489578591
  1441. DEBUG 16:19:38.830 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489578823
  1442. DEBUG 16:19:39.062 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489579055
  1443. DEBUG 16:19:39.294 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489579287
  1444. DEBUG 16:19:39.526 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489579519
  1445. DEBUG 16:19:39.528 22867 aiservice.py 51b76487f5c8 Batch for camera test collected, spend 5.567601680755615 sec
  1446. INFO 16:19:39.529 22867 aiservice.py 34ed58fcadd2 Start collecting batch for camera test, spend 5.524108409881592 sec
  1447. INFO 16:19:39.556 22842 aiservice.py 51b76487f5c8 Start processing batch for camera test, spend 5.595614433288574 sec
  1448. DEBUG 16:19:39.556 22842 foreman.py 51b76487f5c8 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1449. DEBUG 16:19:39.572 22842 foreman.py 51b76487f5c8 Running=0 finished=1
  1450. DEBUG 16:19:39.572 22842 foreman.py 51b76487f5c8 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1451. DEBUG 16:19:39.574 22865 command.py 51b76487f5c8 Start core.cnn
  1452. 24
  1453. [TensorRT] INFO: Detecting input data format
  1454. [TensorRT] INFO: Dectected data format LCHW
  1455. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1456. [TensorRT] INFO: Verifying batches are the expected data type
  1457. [TensorRT] INFO: Executing inference
  1458. [TensorRT] INFO: Number of Batches: 1
  1459. ERROR 16:19:39.633 22865 command.py 51b76487f5c8 Worker core.cnn failed
  1460. Traceback (most recent call last):
  1461. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1462. result = worker.do(batch)
  1463. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1464. batch.payload.get("image_idx")
  1465. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1466. frames = self.process_frames(images, image_indexes)
  1467. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1468. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1469. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1470. result = self.process(imgs, scale=scale)
  1471. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1472. result.append(self._process(batch, **kwargs))
  1473. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1474. l1s, l2s = self.engine.infer(items)
  1475. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1476. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1477. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1478. DEBUG 16:19:39.635 22842 foreman.py 51b76487f5c8 Running=0 finished=1
  1479. CRITICAL 16:19:39.635 22842 foreman.py 51b76487f5c8 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f4e0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1480. CRITICAL 16:19:39.635 22842 foreman.py 51b76487f5c8 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1481. ERROR 16:19:39.636 22842 conclusion.py ------------ No Telegram's chats connected
  1482. INFO 16:19:39.636 22842 aiservice.py 51b76487f5c8 Batch for camera test processed, spend 0.08015298843383789 sec
  1483. DEBUG 16:19:39.636 22842 shared_array.py ------------ Shared memory free 176, used 24
  1484. DEBUG 16:19:39.651 22842 shared_array.py ------------ Shared memory free 176, used 24
  1485. INFO 16:19:39.652 22842 aiservice.py 293d931ea738 Put batch for camera test request to queue, spend 0.0005486011505126953 sec
  1486. DEBUG 16:19:39.758 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489579751
  1487. DEBUG 16:19:39.991 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489579983
  1488. DEBUG 16:19:40.224 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489580215
  1489. DEBUG 16:19:40.454 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489580447
  1490. DEBUG 16:19:40.686 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489580679
  1491. DEBUG 16:19:40.918 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489580911
  1492. DEBUG 16:19:41.151 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489581143
  1493. DEBUG 16:19:41.382 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489581375
  1494. DEBUG 16:19:41.609 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489581607
  1495. DEBUG 16:19:41.847 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489581839
  1496. DEBUG 16:19:42.078 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489582071
  1497. DEBUG 16:19:42.310 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489582303
  1498. DEBUG 16:19:42.541 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489582535
  1499. DEBUG 16:19:42.770 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489582767
  1500. DEBUG 16:19:43.008 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489582999
  1501. DEBUG 16:19:43.238 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489583231
  1502. DEBUG 16:19:43.470 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489583463
  1503. DEBUG 16:19:43.702 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489583695
  1504. DEBUG 16:19:43.934 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489583927
  1505. DEBUG 16:19:44.167 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489584159
  1506. DEBUG 16:19:44.399 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489584391
  1507. DEBUG 16:19:44.630 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489584623
  1508. DEBUG 16:19:44.862 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489584855
  1509. DEBUG 16:19:45.090 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489585087
  1510. DEBUG 16:19:45.091 22867 aiservice.py 34ed58fcadd2 Batch for camera test collected, spend 5.561596393585205 sec
  1511. INFO 16:19:45.091 22867 aiservice.py 293d931ea738 Start collecting batch for camera test, spend 5.440052270889282 sec
  1512. INFO 16:19:45.152 22842 aiservice.py 34ed58fcadd2 Start processing batch for camera test, spend 5.6228461265563965 sec
  1513. DEBUG 16:19:45.153 22842 foreman.py 34ed58fcadd2 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1514. DEBUG 16:19:45.166 22842 foreman.py 34ed58fcadd2 Running=0 finished=1
  1515. DEBUG 16:19:45.167 22842 foreman.py 34ed58fcadd2 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1516. DEBUG 16:19:45.168 22865 command.py 34ed58fcadd2 Start core.cnn
  1517. 24
  1518. [TensorRT] INFO: Detecting input data format
  1519. [TensorRT] INFO: Dectected data format LCHW
  1520. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1521. [TensorRT] INFO: Verifying batches are the expected data type
  1522. [TensorRT] INFO: Executing inference
  1523. [TensorRT] INFO: Number of Batches: 1
  1524. ERROR 16:19:45.248 22865 command.py 34ed58fcadd2 Worker core.cnn failed
  1525. Traceback (most recent call last):
  1526. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1527. result = worker.do(batch)
  1528. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1529. batch.payload.get("image_idx")
  1530. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1531. frames = self.process_frames(images, image_indexes)
  1532. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1533. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1534. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1535. result = self.process(imgs, scale=scale)
  1536. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1537. result.append(self._process(batch, **kwargs))
  1538. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1539. l1s, l2s = self.engine.infer(items)
  1540. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1541. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1542. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1543. DEBUG 16:19:45.251 22842 foreman.py 34ed58fcadd2 Running=0 finished=1
  1544. CRITICAL 16:19:45.251 22842 foreman.py 34ed58fcadd2 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f668> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1545. CRITICAL 16:19:45.251 22842 foreman.py 34ed58fcadd2 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1546. ERROR 16:19:45.252 22842 conclusion.py ------------ No Telegram's chats connected
  1547. INFO 16:19:45.252 22842 aiservice.py 34ed58fcadd2 Batch for camera test processed, spend 0.09938812255859375 sec
  1548. DEBUG 16:19:45.252 22842 shared_array.py ------------ Shared memory free 176, used 24
  1549. DEBUG 16:19:45.293 22842 shared_array.py ------------ Shared memory free 176, used 24
  1550. INFO 16:19:45.293 22842 aiservice.py f2625d608ff6 Put batch for camera test request to queue, spend 0.00080108642578125 sec
  1551. DEBUG 16:19:45.320 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489585319
  1552. DEBUG 16:19:45.556 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489585551
  1553. DEBUG 16:19:45.786 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489585783
  1554. DEBUG 16:19:46.015 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489586015
  1555. DEBUG 16:19:46.250 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489586248
  1556. DEBUG 16:19:46.482 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489586479
  1557. DEBUG 16:19:46.714 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489586711
  1558. DEBUG 16:19:46.946 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489586943
  1559. DEBUG 16:19:47.178 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489587175
  1560. DEBUG 16:19:47.410 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489587407
  1561. DEBUG 16:19:47.642 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489587639
  1562. DEBUG 16:19:47.874 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489587871
  1563. DEBUG 16:19:48.106 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489588103
  1564. DEBUG 16:19:48.338 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489588335
  1565. DEBUG 16:19:48.570 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489588567
  1566. DEBUG 16:19:48.802 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489588800
  1567. DEBUG 16:19:49.030 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489589031
  1568. DEBUG 16:19:49.262 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489589263
  1569. DEBUG 16:19:49.494 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489589495
  1570. DEBUG 16:19:49.726 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489589728
  1571. DEBUG 16:19:49.962 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489589960
  1572. DEBUG 16:19:50.198 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489590192
  1573. DEBUG 16:19:50.422 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489590423
  1574. DEBUG 16:19:50.654 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489590655
  1575. DEBUG 16:19:50.654 22867 aiservice.py 293d931ea738 Batch for camera test collected, spend 5.562291383743286 sec
  1576. INFO 16:19:50.655 22867 aiservice.py f2625d608ff6 Start collecting batch for camera test, spend 5.362393617630005 sec
  1577. INFO 16:19:50.706 22842 aiservice.py 293d931ea738 Start processing batch for camera test, spend 5.6136314868927 sec
  1578. DEBUG 16:19:50.706 22842 foreman.py 293d931ea738 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1579. DEBUG 16:19:50.713 22842 foreman.py 293d931ea738 Running=0 finished=1
  1580. DEBUG 16:19:50.713 22842 foreman.py 293d931ea738 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1581. DEBUG 16:19:50.714 22865 command.py 293d931ea738 Start core.cnn
  1582. 24
  1583. DEBUG 16:19:50.733 22842 shared_array.py ------------ Shared memory free 152, used 48
  1584. INFO 16:19:50.733 22842 aiservice.py 586e04f5c415 Put batch for camera test request to queue, spend 0.0011677742004394531 sec
  1585. [TensorRT] INFO: Detecting input data format
  1586. [TensorRT] INFO: Dectected data format LCHW
  1587. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1588. [TensorRT] INFO: Verifying batches are the expected data type
  1589. [TensorRT] INFO: Executing inference
  1590. [TensorRT] INFO: Number of Batches: 1
  1591. ERROR 16:19:50.763 22865 command.py 293d931ea738 Worker core.cnn failed
  1592. Traceback (most recent call last):
  1593. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1594. result = worker.do(batch)
  1595. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1596. batch.payload.get("image_idx")
  1597. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1598. frames = self.process_frames(images, image_indexes)
  1599. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1600. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1601. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1602. result = self.process(imgs, scale=scale)
  1603. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1604. result.append(self._process(batch, **kwargs))
  1605. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1606. l1s, l2s = self.engine.infer(items)
  1607. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1608. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1609. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1610. DEBUG 16:19:50.769 22842 foreman.py 293d931ea738 Running=0 finished=1
  1611. CRITICAL 16:19:50.769 22842 foreman.py 293d931ea738 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f6d8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1612. CRITICAL 16:19:50.769 22842 foreman.py 293d931ea738 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1613. ERROR 16:19:50.769 22842 conclusion.py ------------ No Telegram's chats connected
  1614. INFO 16:19:50.770 22842 aiservice.py 293d931ea738 Batch for camera test processed, spend 0.06362557411193848 sec
  1615. DEBUG 16:19:50.770 22842 shared_array.py ------------ Shared memory free 152, used 48
  1616. DEBUG 16:19:50.890 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489590888
  1617. DEBUG 16:19:51.121 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489591120
  1618. DEBUG 16:19:51.350 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489591352
  1619. DEBUG 16:19:51.587 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489591584
  1620. DEBUG 16:19:51.814 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489591812
  1621. DEBUG 16:19:52.050 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489592044
  1622. DEBUG 16:19:52.282 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489592276
  1623. DEBUG 16:19:52.515 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489592508
  1624. DEBUG 16:19:52.742 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489592740
  1625. DEBUG 16:19:52.976 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489592972
  1626. DEBUG 16:19:53.206 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489593204
  1627. DEBUG 16:19:53.439 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489593436
  1628. DEBUG 16:19:53.672 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489593668
  1629. DEBUG 16:19:53.906 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489593900
  1630. DEBUG 16:19:54.138 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489594132
  1631. DEBUG 16:19:54.366 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489594364
  1632. DEBUG 16:19:54.602 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489594596
  1633. DEBUG 16:19:54.834 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489594828
  1634. DEBUG 16:19:55.061 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489595060
  1635. DEBUG 16:19:55.295 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489595292
  1636. DEBUG 16:19:55.529 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489595524
  1637. DEBUG 16:19:55.758 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489595756
  1638. DEBUG 16:19:55.991 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489595988
  1639. DEBUG 16:19:56.226 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489596220
  1640. DEBUG 16:19:56.227 22867 aiservice.py f2625d608ff6 Batch for camera test collected, spend 5.57224702835083 sec
  1641. INFO 16:19:56.228 22867 aiservice.py 586e04f5c415 Start collecting batch for camera test, spend 5.496081352233887 sec
  1642. INFO 16:19:56.261 22842 aiservice.py f2625d608ff6 Start processing batch for camera test, spend 5.606155872344971 sec
  1643. DEBUG 16:19:56.262 22842 foreman.py f2625d608ff6 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1644. 24
  1645. DEBUG 16:19:56.278 22842 foreman.py f2625d608ff6 Running=0 finished=1
  1646. DEBUG 16:19:56.278 22842 foreman.py f2625d608ff6 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1647. DEBUG 16:19:56.280 22865 command.py f2625d608ff6 Start core.cnn
  1648. [TensorRT] INFO: Detecting input data format
  1649. [TensorRT] INFO: Dectected data format LCHW
  1650. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1651. [TensorRT] INFO: Verifying batches are the expected data type
  1652. [TensorRT] INFO: Executing inference
  1653. [TensorRT] INFO: Number of Batches: 1
  1654. ERROR 16:19:56.348 22865 command.py f2625d608ff6 Worker core.cnn failed
  1655. Traceback (most recent call last):
  1656. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1657. result = worker.do(batch)
  1658. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1659. batch.payload.get("image_idx")
  1660. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1661. frames = self.process_frames(images, image_indexes)
  1662. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1663. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1664. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1665. result = self.process(imgs, scale=scale)
  1666. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1667. result.append(self._process(batch, **kwargs))
  1668. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1669. l1s, l2s = self.engine.infer(items)
  1670. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1671. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1672. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1673. DEBUG 16:19:56.351 22842 foreman.py f2625d608ff6 Running=0 finished=1
  1674. CRITICAL 16:19:56.351 22842 foreman.py f2625d608ff6 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f588> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1675. CRITICAL 16:19:56.351 22842 foreman.py f2625d608ff6 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1676. ERROR 16:19:56.352 22842 conclusion.py ------------ No Telegram's chats connected
  1677. INFO 16:19:56.352 22842 aiservice.py f2625d608ff6 Batch for camera test processed, spend 0.09092140197753906 sec
  1678. DEBUG 16:19:56.353 22842 shared_array.py ------------ Shared memory free 176, used 24
  1679. DEBUG 16:19:56.376 22842 shared_array.py ------------ Shared memory free 176, used 24
  1680. INFO 16:19:56.376 22842 aiservice.py 76f79e2c8ef8 Put batch for camera test request to queue, spend 0.000518798828125 sec
  1681. DEBUG 16:19:56.458 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489596452
  1682. DEBUG 16:19:56.690 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489596684
  1683. DEBUG 16:19:56.921 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489596916
  1684. DEBUG 16:19:57.150 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489597148
  1685. DEBUG 16:19:57.386 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489597380
  1686. DEBUG 16:19:57.618 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489597612
  1687. DEBUG 16:19:57.852 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489597844
  1688. DEBUG 16:19:58.082 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489598076
  1689. DEBUG 16:19:58.311 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489598308
  1690. DEBUG 16:19:58.543 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489598540
  1691. DEBUG 16:19:58.781 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489598772
  1692. DEBUG 16:19:59.006 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489599004
  1693. DEBUG 16:19:59.238 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489599236
  1694. DEBUG 16:19:59.472 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489599468
  1695. DEBUG 16:19:59.702 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489599700
  1696. DEBUG 16:19:59.934 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489599932
  1697. DEBUG 16:20:00.170 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489600164
  1698. DEBUG 16:20:00.399 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489600396
  1699. DEBUG 16:20:00.630 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489600628
  1700. DEBUG 16:20:00.862 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489600860
  1701. DEBUG 16:20:01.099 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489601092
  1702. DEBUG 16:20:01.331 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489601324
  1703. DEBUG 16:20:01.562 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489601556
  1704. DEBUG 16:20:01.795 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489601788
  1705. DEBUG 16:20:01.797 22867 aiservice.py 586e04f5c415 Batch for camera test collected, spend 5.568392992019653 sec
  1706. INFO 16:20:01.798 22867 aiservice.py 76f79e2c8ef8 Start collecting batch for camera test, spend 5.422960519790649 sec
  1707. INFO 16:20:01.802 22842 aiservice.py 586e04f5c415 Start processing batch for camera test, spend 5.572845220565796 sec
  1708. DEBUG 16:20:01.802 22842 foreman.py 586e04f5c415 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1709. DEBUG 16:20:01.815 22842 foreman.py 586e04f5c415 Running=0 finished=1
  1710. DEBUG 16:20:01.815 22842 foreman.py 586e04f5c415 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1711. DEBUG 16:20:01.816 22842 shared_array.py ------------ Shared memory free 152, used 48
  1712. INFO 16:20:01.817 22842 aiservice.py 1ba10e18774a Put batch for camera test request to queue, spend 0.0012590885162353516 sec
  1713. 24
  1714. DEBUG 16:20:01.818 22865 command.py 586e04f5c415 Start core.cnn
  1715. [TensorRT] INFO: Detecting input data format
  1716. [TensorRT] INFO: Dectected data format LCHW
  1717. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1718. [TensorRT] INFO: Verifying batches are the expected data type
  1719. [TensorRT] INFO: Executing inference
  1720. [TensorRT] INFO: Number of Batches: 1
  1721. ERROR 16:20:01.873 22865 command.py 586e04f5c415 Worker core.cnn failed
  1722. Traceback (most recent call last):
  1723. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1724. result = worker.do(batch)
  1725. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1726. batch.payload.get("image_idx")
  1727. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1728. frames = self.process_frames(images, image_indexes)
  1729. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1730. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1731. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1732. result = self.process(imgs, scale=scale)
  1733. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1734. result.append(self._process(batch, **kwargs))
  1735. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1736. l1s, l2s = self.engine.infer(items)
  1737. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1738. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1739. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1740. DEBUG 16:20:01.877 22842 foreman.py 586e04f5c415 Running=0 finished=1
  1741. CRITICAL 16:20:01.877 22842 foreman.py 586e04f5c415 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f630> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1742. CRITICAL 16:20:01.877 22842 foreman.py 586e04f5c415 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1743. ERROR 16:20:01.878 22842 conclusion.py ------------ No Telegram's chats connected
  1744. INFO 16:20:01.878 22842 aiservice.py 586e04f5c415 Batch for camera test processed, spend 0.07609319686889648 sec
  1745. DEBUG 16:20:01.878 22842 shared_array.py ------------ Shared memory free 152, used 48
  1746. DEBUG 16:20:02.026 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489602020
  1747. DEBUG 16:20:02.258 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489602252
  1748. DEBUG 16:20:02.490 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489602484
  1749. DEBUG 16:20:02.722 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489602716
  1750. DEBUG 16:20:02.955 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489602948
  1751. DEBUG 16:20:03.182 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489603180
  1752. DEBUG 16:20:03.417 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489603412
  1753. DEBUG 16:20:03.650 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489603644
  1754. DEBUG 16:20:03.882 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489603876
  1755. DEBUG 16:20:04.115 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489604108
  1756. DEBUG 16:20:04.344 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489604340
  1757. DEBUG 16:20:04.574 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489604572
  1758. DEBUG 16:20:04.807 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489604804
  1759. DEBUG 16:20:05.038 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489605036
  1760. DEBUG 16:20:05.270 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489605268
  1761. DEBUG 16:20:05.502 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489605500
  1762. DEBUG 16:20:05.734 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489605732
  1763. DEBUG 16:20:05.966 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489605964
  1764. DEBUG 16:20:06.198 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489606196
  1765. DEBUG 16:20:06.430 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489606428
  1766. DEBUG 16:20:06.662 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489606660
  1767. DEBUG 16:20:06.894 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489606892
  1768. DEBUG 16:20:07.126 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489607124
  1769. DEBUG 16:20:07.358 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489607356
  1770. DEBUG 16:20:07.359 22867 aiservice.py 76f79e2c8ef8 Batch for camera test collected, spend 5.560208559036255 sec
  1771. INFO 16:20:07.359 22867 aiservice.py 1ba10e18774a Start collecting batch for camera test, spend 5.543436765670776 sec
  1772. INFO 16:20:07.388 22842 aiservice.py 76f79e2c8ef8 Start processing batch for camera test, spend 5.589170932769775 sec
  1773. DEBUG 16:20:07.388 22842 foreman.py 76f79e2c8ef8 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1774. DEBUG 16:20:07.392 22842 foreman.py 76f79e2c8ef8 Running=0 finished=1
  1775. DEBUG 16:20:07.392 22842 foreman.py 76f79e2c8ef8 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1776. DEBUG 16:20:07.393 22865 command.py 76f79e2c8ef8 Start core.cnn
  1777. 24
  1778. [TensorRT] INFO: Detecting input data format
  1779. [TensorRT] INFO: Dectected data format LCHW
  1780. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1781. [TensorRT] INFO: Verifying batches are the expected data type
  1782. [TensorRT] INFO: Executing inference
  1783. [TensorRT] INFO: Number of Batches: 1
  1784. DEBUG 16:20:07.452 22842 shared_array.py ------------ Shared memory free 152, used 48
  1785. ERROR 16:20:07.453 22865 command.py 76f79e2c8ef8 Worker core.cnn failed
  1786. Traceback (most recent call last):
  1787. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1788. result = worker.do(batch)
  1789. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1790. batch.payload.get("image_idx")
  1791. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1792. frames = self.process_frames(images, image_indexes)
  1793. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1794. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1795. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1796. result = self.process(imgs, scale=scale)
  1797. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1798. result.append(self._process(batch, **kwargs))
  1799. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1800. l1s, l2s = self.engine.infer(items)
  1801. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1802. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1803. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1804. INFO 16:20:07.453 22842 aiservice.py a9774a550311 Put batch for camera test request to queue, spend 0.000843048095703125 sec
  1805. DEBUG 16:20:07.456 22842 foreman.py 76f79e2c8ef8 Running=0 finished=1
  1806. CRITICAL 16:20:07.456 22842 foreman.py 76f79e2c8ef8 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f748> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1807. CRITICAL 16:20:07.456 22842 foreman.py 76f79e2c8ef8 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1808. ERROR 16:20:07.456 22842 conclusion.py ------------ No Telegram's chats connected
  1809. INFO 16:20:07.457 22842 aiservice.py 76f79e2c8ef8 Batch for camera test processed, spend 0.06865334510803223 sec
  1810. DEBUG 16:20:07.457 22842 shared_array.py ------------ Shared memory free 152, used 48
  1811. DEBUG 16:20:07.591 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489607588
  1812. DEBUG 16:20:07.823 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489607820
  1813. DEBUG 16:20:08.054 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489608052
  1814. DEBUG 16:20:08.292 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489608284
  1815. DEBUG 16:20:08.518 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489608516
  1816. DEBUG 16:20:08.751 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489608748
  1817. DEBUG 16:20:08.982 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489608980
  1818. DEBUG 16:20:09.214 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489609212
  1819. DEBUG 16:20:09.450 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489609444
  1820. DEBUG 16:20:09.679 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489609676
  1821. DEBUG 16:20:09.913 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489609908
  1822. DEBUG 16:20:10.145 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489610140
  1823. DEBUG 16:20:10.378 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489610372
  1824. DEBUG 16:20:10.610 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489610604
  1825. DEBUG 16:20:10.840 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489610836
  1826. DEBUG 16:20:11.074 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489611068
  1827. DEBUG 16:20:11.302 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489611300
  1828. DEBUG 16:20:11.536 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489611532
  1829. DEBUG 16:20:11.772 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489611764
  1830. DEBUG 16:20:12.005 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489611996
  1831. DEBUG 16:20:12.234 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489612228
  1832. DEBUG 16:20:12.466 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489612460
  1833. DEBUG 16:20:12.700 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489612692
  1834. DEBUG 16:20:12.930 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489612924
  1835. DEBUG 16:20:12.932 22867 aiservice.py 1ba10e18774a Batch for camera test collected, spend 5.572662830352783 sec
  1836. INFO 16:20:12.933 22867 aiservice.py a9774a550311 Start collecting batch for camera test, spend 5.480820655822754 sec
  1837. INFO 16:20:12.975 22842 aiservice.py 1ba10e18774a Start processing batch for camera test, spend 5.615530967712402 sec
  1838. DEBUG 16:20:12.976 22842 foreman.py 1ba10e18774a Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1839. DEBUG 16:20:12.990 22842 foreman.py 1ba10e18774a Running=0 finished=1
  1840. DEBUG 16:20:12.990 22842 foreman.py 1ba10e18774a Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1841. DEBUG 16:20:12.992 22865 command.py 1ba10e18774a Start core.cnn
  1842. 24
  1843. [TensorRT] INFO: Detecting input data format
  1844. [TensorRT] INFO: Dectected data format LCHW
  1845. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1846. [TensorRT] INFO: Verifying batches are the expected data type
  1847. [TensorRT] INFO: Executing inference
  1848. [TensorRT] INFO: Number of Batches: 1
  1849. ERROR 16:20:13.059 22865 command.py 1ba10e18774a Worker core.cnn failed
  1850. Traceback (most recent call last):
  1851. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1852. result = worker.do(batch)
  1853. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1854. batch.payload.get("image_idx")
  1855. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1856. frames = self.process_frames(images, image_indexes)
  1857. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1858. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1859. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1860. result = self.process(imgs, scale=scale)
  1861. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1862. result.append(self._process(batch, **kwargs))
  1863. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1864. l1s, l2s = self.engine.infer(items)
  1865. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1866. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1867. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1868. DEBUG 16:20:13.063 22842 foreman.py 1ba10e18774a Running=0 finished=1
  1869. CRITICAL 16:20:13.064 22842 foreman.py 1ba10e18774a Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f438> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1870. CRITICAL 16:20:13.064 22842 foreman.py 1ba10e18774a Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1871. ERROR 16:20:13.064 22842 conclusion.py ------------ No Telegram's chats connected
  1872. INFO 16:20:13.064 22842 aiservice.py 1ba10e18774a Batch for camera test processed, spend 0.08914780616760254 sec
  1873. DEBUG 16:20:13.065 22842 shared_array.py ------------ Shared memory free 176, used 24
  1874. DEBUG 16:20:13.102 22842 shared_array.py ------------ Shared memory free 176, used 24
  1875. INFO 16:20:13.102 22842 aiservice.py 733d5d959c13 Put batch for camera test request to queue, spend 0.0004458427429199219 sec
  1876. DEBUG 16:20:13.162 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489613156
  1877. DEBUG 16:20:13.394 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489613388
  1878. DEBUG 16:20:13.626 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489613620
  1879. DEBUG 16:20:13.858 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489613852
  1880. DEBUG 16:20:14.091 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489614084
  1881. DEBUG 16:20:14.322 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489614316
  1882. DEBUG 16:20:14.550 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489614548
  1883. DEBUG 16:20:14.782 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489614780
  1884. DEBUG 16:20:15.015 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489615012
  1885. DEBUG 16:20:15.246 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489615244
  1886. DEBUG 16:20:15.478 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489615476
  1887. DEBUG 16:20:15.710 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489615708
  1888. DEBUG 16:20:15.938 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489615940
  1889. DEBUG 16:20:16.174 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489616172
  1890. DEBUG 16:20:16.406 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489616404
  1891. DEBUG 16:20:16.639 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489616636
  1892. DEBUG 16:20:16.866 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489616868
  1893. DEBUG 16:20:17.103 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489617100
  1894. DEBUG 16:20:17.331 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489617332
  1895. DEBUG 16:20:17.562 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489617564
  1896. DEBUG 16:20:17.798 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489617796
  1897. DEBUG 16:20:18.030 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489618028
  1898. DEBUG 16:20:18.262 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489618260
  1899. DEBUG 16:20:18.494 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489618492
  1900. DEBUG 16:20:18.495 22867 aiservice.py a9774a550311 Batch for camera test collected, spend 5.561984539031982 sec
  1901. INFO 16:20:18.496 22867 aiservice.py 733d5d959c13 Start collecting batch for camera test, spend 5.394415855407715 sec
  1902. INFO 16:20:18.518 22842 aiservice.py a9774a550311 Start processing batch for camera test, spend 5.5850160121917725 sec
  1903. DEBUG 16:20:18.519 22842 foreman.py a9774a550311 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1904. 24
  1905. DEBUG 16:20:18.532 22842 foreman.py a9774a550311 Running=0 finished=1
  1906. DEBUG 16:20:18.533 22842 foreman.py a9774a550311 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1907. DEBUG 16:20:18.534 22865 command.py a9774a550311 Start core.cnn
  1908. DEBUG 16:20:18.552 22842 shared_array.py ------------ Shared memory free 152, used 48
  1909. INFO 16:20:18.553 22842 aiservice.py bdf88a3ab7fa Put batch for camera test request to queue, spend 0.00386810302734375 sec
  1910. [TensorRT] INFO: Detecting input data format
  1911. [TensorRT] INFO: Dectected data format LCHW
  1912. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1913. [TensorRT] INFO: Verifying batches are the expected data type
  1914. [TensorRT] INFO: Executing inference
  1915. [TensorRT] INFO: Number of Batches: 1
  1916. ERROR 16:20:18.610 22865 command.py a9774a550311 Worker core.cnn failed
  1917. Traceback (most recent call last):
  1918. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1919. result = worker.do(batch)
  1920. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1921. batch.payload.get("image_idx")
  1922. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1923. frames = self.process_frames(images, image_indexes)
  1924. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1925. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1926. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1927. result = self.process(imgs, scale=scale)
  1928. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1929. result.append(self._process(batch, **kwargs))
  1930. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1931. l1s, l2s = self.engine.infer(items)
  1932. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1933. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1934. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1935. DEBUG 16:20:18.618 22842 foreman.py a9774a550311 Running=0 finished=1
  1936. CRITICAL 16:20:18.618 22842 foreman.py a9774a550311 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5f8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1937. CRITICAL 16:20:18.618 22842 foreman.py a9774a550311 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  1938. ERROR 16:20:18.619 22842 conclusion.py ------------ No Telegram's chats connected
  1939. INFO 16:20:18.619 22842 aiservice.py a9774a550311 Batch for camera test processed, spend 0.10026192665100098 sec
  1940. DEBUG 16:20:18.619 22842 shared_array.py ------------ Shared memory free 152, used 48
  1941. DEBUG 16:20:18.722 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489618724
  1942. DEBUG 16:20:18.954 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489618956
  1943. DEBUG 16:20:19.186 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489619188
  1944. DEBUG 16:20:19.418 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489619420
  1945. DEBUG 16:20:19.651 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489619652
  1946. DEBUG 16:20:19.882 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489619884
  1947. DEBUG 16:20:20.118 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489620116
  1948. DEBUG 16:20:20.351 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489620348
  1949. DEBUG 16:20:20.578 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489620580
  1950. DEBUG 16:20:20.816 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489620808
  1951. DEBUG 16:20:21.048 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489621040
  1952. DEBUG 16:20:21.278 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489621272
  1953. DEBUG 16:20:21.511 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489621504
  1954. DEBUG 16:20:21.742 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489621736
  1955. DEBUG 16:20:21.974 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489621968
  1956. DEBUG 16:20:22.206 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489622200
  1957. DEBUG 16:20:22.439 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489622432
  1958. DEBUG 16:20:22.670 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489622664
  1959. DEBUG 16:20:22.903 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489622896
  1960. DEBUG 16:20:23.130 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489623128
  1961. DEBUG 16:20:23.363 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489623360
  1962. DEBUG 16:20:23.598 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489623592
  1963. DEBUG 16:20:23.830 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489623824
  1964. DEBUG 16:20:24.058 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489624056
  1965. DEBUG 16:20:24.058 22867 aiservice.py 733d5d959c13 Batch for camera test collected, spend 5.562495231628418 sec
  1966. INFO 16:20:24.059 22867 aiservice.py bdf88a3ab7fa Start collecting batch for camera test, spend 5.509286165237427 sec
  1967. INFO 16:20:24.079 22842 aiservice.py 733d5d959c13 Start processing batch for camera test, spend 5.583444595336914 sec
  1968. DEBUG 16:20:24.080 22842 foreman.py 733d5d959c13 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  1969. DEBUG 16:20:24.083 22842 foreman.py 733d5d959c13 Running=0 finished=1
  1970. DEBUG 16:20:24.083 22842 foreman.py 733d5d959c13 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  1971. DEBUG 16:20:24.084 22865 command.py 733d5d959c13 Start core.cnn
  1972. 24
  1973. [TensorRT] INFO: Detecting input data format
  1974. [TensorRT] INFO: Dectected data format LCHW
  1975. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  1976. [TensorRT] INFO: Verifying batches are the expected data type
  1977. [TensorRT] INFO: Executing inference
  1978. [TensorRT] INFO: Number of Batches: 1
  1979. ERROR 16:20:24.138 22865 command.py 733d5d959c13 Worker core.cnn failed
  1980. Traceback (most recent call last):
  1981. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  1982. result = worker.do(batch)
  1983. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  1984. batch.payload.get("image_idx")
  1985. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  1986. frames = self.process_frames(images, image_indexes)
  1987. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  1988. estimated = self.pose_estimator.estimate(images, self.scale_level)
  1989. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  1990. result = self.process(imgs, scale=scale)
  1991. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  1992. result.append(self._process(batch, **kwargs))
  1993. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  1994. l1s, l2s = self.engine.infer(items)
  1995. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  1996. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  1997. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  1998. DEBUG 16:20:24.141 22842 foreman.py 733d5d959c13 Running=0 finished=1
  1999. CRITICAL 16:20:24.141 22842 foreman.py 733d5d959c13 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f710> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2000. CRITICAL 16:20:24.141 22842 foreman.py 733d5d959c13 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2001. ERROR 16:20:24.142 22842 conclusion.py ------------ No Telegram's chats connected
  2002. INFO 16:20:24.142 22842 aiservice.py 733d5d959c13 Batch for camera test processed, spend 0.062163591384887695 sec
  2003. DEBUG 16:20:24.142 22842 shared_array.py ------------ Shared memory free 176, used 24
  2004. DEBUG 16:20:24.190 22842 shared_array.py ------------ Shared memory free 176, used 24
  2005. INFO 16:20:24.190 22842 aiservice.py f222ddf34a4c Put batch for camera test request to queue, spend 0.0005888938903808594 sec
  2006. DEBUG 16:20:24.294 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489624288
  2007. DEBUG 16:20:24.526 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489624520
  2008. DEBUG 16:20:24.758 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489624752
  2009. DEBUG 16:20:24.990 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489624984
  2010. DEBUG 16:20:25.218 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489625216
  2011. DEBUG 16:20:25.450 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489625448
  2012. DEBUG 16:20:25.682 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489625680
  2013. DEBUG 16:20:25.918 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489625912
  2014. DEBUG 16:20:26.146 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489626144
  2015. DEBUG 16:20:26.583 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489626576
  2016. DEBUG 16:20:27.046 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489627040
  2017. DEBUG 16:20:27.506 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489627504
  2018. DEBUG 16:20:27.970 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489627968
  2019. DEBUG 16:20:28.434 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489628432
  2020. DEBUG 16:20:28.898 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489628896
  2021. DEBUG 16:20:29.362 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489629360
  2022. DEBUG 16:20:29.827 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489629824
  2023. DEBUG 16:20:30.294 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489630288
  2024. DEBUG 16:20:30.758 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489630752
  2025. DEBUG 16:20:31.222 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489631216
  2026. DEBUG 16:20:31.686 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489631680
  2027. DEBUG 16:20:32.150 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489632144
  2028. DEBUG 16:20:32.614 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489632608
  2029. DEBUG 16:20:33.079 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489633072
  2030. DEBUG 16:20:33.081 22867 aiservice.py bdf88a3ab7fa Batch for camera test collected, spend 9.02200722694397 sec
  2031. INFO 16:20:33.081 22867 aiservice.py f222ddf34a4c Start collecting batch for camera test, spend 8.891964435577393 sec
  2032. INFO 16:20:33.112 22842 aiservice.py bdf88a3ab7fa Start processing batch for camera test, spend 9.052941083908081 sec
  2033. DEBUG 16:20:33.112 22842 foreman.py bdf88a3ab7fa Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2034. DEBUG 16:20:33.125 22842 foreman.py bdf88a3ab7fa Running=0 finished=1
  2035. DEBUG 16:20:33.125 22842 foreman.py bdf88a3ab7fa Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2036. DEBUG 16:20:33.127 22865 command.py bdf88a3ab7fa Start core.cnn
  2037. 24
  2038. [TensorRT] INFO: Detecting input data format
  2039. [TensorRT] INFO: Dectected data format LCHW
  2040. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2041. [TensorRT] INFO: Verifying batches are the expected data type
  2042. [TensorRT] INFO: Executing inference
  2043. [TensorRT] INFO: Number of Batches: 1
  2044. ERROR 16:20:33.204 22865 command.py bdf88a3ab7fa Worker core.cnn failed
  2045. Traceback (most recent call last):
  2046. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2047. result = worker.do(batch)
  2048. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2049. batch.payload.get("image_idx")
  2050. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2051. frames = self.process_frames(images, image_indexes)
  2052. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2053. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2054. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2055. result = self.process(imgs, scale=scale)
  2056. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2057. result.append(self._process(batch, **kwargs))
  2058. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2059. l1s, l2s = self.engine.infer(items)
  2060. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2061. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2062. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2063. DEBUG 16:20:33.209 22842 foreman.py bdf88a3ab7fa Running=0 finished=1
  2064. CRITICAL 16:20:33.209 22842 foreman.py bdf88a3ab7fa Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5c0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2065. CRITICAL 16:20:33.209 22842 foreman.py bdf88a3ab7fa Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2066. ERROR 16:20:33.210 22842 conclusion.py ------------ No Telegram's chats connected
  2067. INFO 16:20:33.210 22842 aiservice.py bdf88a3ab7fa Batch for camera test processed, spend 0.09783577919006348 sec
  2068. DEBUG 16:20:33.210 22842 shared_array.py ------------ Shared memory free 176, used 24
  2069. DEBUG 16:20:33.252 22842 shared_array.py ------------ Shared memory free 176, used 24
  2070. INFO 16:20:33.252 22842 aiservice.py 06085efc57dc Put batch for camera test request to queue, spend 0.0005548000335693359 sec
  2071. DEBUG 16:20:33.541 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489633536
  2072. DEBUG 16:20:34.002 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489634000
  2073. DEBUG 16:20:34.466 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489634464
  2074. DEBUG 16:20:34.930 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489634928
  2075. DEBUG 16:20:35.394 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489635392
  2076. DEBUG 16:20:35.858 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489635856
  2077. DEBUG 16:20:36.322 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489636320
  2078. DEBUG 16:20:36.788 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489636784
  2079. DEBUG 16:20:37.251 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489637248
  2080. DEBUG 16:20:37.714 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489637712
  2081. DEBUG 16:20:38.178 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489638176
  2082. DEBUG 16:20:38.642 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489638640
  2083. DEBUG 16:20:39.106 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489639104
  2084. DEBUG 16:20:39.571 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489639568
  2085. DEBUG 16:20:40.034 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489640028
  2086. DEBUG 16:20:40.499 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489640492
  2087. DEBUG 16:20:40.962 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489640956
  2088. DEBUG 16:20:41.423 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489641420
  2089. DEBUG 16:20:41.886 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489641884
  2090. DEBUG 16:20:42.351 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489642348
  2091. DEBUG 16:20:42.819 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489642812
  2092. DEBUG 16:20:43.284 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489643276
  2093. DEBUG 16:20:43.746 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489643740
  2094. DEBUG 16:20:44.210 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489644204
  2095. DEBUG 16:20:44.211 22867 aiservice.py f222ddf34a4c Batch for camera test collected, spend 11.129513263702393 sec
  2096. INFO 16:20:44.211 22867 aiservice.py 06085efc57dc Start collecting batch for camera test, spend 10.959887027740479 sec
  2097. INFO 16:20:44.229 22842 aiservice.py f222ddf34a4c Start processing batch for camera test, spend 11.147498369216919 sec
  2098. DEBUG 16:20:44.230 22842 foreman.py f222ddf34a4c Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2099. DEBUG 16:20:44.242 22842 foreman.py f222ddf34a4c Running=0 finished=1
  2100. DEBUG 16:20:44.243 22842 foreman.py f222ddf34a4c Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2101. DEBUG 16:20:44.245 22865 command.py f222ddf34a4c Start core.cnn
  2102. 24
  2103. [TensorRT] INFO: Detecting input data format
  2104. [TensorRT] INFO: Dectected data format LCHW
  2105. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2106. [TensorRT] INFO: Verifying batches are the expected data type
  2107. [TensorRT] INFO: Executing inference
  2108. [TensorRT] INFO: Number of Batches: 1
  2109. ERROR 16:20:44.324 22865 command.py f222ddf34a4c Worker core.cnn failed
  2110. Traceback (most recent call last):
  2111. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2112. result = worker.do(batch)
  2113. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2114. batch.payload.get("image_idx")
  2115. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2116. frames = self.process_frames(images, image_indexes)
  2117. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2118. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2119. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2120. result = self.process(imgs, scale=scale)
  2121. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2122. result.append(self._process(batch, **kwargs))
  2123. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2124. l1s, l2s = self.engine.infer(items)
  2125. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2126. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2127. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2128. DEBUG 16:20:44.327 22842 foreman.py f222ddf34a4c Running=0 finished=1
  2129. CRITICAL 16:20:44.327 22842 foreman.py f222ddf34a4c Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f4e0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2130. CRITICAL 16:20:44.327 22842 foreman.py f222ddf34a4c Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2131. ERROR 16:20:44.328 22842 conclusion.py ------------ No Telegram's chats connected
  2132. INFO 16:20:44.328 22842 aiservice.py f222ddf34a4c Batch for camera test processed, spend 0.09878873825073242 sec
  2133. DEBUG 16:20:44.328 22842 shared_array.py ------------ Shared memory free 176, used 24
  2134. DEBUG 16:20:44.335 22842 shared_array.py ------------ Shared memory free 176, used 24
  2135. INFO 16:20:44.335 22842 aiservice.py f374c07dd86e Put batch for camera test request to queue, spend 0.00037217140197753906 sec
  2136. DEBUG 16:20:44.674 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489644668
  2137. DEBUG 16:20:45.134 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489645132
  2138. DEBUG 16:20:45.598 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489645596
  2139. DEBUG 16:20:46.062 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489646060
  2140. DEBUG 16:20:46.527 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489646524
  2141. DEBUG 16:20:46.996 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489646988
  2142. DEBUG 16:20:47.460 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489647452
  2143. DEBUG 16:20:47.922 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489647916
  2144. DEBUG 16:20:48.386 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489648380
  2145. DEBUG 16:20:48.850 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489648844
  2146. DEBUG 16:20:49.314 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489649308
  2147. DEBUG 16:20:49.774 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489649772
  2148. DEBUG 16:20:50.238 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489650236
  2149. DEBUG 16:20:50.702 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489650700
  2150. DEBUG 16:20:51.166 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489651164
  2151. DEBUG 16:20:51.630 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489651628
  2152. DEBUG 16:20:52.098 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489652092
  2153. DEBUG 16:20:52.562 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489652556
  2154. DEBUG 16:20:53.026 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489653020
  2155. DEBUG 16:20:53.490 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489653484
  2156. DEBUG 16:20:53.954 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489653948
  2157. DEBUG 16:20:54.420 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489654412
  2158. DEBUG 16:20:54.883 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489654876
  2159. DEBUG 16:20:55.348 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489655340
  2160. DEBUG 16:20:55.349 22867 aiservice.py 06085efc57dc Batch for camera test collected, spend 11.13683557510376 sec
  2161. INFO 16:20:55.350 22867 aiservice.py f374c07dd86e Start collecting batch for camera test, spend 11.014968156814575 sec
  2162. INFO 16:20:55.371 22842 aiservice.py 06085efc57dc Start processing batch for camera test, spend 11.159249067306519 sec
  2163. DEBUG 16:20:55.372 22842 foreman.py 06085efc57dc Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2164. 24
  2165. DEBUG 16:20:55.385 22842 foreman.py 06085efc57dc Running=0 finished=1
  2166. DEBUG 16:20:55.385 22842 foreman.py 06085efc57dc Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2167. DEBUG 16:20:55.387 22865 command.py 06085efc57dc Start core.cnn
  2168. DEBUG 16:20:55.401 22842 shared_array.py ------------ Shared memory free 152, used 48
  2169. INFO 16:20:55.402 22842 aiservice.py 8833c22f48f3 Put batch for camera test request to queue, spend 0.0007660388946533203 sec
  2170. [TensorRT] INFO: Detecting input data format
  2171. [TensorRT] INFO: Dectected data format LCHW
  2172. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2173. [TensorRT] INFO: Verifying batches are the expected data type
  2174. [TensorRT] INFO: Executing inference
  2175. [TensorRT] INFO: Number of Batches: 1
  2176. ERROR 16:20:55.438 22865 command.py 06085efc57dc Worker core.cnn failed
  2177. Traceback (most recent call last):
  2178. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2179. result = worker.do(batch)
  2180. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2181. batch.payload.get("image_idx")
  2182. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2183. frames = self.process_frames(images, image_indexes)
  2184. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2185. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2186. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2187. result = self.process(imgs, scale=scale)
  2188. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2189. result.append(self._process(batch, **kwargs))
  2190. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2191. l1s, l2s = self.engine.infer(items)
  2192. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2193. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2194. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2195. DEBUG 16:20:55.448 22842 foreman.py 06085efc57dc Running=0 finished=1
  2196. CRITICAL 16:20:55.448 22842 foreman.py 06085efc57dc Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f668> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2197. CRITICAL 16:20:55.448 22842 foreman.py 06085efc57dc Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2198. ERROR 16:20:55.448 22842 conclusion.py ------------ No Telegram's chats connected
  2199. INFO 16:20:55.449 22842 aiservice.py 06085efc57dc Batch for camera test processed, spend 0.07730674743652344 sec
  2200. DEBUG 16:20:55.449 22842 shared_array.py ------------ Shared memory free 152, used 48
  2201. DEBUG 16:20:55.811 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489655804
  2202. DEBUG 16:20:56.276 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489656268
  2203. DEBUG 16:20:56.738 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489656732
  2204. DEBUG 16:20:57.203 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489657196
  2205. DEBUG 16:20:57.666 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489657660
  2206. DEBUG 16:20:58.130 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489658124
  2207. DEBUG 16:20:58.594 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489658588
  2208. DEBUG 16:20:59.058 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489659052
  2209. DEBUG 16:20:59.522 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489659516
  2210. DEBUG 16:20:59.982 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489659980
  2211. DEBUG 16:21:00.447 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489660444
  2212. DEBUG 16:21:00.911 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489660908
  2213. DEBUG 16:21:01.374 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489661372
  2214. DEBUG 16:21:01.839 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489661836
  2215. DEBUG 16:21:02.303 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489662300
  2216. DEBUG 16:21:02.766 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489662764
  2217. DEBUG 16:21:03.231 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489663228
  2218. DEBUG 16:21:03.691 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489663692
  2219. DEBUG 16:21:04.161 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489664156
  2220. DEBUG 16:21:04.623 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489664620
  2221. DEBUG 16:21:05.086 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489665084
  2222. DEBUG 16:21:05.550 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489665548
  2223. DEBUG 16:21:06.014 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489666008
  2224. DEBUG 16:21:06.478 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489666472
  2225. DEBUG 16:21:06.480 22867 aiservice.py f374c07dd86e Batch for camera test collected, spend 11.129939317703247 sec
  2226. INFO 16:21:06.481 22867 aiservice.py 8833c22f48f3 Start collecting batch for camera test, spend 11.079819440841675 sec
  2227. DEBUG 16:21:06.497 22842 shared_array.py ------------ Shared memory free 152, used 48
  2228. INFO 16:21:06.498 22842 aiservice.py d6858e83b6db Put batch for camera test request to queue, spend 0.0024137496948242188 sec
  2229. INFO 16:21:06.546 22842 aiservice.py f374c07dd86e Start processing batch for camera test, spend 11.196073293685913 sec
  2230. DEBUG 16:21:06.547 22842 foreman.py f374c07dd86e Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2231. DEBUG 16:21:06.559 22842 foreman.py f374c07dd86e Running=0 finished=1
  2232. DEBUG 16:21:06.560 22842 foreman.py f374c07dd86e Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2233. 24
  2234. DEBUG 16:21:06.561 22865 command.py f374c07dd86e Start core.cnn
  2235. [TensorRT] INFO: Detecting input data format
  2236. [TensorRT] INFO: Dectected data format LCHW
  2237. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2238. [TensorRT] INFO: Verifying batches are the expected data type
  2239. [TensorRT] INFO: Executing inference
  2240. [TensorRT] INFO: Number of Batches: 1
  2241. ERROR 16:21:06.640 22865 command.py f374c07dd86e Worker core.cnn failed
  2242. Traceback (most recent call last):
  2243. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2244. result = worker.do(batch)
  2245. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2246. batch.payload.get("image_idx")
  2247. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2248. frames = self.process_frames(images, image_indexes)
  2249. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2250. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2251. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2252. result = self.process(imgs, scale=scale)
  2253. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2254. result.append(self._process(batch, **kwargs))
  2255. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2256. l1s, l2s = self.engine.infer(items)
  2257. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2258. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2259. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2260. DEBUG 16:21:06.642 22842 foreman.py f374c07dd86e Running=0 finished=1
  2261. CRITICAL 16:21:06.643 22842 foreman.py f374c07dd86e Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f4a8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2262. CRITICAL 16:21:06.643 22842 foreman.py f374c07dd86e Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2263. ERROR 16:21:06.643 22842 conclusion.py ------------ No Telegram's chats connected
  2264. INFO 16:21:06.643 22842 aiservice.py f374c07dd86e Batch for camera test processed, spend 0.0969233512878418 sec
  2265. DEBUG 16:21:06.644 22842 shared_array.py ------------ Shared memory free 152, used 48
  2266. DEBUG 16:21:06.942 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489666936
  2267. DEBUG 16:21:07.406 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489667400
  2268. DEBUG 16:21:07.871 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489667864
  2269. DEBUG 16:21:08.334 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489668328
  2270. DEBUG 16:21:08.799 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489668792
  2271. DEBUG 16:21:09.262 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489669256
  2272. DEBUG 16:21:09.726 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489669720
  2273. DEBUG 16:21:10.191 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489670184
  2274. DEBUG 16:21:10.657 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489670648
  2275. DEBUG 16:21:11.119 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489671112
  2276. DEBUG 16:21:11.583 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489671576
  2277. DEBUG 16:21:12.046 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489672040
  2278. DEBUG 16:21:12.510 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489672504
  2279. DEBUG 16:21:12.974 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489672968
  2280. DEBUG 16:21:13.439 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489673432
  2281. DEBUG 16:21:13.902 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489673896
  2282. DEBUG 16:21:14.366 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489674360
  2283. DEBUG 16:21:14.831 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489674824
  2284. DEBUG 16:21:15.290 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489675288
  2285. DEBUG 16:21:15.755 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489675752
  2286. DEBUG 16:21:16.219 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489676216
  2287. DEBUG 16:21:16.684 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489676680
  2288. DEBUG 16:21:17.151 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489677144
  2289. DEBUG 16:21:17.615 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489677608
  2290. DEBUG 16:21:17.617 22867 aiservice.py 8833c22f48f3 Batch for camera test collected, spend 11.135419368743896 sec
  2291. INFO 16:21:17.618 22867 aiservice.py d6858e83b6db Start collecting batch for camera test, spend 11.121983528137207 sec
  2292. INFO 16:21:17.651 22842 aiservice.py 8833c22f48f3 Start processing batch for camera test, spend 11.169381856918335 sec
  2293. DEBUG 16:21:17.651 22842 foreman.py 8833c22f48f3 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2294. DEBUG 16:21:17.664 22842 foreman.py 8833c22f48f3 Running=0 finished=1
  2295. DEBUG 16:21:17.664 22842 foreman.py 8833c22f48f3 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2296. 24
  2297. DEBUG 16:21:17.666 22865 command.py 8833c22f48f3 Start core.cnn
  2298. [TensorRT] INFO: Detecting input data format
  2299. [TensorRT] INFO: Dectected data format LCHW
  2300. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2301. [TensorRT] INFO: Verifying batches are the expected data type
  2302. [TensorRT] INFO: Executing inference
  2303. [TensorRT] INFO: Number of Batches: 1
  2304. ERROR 16:21:17.715 22865 command.py 8833c22f48f3 Worker core.cnn failed
  2305. Traceback (most recent call last):
  2306. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2307. result = worker.do(batch)
  2308. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2309. batch.payload.get("image_idx")
  2310. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2311. frames = self.process_frames(images, image_indexes)
  2312. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2313. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2314. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2315. result = self.process(imgs, scale=scale)
  2316. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2317. result.append(self._process(batch, **kwargs))
  2318. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2319. l1s, l2s = self.engine.infer(items)
  2320. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2321. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2322. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2323. DEBUG 16:21:17.716 22842 foreman.py 8833c22f48f3 Running=0 finished=1
  2324. CRITICAL 16:21:17.716 22842 foreman.py 8833c22f48f3 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f588> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2325. CRITICAL 16:21:17.716 22842 foreman.py 8833c22f48f3 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2326. ERROR 16:21:17.717 22842 conclusion.py ------------ No Telegram's chats connected
  2327. INFO 16:21:17.717 22842 aiservice.py 8833c22f48f3 Batch for camera test processed, spend 0.06614375114440918 sec
  2328. DEBUG 16:21:17.717 22842 shared_array.py ------------ Shared memory free 176, used 24
  2329. DEBUG 16:21:17.790 22842 shared_array.py ------------ Shared memory free 176, used 24
  2330. INFO 16:21:17.791 22842 aiservice.py 9c79949dfb35 Put batch for camera test request to queue, spend 0.0013041496276855469 sec
  2331. DEBUG 16:21:18.079 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489678072
  2332. DEBUG 16:21:18.543 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489678536
  2333. DEBUG 16:21:19.006 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489679000
  2334. DEBUG 16:21:19.470 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489679464
  2335. DEBUG 16:21:19.934 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489679928
  2336. DEBUG 16:21:20.399 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489680392
  2337. DEBUG 16:21:20.862 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489680856
  2338. DEBUG 16:21:21.327 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489681320
  2339. DEBUG 16:21:21.790 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489681784
  2340. DEBUG 16:21:22.254 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489682248
  2341. DEBUG 16:21:22.718 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489682712
  2342. DEBUG 16:21:23.183 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489683176
  2343. DEBUG 16:21:23.646 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489683640
  2344. DEBUG 16:21:24.110 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489684104
  2345. DEBUG 16:21:24.575 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489684568
  2346. DEBUG 16:21:25.039 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489685032
  2347. DEBUG 16:21:25.504 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489685496
  2348. DEBUG 16:21:25.962 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489685960
  2349. DEBUG 16:21:26.427 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489686424
  2350. DEBUG 16:21:26.890 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489686888
  2351. DEBUG 16:21:27.355 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489687352
  2352. DEBUG 16:21:27.818 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489687816
  2353. DEBUG 16:21:28.282 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489688280
  2354. DEBUG 16:21:28.746 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489688744
  2355. DEBUG 16:21:28.748 22867 aiservice.py d6858e83b6db Batch for camera test collected, spend 11.130508661270142 sec
  2356. INFO 16:21:28.749 22867 aiservice.py 9c79949dfb35 Start collecting batch for camera test, spend 10.958952903747559 sec
  2357. INFO 16:21:28.786 22842 aiservice.py d6858e83b6db Start processing batch for camera test, spend 11.168225049972534 sec
  2358. DEBUG 16:21:28.787 22842 foreman.py d6858e83b6db Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2359. DEBUG 16:21:28.799 22842 foreman.py d6858e83b6db Running=0 finished=1
  2360. DEBUG 16:21:28.799 22842 foreman.py d6858e83b6db Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2361. DEBUG 16:21:28.801 22865 command.py d6858e83b6db Start core.cnn
  2362. 24
  2363. [TensorRT] INFO: Detecting input data format
  2364. [TensorRT] INFO: Dectected data format LCHW
  2365. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2366. [TensorRT] INFO: Verifying batches are the expected data type
  2367. [TensorRT] INFO: Executing inference
  2368. [TensorRT] INFO: Number of Batches: 1
  2369. ERROR 16:21:28.880 22865 command.py d6858e83b6db Worker core.cnn failed
  2370. Traceback (most recent call last):
  2371. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2372. result = worker.do(batch)
  2373. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2374. batch.payload.get("image_idx")
  2375. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2376. frames = self.process_frames(images, image_indexes)
  2377. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2378. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2379. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2380. result = self.process(imgs, scale=scale)
  2381. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2382. result.append(self._process(batch, **kwargs))
  2383. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2384. l1s, l2s = self.engine.infer(items)
  2385. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2386. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2387. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2388. DEBUG 16:21:28.883 22842 foreman.py d6858e83b6db Running=0 finished=1
  2389. CRITICAL 16:21:28.884 22842 foreman.py d6858e83b6db Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f630> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2390. CRITICAL 16:21:28.884 22842 foreman.py d6858e83b6db Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2391. ERROR 16:21:28.884 22842 conclusion.py ------------ No Telegram's chats connected
  2392. INFO 16:21:28.884 22842 aiservice.py d6858e83b6db Batch for camera test processed, spend 0.09810233116149902 sec
  2393. DEBUG 16:21:28.885 22842 shared_array.py ------------ Shared memory free 176, used 24
  2394. DEBUG 16:21:28.888 22842 shared_array.py ------------ Shared memory free 176, used 24
  2395. INFO 16:21:28.889 22842 aiservice.py 4961a511dfdc Put batch for camera test request to queue, spend 0.00046944618225097656 sec
  2396. DEBUG 16:21:29.210 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489689208
  2397. DEBUG 16:21:29.674 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489689672
  2398. DEBUG 16:21:30.138 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489690136
  2399. DEBUG 16:21:30.598 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489690600
  2400. DEBUG 16:21:31.068 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489691064
  2401. DEBUG 16:21:31.530 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489691524
  2402. DEBUG 16:21:31.994 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489691988
  2403. DEBUG 16:21:32.458 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489692452
  2404. DEBUG 16:21:32.923 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489692916
  2405. DEBUG 16:21:33.386 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489693380
  2406. DEBUG 16:21:33.851 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489693844
  2407. DEBUG 16:21:34.315 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489694308
  2408. DEBUG 16:21:34.779 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489694772
  2409. DEBUG 16:21:35.244 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489695236
  2410. DEBUG 16:21:35.707 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489695700
  2411. DEBUG 16:21:36.170 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489696164
  2412. DEBUG 16:21:36.634 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489696628
  2413. DEBUG 16:21:37.100 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489697092
  2414. DEBUG 16:21:37.565 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489697556
  2415. DEBUG 16:21:38.027 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489698020
  2416. DEBUG 16:21:38.490 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489698484
  2417. DEBUG 16:21:38.955 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489698948
  2418. DEBUG 16:21:39.420 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489699412
  2419. DEBUG 16:21:39.882 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489699876
  2420. DEBUG 16:21:39.884 22867 aiservice.py 9c79949dfb35 Batch for camera test collected, spend 11.135164260864258 sec
  2421. INFO 16:21:39.885 22867 aiservice.py 4961a511dfdc Start collecting batch for camera test, spend 10.997235536575317 sec
  2422. INFO 16:21:39.892 22842 aiservice.py 9c79949dfb35 Start processing batch for camera test, spend 11.143388748168945 sec
  2423. DEBUG 16:21:39.893 22842 foreman.py 9c79949dfb35 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2424. DEBUG 16:21:39.905 22842 foreman.py 9c79949dfb35 Running=0 finished=1
  2425. DEBUG 16:21:39.906 22842 foreman.py 9c79949dfb35 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2426. DEBUG 16:21:39.907 22865 command.py 9c79949dfb35 Start core.cnn
  2427. 24
  2428. [TensorRT] INFO: Detecting input data format
  2429. [TensorRT] INFO: Dectected data format LCHW
  2430. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2431. [TensorRT] INFO: Verifying batches are the expected data type
  2432. [TensorRT] INFO: Executing inference
  2433. [TensorRT] INFO: Number of Batches: 1
  2434. ERROR 16:21:39.958 22865 command.py 9c79949dfb35 Worker core.cnn failed
  2435. Traceback (most recent call last):
  2436. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2437. result = worker.do(batch)
  2438. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2439. batch.payload.get("image_idx")
  2440. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2441. frames = self.process_frames(images, image_indexes)
  2442. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2443. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2444. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2445. result = self.process(imgs, scale=scale)
  2446. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2447. result.append(self._process(batch, **kwargs))
  2448. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2449. l1s, l2s = self.engine.infer(items)
  2450. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2451. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2452. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2453. DEBUG 16:21:39.968 22842 foreman.py 9c79949dfb35 Running=0 finished=1
  2454. CRITICAL 16:21:39.968 22842 foreman.py 9c79949dfb35 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f748> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2455. CRITICAL 16:21:39.969 22842 foreman.py 9c79949dfb35 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2456. ERROR 16:21:39.969 22842 conclusion.py ------------ No Telegram's chats connected
  2457. INFO 16:21:39.969 22842 aiservice.py 9c79949dfb35 Batch for camera test processed, spend 0.07663345336914062 sec
  2458. DEBUG 16:21:39.970 22842 shared_array.py ------------ Shared memory free 176, used 24
  2459. DEBUG 16:21:39.981 22842 shared_array.py ------------ Shared memory free 176, used 24
  2460. INFO 16:21:39.982 22842 aiservice.py ffe926e97335 Put batch for camera test request to queue, spend 0.00037598609924316406 sec
  2461. DEBUG 16:21:40.346 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489700340
  2462. DEBUG 16:21:40.811 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489700804
  2463. DEBUG 16:21:41.275 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489701268
  2464. DEBUG 16:21:41.738 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489701732
  2465. DEBUG 16:21:42.202 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489702196
  2466. DEBUG 16:21:42.667 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489702660
  2467. DEBUG 16:21:43.130 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489703125
  2468. DEBUG 16:21:43.595 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489703588
  2469. DEBUG 16:21:44.059 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489704052
  2470. DEBUG 16:21:44.523 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489704516
  2471. DEBUG 16:21:44.987 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489704981
  2472. DEBUG 16:21:45.451 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489705444
  2473. DEBUG 16:21:45.914 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489705909
  2474. DEBUG 16:21:46.379 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489706373
  2475. DEBUG 16:21:46.843 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489706837
  2476. DEBUG 16:21:47.303 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489707301
  2477. DEBUG 16:21:47.772 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489707765
  2478. DEBUG 16:21:48.235 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489708229
  2479. DEBUG 16:21:48.699 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489708693
  2480. DEBUG 16:21:49.160 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489709157
  2481. DEBUG 16:21:49.623 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489709621
  2482. DEBUG 16:21:50.087 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489710085
  2483. DEBUG 16:21:50.555 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489710549
  2484. DEBUG 16:21:51.019 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489711013
  2485. DEBUG 16:21:51.021 22867 aiservice.py 4961a511dfdc Batch for camera test collected, spend 11.135332345962524 sec
  2486. INFO 16:21:51.022 22867 aiservice.py ffe926e97335 Start collecting batch for camera test, spend 11.040886640548706 sec
  2487. INFO 16:21:51.036 22842 aiservice.py 4961a511dfdc Start processing batch for camera test, spend 11.150727987289429 sec
  2488. DEBUG 16:21:51.037 22842 foreman.py 4961a511dfdc Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2489. DEBUG 16:21:51.049 22842 foreman.py 4961a511dfdc Running=0 finished=1
  2490. DEBUG 16:21:51.050 22842 foreman.py 4961a511dfdc Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2491. 24
  2492. DEBUG 16:21:51.051 22865 command.py 4961a511dfdc Start core.cnn
  2493. DEBUG 16:21:51.070 22842 shared_array.py ------------ Shared memory free 152, used 48
  2494. INFO 16:21:51.071 22842 aiservice.py 52a102120d5b Put batch for camera test request to queue, spend 0.0019698143005371094 sec
  2495. [TensorRT] INFO: Detecting input data format
  2496. [TensorRT] INFO: Dectected data format LCHW
  2497. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2498. [TensorRT] INFO: Verifying batches are the expected data type
  2499. [TensorRT] INFO: Executing inference
  2500. [TensorRT] INFO: Number of Batches: 1
  2501. ERROR 16:21:51.124 22865 command.py 4961a511dfdc Worker core.cnn failed
  2502. Traceback (most recent call last):
  2503. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2504. result = worker.do(batch)
  2505. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2506. batch.payload.get("image_idx")
  2507. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2508. frames = self.process_frames(images, image_indexes)
  2509. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2510. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2511. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2512. result = self.process(imgs, scale=scale)
  2513. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2514. result.append(self._process(batch, **kwargs))
  2515. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2516. l1s, l2s = self.engine.infer(items)
  2517. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2518. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2519. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2520. DEBUG 16:21:51.125 22842 foreman.py 4961a511dfdc Running=0 finished=1
  2521. CRITICAL 16:21:51.125 22842 foreman.py 4961a511dfdc Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f438> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2522. CRITICAL 16:21:51.126 22842 foreman.py 4961a511dfdc Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2523. ERROR 16:21:51.126 22842 conclusion.py ------------ No Telegram's chats connected
  2524. INFO 16:21:51.126 22842 aiservice.py 4961a511dfdc Batch for camera test processed, spend 0.08957862854003906 sec
  2525. DEBUG 16:21:51.127 22842 shared_array.py ------------ Shared memory free 152, used 48
  2526. DEBUG 16:21:51.479 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489711477
  2527. DEBUG 16:21:51.943 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489711941
  2528. DEBUG 16:21:52.407 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489712405
  2529. DEBUG 16:21:52.872 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489712869
  2530. DEBUG 16:21:53.331 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489713333
  2531. DEBUG 16:21:53.798 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489713797
  2532. DEBUG 16:21:54.263 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489714261
  2533. DEBUG 16:21:54.727 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489714725
  2534. DEBUG 16:21:55.191 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489715189
  2535. DEBUG 16:21:55.655 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489715653
  2536. DEBUG 16:21:56.119 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489716117
  2537. DEBUG 16:21:56.583 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489716581
  2538. DEBUG 16:21:57.047 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489717045
  2539. DEBUG 16:21:57.511 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489717505
  2540. DEBUG 16:21:57.975 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489717969
  2541. DEBUG 16:21:58.439 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489718433
  2542. DEBUG 16:21:58.903 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489718897
  2543. DEBUG 16:21:59.367 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489719361
  2544. DEBUG 16:21:59.831 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489719825
  2545. DEBUG 16:22:00.295 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489720289
  2546. DEBUG 16:22:00.759 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489720753
  2547. DEBUG 16:22:01.223 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489721217
  2548. DEBUG 16:22:01.687 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489721681
  2549. DEBUG 16:22:02.151 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489722145
  2550. DEBUG 16:22:02.151 22867 aiservice.py ffe926e97335 Batch for camera test collected, spend 11.129013299942017 sec
  2551. INFO 16:22:02.153 22867 aiservice.py 52a102120d5b Start collecting batch for camera test, spend 11.083084106445312 sec
  2552. DEBUG 16:22:02.164 22842 shared_array.py ------------ Shared memory free 152, used 48
  2553. INFO 16:22:02.165 22842 aiservice.py 820f51d36fd9 Put batch for camera test request to queue, spend 0.0022878646850585938 sec
  2554. INFO 16:22:02.204 22842 aiservice.py ffe926e97335 Start processing batch for camera test, spend 11.181411504745483 sec
  2555. DEBUG 16:22:02.205 22842 foreman.py ffe926e97335 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2556. DEBUG 16:22:02.217 22842 foreman.py ffe926e97335 Running=0 finished=1
  2557. DEBUG 16:22:02.217 22842 foreman.py ffe926e97335 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2558. 24
  2559. DEBUG 16:22:02.219 22865 command.py ffe926e97335 Start core.cnn
  2560. [TensorRT] INFO: Detecting input data format
  2561. [TensorRT] INFO: Dectected data format LCHW
  2562. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2563. [TensorRT] INFO: Verifying batches are the expected data type
  2564. [TensorRT] INFO: Executing inference
  2565. [TensorRT] INFO: Number of Batches: 1
  2566. ERROR 16:22:02.285 22865 command.py ffe926e97335 Worker core.cnn failed
  2567. Traceback (most recent call last):
  2568. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2569. result = worker.do(batch)
  2570. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2571. batch.payload.get("image_idx")
  2572. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2573. frames = self.process_frames(images, image_indexes)
  2574. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2575. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2576. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2577. result = self.process(imgs, scale=scale)
  2578. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2579. result.append(self._process(batch, **kwargs))
  2580. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2581. l1s, l2s = self.engine.infer(items)
  2582. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2583. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2584. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2585. DEBUG 16:22:02.290 22842 foreman.py ffe926e97335 Running=0 finished=1
  2586. CRITICAL 16:22:02.290 22842 foreman.py ffe926e97335 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f6d8> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2587. CRITICAL 16:22:02.290 22842 foreman.py ffe926e97335 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2588. ERROR 16:22:02.291 22842 conclusion.py ------------ No Telegram's chats connected
  2589. INFO 16:22:02.291 22842 aiservice.py ffe926e97335 Batch for camera test processed, spend 0.08663058280944824 sec
  2590. DEBUG 16:22:02.291 22842 shared_array.py ------------ Shared memory free 152, used 48
  2591. DEBUG 16:22:02.615 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489722609
  2592. DEBUG 16:22:03.079 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489723073
  2593. DEBUG 16:22:03.543 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489723537
  2594. DEBUG 16:22:04.008 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489724001
  2595. DEBUG 16:22:04.472 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489724465
  2596. DEBUG 16:22:04.935 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489724929
  2597. DEBUG 16:22:05.399 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489725393
  2598. DEBUG 16:22:05.863 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489725857
  2599. DEBUG 16:22:06.327 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489726321
  2600. DEBUG 16:22:06.787 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489726785
  2601. DEBUG 16:22:07.251 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489727249
  2602. DEBUG 16:22:07.715 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489727713
  2603. DEBUG 16:22:08.179 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489728177
  2604. DEBUG 16:22:08.647 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489728641
  2605. DEBUG 16:22:09.111 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489729105
  2606. DEBUG 16:22:09.571 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489729569
  2607. DEBUG 16:22:10.035 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489730033
  2608. DEBUG 16:22:10.503 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489730497
  2609. DEBUG 16:22:10.969 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489730961
  2610. DEBUG 16:22:11.431 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489731425
  2611. DEBUG 16:22:11.896 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489731889
  2612. DEBUG 16:22:12.360 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489732353
  2613. DEBUG 16:22:12.824 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489732817
  2614. DEBUG 16:22:13.287 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489733281
  2615. DEBUG 16:22:13.289 22867 aiservice.py 52a102120d5b Batch for camera test collected, spend 11.136087894439697 sec
  2616. INFO 16:22:13.290 22867 aiservice.py 820f51d36fd9 Start collecting batch for camera test, spend 11.127214431762695 sec
  2617. INFO 16:22:13.302 22842 aiservice.py 52a102120d5b Start processing batch for camera test, spend 11.149511575698853 sec
  2618. DEBUG 16:22:13.303 22842 foreman.py 52a102120d5b Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2619. DEBUG 16:22:13.315 22842 foreman.py 52a102120d5b Running=0 finished=1
  2620. DEBUG 16:22:13.315 22842 foreman.py 52a102120d5b Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2621. DEBUG 16:22:13.317 22865 command.py 52a102120d5b Start core.cnn
  2622. 24
  2623. [TensorRT] INFO: Detecting input data format
  2624. [TensorRT] INFO: Dectected data format LCHW
  2625. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2626. [TensorRT] INFO: Verifying batches are the expected data type
  2627. [TensorRT] INFO: Executing inference
  2628. [TensorRT] INFO: Number of Batches: 1
  2629. ERROR 16:22:13.367 22865 command.py 52a102120d5b Worker core.cnn failed
  2630. Traceback (most recent call last):
  2631. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2632. result = worker.do(batch)
  2633. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2634. batch.payload.get("image_idx")
  2635. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2636. frames = self.process_frames(images, image_indexes)
  2637. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2638. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2639. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2640. result = self.process(imgs, scale=scale)
  2641. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2642. result.append(self._process(batch, **kwargs))
  2643. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2644. l1s, l2s = self.engine.infer(items)
  2645. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2646. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2647. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2648. DEBUG 16:22:13.378 22842 foreman.py 52a102120d5b Running=0 finished=1
  2649. CRITICAL 16:22:13.378 22842 foreman.py 52a102120d5b Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f710> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2650. CRITICAL 16:22:13.378 22842 foreman.py 52a102120d5b Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f400> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2651. ERROR 16:22:13.378 22842 conclusion.py ------------ No Telegram's chats connected
  2652. INFO 16:22:13.378 22842 aiservice.py 52a102120d5b Batch for camera test processed, spend 0.07578659057617188 sec
  2653. DEBUG 16:22:13.379 22842 shared_array.py ------------ Shared memory free 176, used 24
  2654. DEBUG 16:22:13.451 22842 shared_array.py ------------ Shared memory free 176, used 24
  2655. INFO 16:22:13.451 22842 aiservice.py 79a64d81b6fb Put batch for camera test request to queue, spend 0.0015377998352050781 sec
  2656. DEBUG 16:22:13.751 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489733745
  2657. DEBUG 16:22:14.215 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489734209
  2658. DEBUG 16:22:14.643 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489734641
  2659. DEBUG 16:22:14.871 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489734873
  2660. DEBUG 16:22:15.107 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489735105
  2661. DEBUG 16:22:15.339 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489735337
  2662. DEBUG 16:22:15.571 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489735569
  2663. DEBUG 16:22:15.804 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489735797
  2664. DEBUG 16:22:16.035 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489736029
  2665. DEBUG 16:22:16.267 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489736261
  2666. DEBUG 16:22:16.499 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489736493
  2667. DEBUG 16:22:16.732 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489736725
  2668. DEBUG 16:22:16.959 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489736957
  2669. DEBUG 16:22:17.195 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489737189
  2670. DEBUG 16:22:17.428 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489737421
  2671. DEBUG 16:22:17.660 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489737653
  2672. DEBUG 16:22:17.887 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489737885
  2673. DEBUG 16:22:18.119 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489738117
  2674. DEBUG 16:22:18.351 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489738349
  2675. DEBUG 16:22:18.583 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489738581
  2676. DEBUG 16:22:18.815 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489738813
  2677. DEBUG 16:22:19.047 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489739045
  2678. DEBUG 16:22:19.283 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489739277
  2679. DEBUG 16:22:19.511 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489739509
  2680. DEBUG 16:22:19.512 22867 aiservice.py 820f51d36fd9 Batch for camera test collected, spend 6.221423387527466 sec
  2681. INFO 16:22:19.512 22867 aiservice.py 79a64d81b6fb Start collecting batch for camera test, spend 6.062250375747681 sec
  2682. INFO 16:22:19.515 22842 aiservice.py 820f51d36fd9 Start processing batch for camera test, spend 6.225287675857544 sec
  2683. DEBUG 16:22:19.516 22842 foreman.py 820f51d36fd9 Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2684. DEBUG 16:22:19.522 22842 foreman.py 820f51d36fd9 Running=0 finished=1
  2685. DEBUG 16:22:19.522 22842 foreman.py 820f51d36fd9 Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2686. DEBUG 16:22:19.523 22865 command.py 820f51d36fd9 Start core.cnn
  2687. 24
  2688. [TensorRT] INFO: Detecting input data format
  2689. [TensorRT] INFO: Dectected data format LCHW
  2690. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2691. [TensorRT] INFO: Verifying batches are the expected data type
  2692. [TensorRT] INFO: Executing inference
  2693. [TensorRT] INFO: Number of Batches: 1
  2694. ERROR 16:22:19.591 22865 command.py 820f51d36fd9 Worker core.cnn failed
  2695. Traceback (most recent call last):
  2696. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2697. result = worker.do(batch)
  2698. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2699. batch.payload.get("image_idx")
  2700. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2701. frames = self.process_frames(images, image_indexes)
  2702. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2703. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2704. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2705. result = self.process(imgs, scale=scale)
  2706. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2707. result.append(self._process(batch, **kwargs))
  2708. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2709. l1s, l2s = self.engine.infer(items)
  2710. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2711. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2712. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2713. DEBUG 16:22:19.595 22842 foreman.py 820f51d36fd9 Running=0 finished=1
  2714. CRITICAL 16:22:19.595 22842 foreman.py 820f51d36fd9 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f5c0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2715. CRITICAL 16:22:19.595 22842 foreman.py 820f51d36fd9 Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f470> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2716. ERROR 16:22:19.595 22842 conclusion.py ------------ No Telegram's chats connected
  2717. INFO 16:22:19.595 22842 aiservice.py 820f51d36fd9 Batch for camera test processed, spend 0.07982707023620605 sec
  2718. DEBUG 16:22:19.596 22842 shared_array.py ------------ Shared memory free 176, used 24
  2719. DEBUG 16:22:19.692 22842 shared_array.py ------------ Shared memory free 176, used 24
  2720. INFO 16:22:19.692 22842 aiservice.py e9a726fb7925 Put batch for camera test request to queue, spend 0.00048279762268066406 sec
  2721. DEBUG 16:22:19.743 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489739741
  2722. DEBUG 16:22:19.975 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489739973
  2723. DEBUG 16:22:20.207 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489740205
  2724. DEBUG 16:22:20.439 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489740437
  2725. DEBUG 16:22:20.672 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489740669
  2726. DEBUG 16:22:20.908 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489740901
  2727. DEBUG 16:22:21.136 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489741133
  2728. DEBUG 16:22:21.371 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489741365
  2729. DEBUG 16:22:21.603 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489741597
  2730. DEBUG 16:22:21.835 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489741829
  2731. DEBUG 16:22:22.063 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489742061
  2732. DEBUG 16:22:22.300 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489742293
  2733. DEBUG 16:22:22.532 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489742525
  2734. DEBUG 16:22:22.764 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489742757
  2735. DEBUG 16:22:22.995 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489742989
  2736. DEBUG 16:22:23.223 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489743221
  2737. DEBUG 16:22:23.459 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489743453
  2738. DEBUG 16:22:23.693 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489743685
  2739. DEBUG 16:22:23.924 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489743917
  2740. DEBUG 16:22:24.156 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489744149
  2741. DEBUG 16:22:24.387 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489744381
  2742. DEBUG 16:22:24.619 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489744613
  2743. DEBUG 16:22:24.851 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489744845
  2744. DEBUG 16:22:25.083 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489745077
  2745. DEBUG 16:22:25.085 22867 aiservice.py 79a64d81b6fb Batch for camera test collected, spend 5.572937726974487 sec
  2746. INFO 16:22:25.086 22867 aiservice.py e9a726fb7925 Start collecting batch for camera test, spend 5.394859075546265 sec
  2747. INFO 16:22:25.092 22842 aiservice.py 79a64d81b6fb Start processing batch for camera test, spend 5.579809665679932 sec
  2748. DEBUG 16:22:25.093 22842 foreman.py 79a64d81b6fb Run cmd core.video_process_worker(<processing.command.SimpleCommand object at 0x7f9b622674a8>)
  2749. DEBUG 16:22:25.105 22842 foreman.py 79a64d81b6fb Running=0 finished=1
  2750. DEBUG 16:22:25.106 22842 foreman.py 79a64d81b6fb Run cmd core.cnn(<processing.command.SeparateProcessCommand object at 0x7f9b62267518>)
  2751. DEBUG 16:22:25.107 22865 command.py 79a64d81b6fb Start core.cnn
  2752. 24
  2753. DEBUG 16:22:25.129 22842 shared_array.py ------------ Shared memory free 152, used 48
  2754. INFO 16:22:25.131 22842 aiservice.py 282ac4e67d23 Put batch for camera test request to queue, spend 0.0025548934936523438 sec
  2755. [TensorRT] INFO: Detecting input data format
  2756. [TensorRT] INFO: Dectected data format LCHW
  2757. [TensorRT] INFO: Verifying data format is uniform accross all input layers
  2758. [TensorRT] INFO: Verifying batches are the expected data type
  2759. [TensorRT] INFO: Executing inference
  2760. [TensorRT] INFO: Number of Batches: 1
  2761. ERROR 16:22:25.185 22865 command.py 79a64d81b6fb Worker core.cnn failed
  2762. Traceback (most recent call last):
  2763. File "/home/denis/beta1/processing/command.py", line 87, in _execute
  2764. result = worker.do(batch)
  2765. File "/home/denis/beta1/core/cnn/worker.py", line 25, in do
  2766. batch.payload.get("image_idx")
  2767. File "/home/denis/beta1/core/cnn/rtpose.py", line 57, in process_frames_by_batch
  2768. frames = self.process_frames(images, image_indexes)
  2769. File "/home/denis/beta1/core/cnn/rtpose.py", line 23, in process_frames
  2770. estimated = self.pose_estimator.estimate(images, self.scale_level)
  2771. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 571, in estimate
  2772. result = self.process(imgs, scale=scale)
  2773. File "/home/denis/beta1/core/BatchProcessor.py", line 36, in process
  2774. result.append(self._process(batch, **kwargs))
  2775. File "/home/denis/beta1/core/cnn/pose_estimation.py", line 595, in _process
  2776. l1s, l2s = self.engine.infer(items)
  2777. File "/usr/lib/python3.5/dist-packages/tensorrt/lite/engine.py", line 668, in infer
  2778. cuda.memcpy_htod_async(self.d_input[i], np.ascontiguousarray(input_data[i][n].astype(self.data_type.input_type())), stream)
  2779. pycuda._driver.LogicError: cuMemcpyHtoDAsync failed: invalid argument
  2780. DEBUG 16:22:25.190 22842 foreman.py 79a64d81b6fb Running=0 finished=1
  2781. CRITICAL 16:22:25.190 22842 foreman.py 79a64d81b6fb Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f4e0> that can not be done. Required {'images_time', 'frames', 'treaded_video', 'image_idx', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2782. CRITICAL 16:22:25.190 22842 foreman.py 79a64d81b6fb Pipe has unfinished command <processing.foreman._CmdWrapper object at 0x7f9b6227f3c8> that can not be done. Required {'treaded_video', 'images_time', 'tracks_merged', 'track_frames', 'images'}, Have dict_keys(['image_idx', 'treaded_video', 'images_time'])
  2783. ERROR 16:22:25.191 22842 conclusion.py ------------ No Telegram's chats connected
  2784. INFO 16:22:25.191 22842 aiservice.py 79a64d81b6fb Batch for camera test processed, spend 0.09888029098510742 sec
  2785. DEBUG 16:22:25.192 22842 shared_array.py ------------ Shared memory free 152, used 48
  2786. DEBUG 16:22:25.315 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489745309
  2787. DEBUG 16:22:25.548 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489745541
  2788. DEBUG 16:22:25.776 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489745773
  2789. DEBUG 16:22:26.011 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489746005
  2790. DEBUG 16:22:26.248 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489746237
  2791. DEBUG 16:22:26.472 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489746469
  2792. DEBUG 16:22:26.703 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489746701
  2793. DEBUG 16:22:26.940 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489746933
  2794. DEBUG 16:22:27.171 22867 streamer_source.py ------------ Frame codec: 4, size: 921600, resolution: 640x480 time: 1524489747165
  2795. Process Process-1:
  2796. Process Process-3:
  2797. Process Process-2:
  2798. Process Process-4:
  2799. Traceback (most recent call last):
  2800. File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
  2801. self.run()
  2802. File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
  2803. self._target(*self._args, **self._kwargs)
  2804. File "/home/denis/beta1/processing/command.py", line 167, in _mainloop
  2805. batch = queue_in.get()
  2806. File "/usr/lib/python3.5/multiprocessing/queues.py", line 94, in get
  2807. res = self._recv_bytes()
  2808. File "/usr/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
  2809. buf = self._recv_bytes(maxlength)
  2810. File "/usr/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
  2811. buf = self._recv(4)
  2812. File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
  2813. chunk = read(handle, remaining)
  2814. Traceback (most recent call last):
  2815. KeyboardInterrupt
  2816. Traceback (most recent call last):
  2817. File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
  2818. self.run()
  2819. File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
  2820. self._target(*self._args, **self._kwargs)
  2821. File "/home/denis/beta1/processing/command.py", line 167, in _mainloop
  2822. batch = queue_in.get()
  2823. File "/usr/lib/python3.5/multiprocessing/queues.py", line 94, in get
  2824. res = self._recv_bytes()
  2825. File "/usr/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
  2826. buf = self._recv_bytes(maxlength)
  2827. File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
  2828. self.run()
  2829. File "/usr/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
  2830. buf = self._recv(4)
  2831. File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
  2832. self._target(*self._args, **self._kwargs)
  2833. File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
  2834. chunk = read(handle, remaining)
  2835. File "/home/denis/beta1/processing/command.py", line 167, in _mainloop
  2836. batch = queue_in.get()
  2837. File "/usr/lib/python3.5/multiprocessing/queues.py", line 94, in get
  2838. res = self._recv_bytes()
  2839. File "/usr/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
  2840. buf = self._recv_bytes(maxlength)
  2841. File "/usr/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
  2842. buf = self._recv(4)
  2843. File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
  2844. chunk = read(handle, remaining)
  2845. KeyboardInterrupt
  2846. KeyboardInterrupt
  2847. Traceback (most recent call last):
  2848. File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
  2849. self.run()
  2850. File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
  2851. self._target(*self._args, **self._kwargs)
  2852. File "/home/denis/beta1/processing/aiservice.py", line 230, in dispatch_images
  2853. array, cfg, payload = stream_reader.next(camera_name, array, cfg)
  2854. File "/home/denis/beta1/processing/batch_reader.py", line 76, in next
  2855. img = stream.next()
  2856. File "/home/denis/beta1/sensor_sources/streamer_source.py", line 77, in next
  2857. data = next(self._stream)
  2858. File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 347, in __next__
  2859. return self._next()
  2860. File "/usr/local/lib/python3.5/dist-packages/grpc/_channel.py", line 332, in _next
  2861. self._state.condition.wait()
  2862. File "/usr/lib/python3.5/threading.py", line 293, in wait
  2863. waiter.acquire()
  2864. KeyboardInterrupt
  2865. Traceback (most recent call last):
  2866. File "/home/denis/beta1/main.py", line 12, in <module>
  2867. main()
  2868. File "/home/denis/beta1/main.py", line 8, in main
  2869. service.serve()
  2870. File "/home/denis/beta1/processing/aiservice.py", line 254, in serve
  2871. loop.run_forever()
  2872. File "/usr/lib/python3.5/asyncio/base_events.py", line 345, in run_forever
  2873. self._run_once()
  2874. File "/usr/lib/python3.5/asyncio/base_events.py", line 1276, in _run_once
  2875. event_list = self._selector.select(timeout)
  2876. File "/usr/lib/python3.5/selectors.py", line 441, in select
  2877. fd_event_list = self._epoll.poll(timeout, max_ev)
  2878. KeyboardInterrupt
  2879. ERROR 16:22:27.663 22842 base_events.py ------------ Task was destroyed but it is pending!
  2880. task: <Task pending coro=<AiService._processing() running at /home/denis/beta1/processing/aiservice.py:173> wait_for=<Future pending cb=[Task._wakeup()]>>
  2881. ERROR 16:22:27.663 22842 base_events.py ------------ Task was destroyed but it is pending!
  2882. task: <Task pending coro=<AiService._processing() running at /home/denis/beta1/processing/aiservice.py:173> wait_for=<Future pending cb=[Task._wakeup()]>>
  2883. ERROR 16:22:27.663 22842 base_events.py ------------ Task was destroyed but it is pending!
  2884. task: <Task pending coro=<AiService._processing() running at /home/denis/beta1/processing/aiservice.py:173> wait_for=<Future pending cb=[Task._wakeup()]>>
  2885. ERROR 16:22:27.663 22842 base_events.py ------------ Task was destroyed but it is pending!
  2886. task: <Task pending coro=<AiService._requesting() running at /home/denis/beta1/processing/aiservice.py:154> wait_for=<Future pending cb=[Task._wakeup()]>>
  2887.  
  2888. Process finished with exit code 1
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!