API tools faq
paste
Guest User

Untitled

a guest
Jul 13th, 2018
64
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 104.26 KB | None | 0 0
raw download clone embed print report
  1. 2018-07-12 23:39:53,194 - MainThread - werkzeug - INFO - * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
  2. 2018-07-12 23:39:53,196 - MainThread - werkzeug - INFO - * Restarting with stat
  3. 2018-07-12 23:39:54,263 - MainThread - werkzeug - WARNING - * Debugger is active!
  4. 2018-07-12 23:39:54,264 - MainThread - werkzeug - INFO - * Debugger PIN: 116-097-559
  5. 2018-07-12 23:41:29,544 - Thread-25 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:29] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  6. 2018-07-12 23:41:35,681 - Thread-27 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:35] "GET /api/v1.0/logs HTTP/1.0" 200 -
  7. 2018-07-12 23:41:37,712 - Thread-28 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:37] "GET /api/v1.0/logs HTTP/1.0" 200 -
  8. 2018-07-12 23:41:39,730 - Thread-29 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:39] "GET /api/v1.0/logs HTTP/1.0" 200 -
  9. 2018-07-12 23:41:41,008 - Thread-26 - create_stack - INFO - Created stack with name - StorPerfAgentGroup
  10. 2018-07-12 23:41:41,748 - Thread-30 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:41] "GET /api/v1.0/logs HTTP/1.0" 200 -
  11. 2018-07-12 23:41:43,765 - Thread-31 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:43] "GET /api/v1.0/logs HTTP/1.0" 200 -
  12. 2018-07-12 23:41:45,784 - Thread-32 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:45] "GET /api/v1.0/logs HTTP/1.0" 200 -
  13. 2018-07-12 23:41:47,802 - Thread-33 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:47] "GET /api/v1.0/logs HTTP/1.0" 200 -
  14. 2018-07-12 23:41:49,819 - Thread-34 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:49] "GET /api/v1.0/logs HTTP/1.0" 200 -
  15. 2018-07-12 23:41:51,847 - Thread-35 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:51] "GET /api/v1.0/logs HTTP/1.0" 200 -
  16. 2018-07-12 23:41:53,865 - Thread-36 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:53] "GET /api/v1.0/logs HTTP/1.0" 200 -
  17. 2018-07-12 23:41:55,887 - Thread-37 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:55] "GET /api/v1.0/logs HTTP/1.0" 200 -
  18. 2018-07-12 23:41:57,906 - Thread-38 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:57] "GET /api/v1.0/logs HTTP/1.0" 200 -
  19. 2018-07-12 23:41:59,925 - Thread-39 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:41:59] "GET /api/v1.0/logs HTTP/1.0" 200 -
  20. 2018-07-12 23:42:01,944 - Thread-40 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:01] "GET /api/v1.0/logs HTTP/1.0" 200 -
  21. 2018-07-12 23:42:03,963 - Thread-41 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:03] "GET /api/v1.0/logs HTTP/1.0" 200 -
  22. 2018-07-12 23:42:05,981 - Thread-42 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:05] "GET /api/v1.0/logs HTTP/1.0" 200 -
  23. 2018-07-12 23:42:08,000 - Thread-43 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:08] "GET /api/v1.0/logs HTTP/1.0" 200 -
  24. 2018-07-12 23:42:10,026 - Thread-44 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:10] "GET /api/v1.0/logs HTTP/1.0" 200 -
  25. 2018-07-12 23:42:12,056 - Thread-45 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:12] "GET /api/v1.0/logs HTTP/1.0" 200 -
  26. 2018-07-12 23:42:14,073 - Thread-46 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:14] "GET /api/v1.0/logs HTTP/1.0" 200 -
  27. 2018-07-12 23:42:16,093 - Thread-47 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:16] "GET /api/v1.0/logs HTTP/1.0" 200 -
  28. 2018-07-12 23:42:18,110 - Thread-48 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:18] "GET /api/v1.0/logs HTTP/1.0" 200 -
  29. 2018-07-12 23:42:20,127 - Thread-49 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:20] "GET /api/v1.0/logs HTTP/1.0" 200 -
  30. 2018-07-12 23:42:22,151 - Thread-50 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:22] "GET /api/v1.0/logs HTTP/1.0" 200 -
  31. 2018-07-12 23:42:24,171 - Thread-51 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:24] "GET /api/v1.0/logs HTTP/1.0" 200 -
  32. 2018-07-12 23:42:26,189 - Thread-52 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:26] "GET /api/v1.0/logs HTTP/1.0" 200 -
  33. 2018-07-12 23:42:28,209 - Thread-53 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:28] "GET /api/v1.0/logs HTTP/1.0" 200 -
  34. 2018-07-12 23:42:30,233 - Thread-54 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:30] "GET /api/v1.0/logs HTTP/1.0" 200 -
  35. 2018-07-12 23:42:30,386 - Thread-26 - create_stack - INFO - Stack is now active with name - StorPerfAgentGroup
  36. 2018-07-12 23:42:30,862 - Thread-26 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  37. 2018-07-12 23:42:32,250 - Thread-55 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:32] "GET /api/v1.0/logs HTTP/1.0" 200 -
  38. 2018-07-12 23:42:34,269 - Thread-56 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:34] "GET /api/v1.0/logs HTTP/1.0" 200 -
  39. 2018-07-12 23:42:36,292 - Thread-57 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:36] "GET /api/v1.0/logs HTTP/1.0" 200 -
  40. 2018-07-12 23:42:36,358 - Thread-4 - create_instance - INFO - Found existing machine with name - storperf-agent
  41. 2018-07-12 23:42:36,417 - Thread-26 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.168
  42. 2018-07-12 23:42:38,001 - Thread-26 - storperf.storperf_master - DEBUG - Volume id f191b0a7-a9a2-4612-b9e3-fb465cb9d6cb, size=4, type=tripleo-ceph
  43. 2018-07-12 23:42:38,204 - Thread-26 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:38] "POST /api/v1.0/configurations HTTP/1.0" 200 -
  44. 2018-07-12 23:42:38,309 - Thread-59 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:38] "GET /api/v1.0/logs HTTP/1.0" 200 -
  45. 2018-07-12 23:42:38,357 - Thread-58 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:38] "GET /api/v1.0/configurations HTTP/1.0" 200 -
  46. 2018-07-12 23:42:48,376 - Thread-60 - __main__ - INFO - {u'queue_depths': u'4', u'block_sizes': u'4096k', u'workloads': {u'rs': {u'rw': u'read'}}, u'metadata': {u'build_tag': u'latest', u'test_case': u'opnfv_yardstick_tc074'}}
  47. 2018-07-12 23:42:48,377 - Thread-60 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  48. 2018-07-12 23:42:48,379 - Thread-60 - storperf.db.job_db - DEBUG - Job table exists
  49. 2018-07-12 23:42:48,379 - Thread-60 - storperf.db.job_db - DEBUG - Job params table exists
  50. 2018-07-12 23:42:48,380 - Thread-60 - storperf.db.job_db - DEBUG - Job summary table exists
  51. 2018-07-12 23:42:48,381 - Thread-60 - storperf.test_executor - ERROR - Cannot determine installer
  52. 2018-07-12 23:42:48,383 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.py'>
  53. 2018-07-12 23:42:48,384 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.py'>
  54. 2018-07-12 23:42:48,385 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.py'>
  55. 2018-07-12 23:42:48,385 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.py'>
  56. 2018-07-12 23:42:48,386 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.py'>
  57. 2018-07-12 23:42:48,386 - Thread-60 - storperf.storperf_master - INFO - Custom workloads = None
  58. 2018-07-12 23:42:48,387 - Thread-60 - storperf.storperf_master - INFO - Custom workloads = {u'rs': {u'rw': u'read'}}
  59. 2018-07-12 23:42:48,387 - Thread-60 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  60. 2018-07-12 23:42:48,388 - Thread-60 - storperf.db.job_db - DEBUG - Job table exists
  61. 2018-07-12 23:42:48,388 - Thread-60 - storperf.db.job_db - DEBUG - Job params table exists
  62. 2018-07-12 23:42:48,389 - Thread-60 - storperf.db.job_db - DEBUG - Job summary table exists
  63. 2018-07-12 23:42:48,389 - Thread-60 - storperf.test_executor - ERROR - Cannot determine installer
  64. 2018-07-12 23:42:48,390 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  65. 2018-07-12 23:42:48,390 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  66. 2018-07-12 23:42:48,391 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  67. 2018-07-12 23:42:48,391 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  68. 2018-07-12 23:42:48,392 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  69. 2018-07-12 23:42:48,392 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  70. 2018-07-12 23:42:48,393 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  71. 2018-07-12 23:42:48,393 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  72. 2018-07-12 23:42:48,393 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  73. 2018-07-12 23:42:48,394 - Thread-60 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  74. 2018-07-12 23:42:48,394 - Thread-60 - storperf.test_executor - DEBUG - Set custom workloads to: {u'rs': {u'rw': u'read'}}
  75. 2018-07-12 23:42:48,394 - Thread-60 - storperf.test_executor - DEBUG - Set block_sizes to: 4096k
  76. 2018-07-12 23:42:48,395 - Thread-60 - storperf.test_executor - DEBUG - Set queue_depths to: 4
  77. 2018-07-12 23:42:48,395 - Thread-61 - storperf.storperf_master:10.5.149.168 - INFO - Initializing slave at 10.5.149.168
  78. 2018-07-12 23:42:48,396 - Thread-61 - storperf.storperf_master:10.5.149.168 - DEBUG - Checking if slave 10.5.149.168 is alive
  79. 2018-07-12 23:42:48,399 - Thread-61 - storperf.storperf_master:10.5.149.168 - DEBUG - Slave 10.5.149.168 is alive and ready
  80. 2018-07-12 23:42:48,416 - Thread-62 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  81. 2018-07-12 23:42:48,555 - Thread-62 - paramiko.transport - INFO - Authentication (publickey) successful!
  82. 2018-07-12 23:42:48,817 - Thread-61 - storperf.storperf_master:10.5.149.168 - DEBUG - Available space on / is 2778924
  83. 2018-07-12 23:42:48,818 - Thread-61 - storperf.storperf_master:10.5.149.168 - DEBUG - Transferring fio to 10.5.149.168
  84. 2018-07-12 23:42:49,064 - Thread-60 - storperf.test_executor - DEBUG - Set slaves to: [u'10.5.149.168']
  85. 2018-07-12 23:42:49,065 - Thread-60 - storperf.test_executor - DEBUG - Set volume count to: 1
  86. 2018-07-12 23:42:49,067 - Thread-60 - storperf.db.job_db - DEBUG - Reserved job id 86fcb20d-0530-49f4-bf36-384d5fba25b1
  87. 2018-07-12 23:42:49,096 - Thread-60 - storperf.test_executor - INFO - Workload 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  88. 2018-07-12 23:42:49,097 - Thread-60 - storperf.test_executor - INFO - Testing FIO parameters for {'status': 'Pending', 'workload': <storperf.workloads._custom_workload._custom_workload object at 0x7ff6728a4a90>, 'name': u'86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k', 'workload_name': u'rs', 'blocksize': u'4096k', 'queue-depth': u'4'}
  89. 2018-07-12 23:42:49,099 - Thread-60 - storperf.test_executor - DEBUG - Device to profile on 10.5.149.168: /dev/vdb
  90. 2018-07-12 23:42:49,100 - Thread-60 - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7ff6728b2f50>
  91. 2018-07-12 23:42:49,100 - Thread-64 - storperf.test_executor - INFO - Starting 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  92. 2018-07-12 23:42:49,112 - Thread-68 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  93. 2018-07-12 23:42:49,224 - Thread-68 - paramiko.transport - INFO - Authentication (publickey) successful!
  94. 2018-07-12 23:42:49,236 - Thread-64 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60 --parse-only
  95. 2018-07-12 23:42:49,404 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  96. 2018-07-12 23:42:49,404 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  97. 2018-07-12 23:42:49,404 - Thread-64 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Started fio on 10.5.149.168
  98. 2018-07-12 23:42:49,976 - Thread-64 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Finished fio on 10.5.149.168 with exit code 0
  99. 2018-07-12 23:42:49,976 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  100. 2018-07-12 23:42:49,977 - Thread-64 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stderr handler
  101. 2018-07-12 23:42:49,979 - Thread-64 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stdout handler
  102. 2018-07-12 23:42:49,979 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  103. 2018-07-12 23:42:49,980 - Thread-64 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Ended
  104. 2018-07-12 23:42:50,094 - Thread-64 - storperf.test_executor - INFO - Ended 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  105. 2018-07-12 23:42:50,095 - Thread-60 - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7ff6728b2f50>, exit status None
  106. 2018-07-12 23:42:50,135 - Thread-60 - storperf.test_executor - ERROR - Cannot determine installer
  107. 2018-07-12 23:42:50,136 - Workload thread - storperf.test_executor - INFO - Starting job 86fcb20d-0530-49f4-bf36-384d5fba25b1
  108. 2018-07-12 23:42:50,137 - Thread-60 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:50] "POST /api/v2.0/jobs HTTP/1.0" 200 -
  109. 2018-07-12 23:42:50,138 - Workload thread - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  110. 2018-07-12 23:42:50,139 - Workload thread - storperf.db.job_db - DEBUG - Job table exists
  111. 2018-07-12 23:42:50,143 - Workload thread - storperf.db.job_db - DEBUG - Job params table exists
  112. 2018-07-12 23:42:50,143 - Workload thread - storperf.db.job_db - DEBUG - Job summary table exists
  113. 2018-07-12 23:42:50,144 - Workload thread - storperf.test_executor - INFO - Workload 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  114. 2018-07-12 23:42:50,145 - Workload thread - storperf.test_executor - INFO - Starting run 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k
  115. 2018-07-12 23:42:50,150 - Thread-70 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:42:50] "GET /api/v2.0/jobs?type=status&id=86fcb20d-0530-49f4-bf36-384d5fba25b1 HTTP/1.0" 404 -
  116. 2018-07-12 23:42:50,154 - Workload thread - storperf.test_executor - DEBUG - Device to profile on 10.5.149.168: /dev/vdb
  117. 2018-07-12 23:42:50,155 - Workload thread - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7ff67279a190>
  118. 2018-07-12 23:42:50,156 - Thread-71 - storperf.test_executor - INFO - Starting 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  119. 2018-07-12 23:42:50,192 - Thread-75 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  120. 2018-07-12 23:42:50,308 - Thread-75 - paramiko.transport - INFO - Authentication (publickey) successful!
  121. 2018-07-12 23:42:50,319 - Thread-71 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60
  122. 2018-07-12 23:42:50,454 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  123. 2018-07-12 23:42:50,455 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  124. 2018-07-12 23:42:50,455 - Thread-71 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Started fio on 10.5.149.168
  125. 2018-07-12 23:42:51,161 - Thread-77 - storperf.test_executor - INFO - Terminating current run
  126. 2018-07-12 23:42:51,162 - Thread-77 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Terminating fio on 10.5.149.168
  127. 2018-07-12 23:42:51,174 - Thread-78 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  128. 2018-07-12 23:42:51,290 - Thread-78 - paramiko.transport - INFO - Authentication (publickey) successful!
  129. 2018-07-12 23:42:51,297 - Thread-77 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Executing on 10.5.149.168: sudo killall fio
  130. 2018-07-12 23:42:51,522 - Thread-77 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Terminating fio on 10.5.149.168
  131. 2018-07-12 23:42:51,535 - Thread-80 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  132. 2018-07-12 23:42:51,614 - Thread-71 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Finished fio on 10.5.149.168 with exit code 128
  133. 2018-07-12 23:42:51,614 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  134. 2018-07-12 23:42:51,614 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  135. 2018-07-12 23:42:51,615 - Thread-71 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stderr handler
  136. 2018-07-12 23:42:51,617 - Thread-71 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stdout handler
  137. 2018-07-12 23:42:51,618 - Thread-71 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Ended
  138. 2018-07-12 23:42:51,656 - Thread-80 - paramiko.transport - INFO - Authentication (publickey) successful!
  139. 2018-07-12 23:42:51,658 - Thread-77 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Executing on 10.5.149.168: sudo killall fio
  140. 2018-07-12 23:42:51,836 - Thread-77 - storperf.fio.fio_invoker:10.5.149.168 - ERROR - fio: no process found
  141. 2018-07-12 23:42:51,865 - Thread-71 - storperf.test_executor - INFO - Ended 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  142. 2018-07-12 23:42:51,866 - Workload thread - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7ff67279a190>, exit status []
  143. 2018-07-12 23:42:51,866 - Workload thread - storperf.test_executor - INFO - Completed run 86fcb20d-0530-49f4-bf36-384d5fba25b1.rs.queue-depth.4.block-size.4096k
  144. 2018-07-12 23:42:51,866 - Workload thread - storperf.test_executor - INFO - Completed job 86fcb20d-0530-49f4-bf36-384d5fba25b1
  145. 2018-07-12 23:42:51,867 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method DataHandler.data_event of <storperf.utilities.data_handler.DataHandler object at 0x7ff6728b21d0>>
  146. 2018-07-12 23:42:51,867 - Workload thread - storperf.utilities.data_handler - DEBUG - Event received
  147. 2018-07-12 23:42:51,868 - Workload thread - storperf.test_executor - ERROR - While notifying listener 'steady_state'
  148. Traceback (most recent call last):
  149. File "/storperf/storperf/test_executor.py", line 170, in broadcast_event
  150. event_listener(self)
  151. File "/storperf/storperf/utilities/data_handler.py", line 35, in data_event
  152. self._push_to_db(executor)
  153. File "/storperf/storperf/utilities/data_handler.py", line 140, in _push_to_db
  154. criteria = self._determine_criteria(executor.metadata)
  155. File "/storperf/storperf/utilities/data_handler.py", line 163, in _determine_criteria
  156. for workload, value in metadata['details']['steady_state'].items():
  157. KeyError: 'steady_state'
  158. 2018-07-12 23:42:51,869 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method StorPerfMaster.executor_event of <storperf.storperf_master.StorPerfMaster object at 0x7ff67374d810>>
  159. 2018-07-12 23:42:51,952 - Thread-77 - create_stack - INFO - Deleting stack - StorPerfAgentGroup
  160. 2018-07-12 23:43:14,024 - Thread-77 - create_stack - INFO - Attempting to delete again stack - StorPerfAgentGroup
  161. 2018-07-12 23:43:14,117 - Thread-77 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:43:14] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  162. 2018-07-12 23:43:17,791 - Thread-82 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:43:17] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  163. 2018-07-12 23:43:17,889 - Thread-83 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:43:17] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  164. 2018-07-12 23:44:25,348 - Thread-84 - create_stack - INFO - Created stack with name - StorPerfAgentGroup
  165. 2018-07-12 23:45:43,721 - Thread-84 - create_stack - INFO - Stack is now active with name - StorPerfAgentGroup
  166. 2018-07-12 23:45:44,215 - Thread-84 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  167. 2018-07-12 23:45:49,864 - Thread-9 - create_instance - INFO - Found existing machine with name - storperf-agent
  168. 2018-07-12 23:45:49,952 - Thread-84 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.155
  169. 2018-07-12 23:45:51,474 - Thread-84 - storperf.storperf_master - DEBUG - Volume id 90b98dd1-2e17-4737-855a-e980ba0756af, size=4, type=tripleo-ceph
  170. 2018-07-12 23:45:51,601 - Thread-84 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:45:51] "POST /api/v1.0/configurations HTTP/1.0" 200 -
  171. 2018-07-12 23:52:48,266 - Thread-85 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  172. 2018-07-12 23:52:53,649 - Thread-14 - create_instance - INFO - Found existing machine with name - storperf-agent
  173. 2018-07-12 23:52:53,706 - Thread-85 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.155
  174. 2018-07-12 23:52:55,367 - Thread-85 - storperf.storperf_master - DEBUG - Volume id 90b98dd1-2e17-4737-855a-e980ba0756af, size=4, type=tripleo-ceph
  175. 2018-07-12 23:52:55,368 - Thread-85 - create_stack - INFO - Deleting stack - StorPerfAgentGroup
  176. 2018-07-12 23:53:14,924 - Thread-85 - create_stack - INFO - Attempting to delete again stack - StorPerfAgentGroup
  177. 2018-07-12 23:53:15,001 - Thread-85 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:53:15] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  178. 2018-07-12 23:55:33,300 - Thread-86 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:33] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  179. 2018-07-12 23:55:39,382 - Thread-88 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:39] "GET /api/v1.0/logs HTTP/1.0" 200 -
  180. 2018-07-12 23:55:41,401 - Thread-89 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:41] "GET /api/v1.0/logs HTTP/1.0" 200 -
  181. 2018-07-12 23:55:43,419 - Thread-90 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:43] "GET /api/v1.0/logs HTTP/1.0" 200 -
  182. 2018-07-12 23:55:44,182 - Thread-87 - create_stack - INFO - Created stack with name - StorPerfAgentGroup
  183. 2018-07-12 23:55:45,438 - Thread-91 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:45] "GET /api/v1.0/logs HTTP/1.0" 200 -
  184. 2018-07-12 23:55:47,455 - Thread-92 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:47] "GET /api/v1.0/logs HTTP/1.0" 200 -
  185. 2018-07-12 23:55:49,472 - Thread-93 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:49] "GET /api/v1.0/logs HTTP/1.0" 200 -
  186. 2018-07-12 23:55:51,490 - Thread-94 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:51] "GET /api/v1.0/logs HTTP/1.0" 200 -
  187. 2018-07-12 23:55:53,507 - Thread-95 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:53] "GET /api/v1.0/logs HTTP/1.0" 200 -
  188. 2018-07-12 23:55:55,529 - Thread-96 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:55] "GET /api/v1.0/logs HTTP/1.0" 200 -
  189. 2018-07-12 23:55:57,546 - Thread-97 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:57] "GET /api/v1.0/logs HTTP/1.0" 200 -
  190. 2018-07-12 23:55:59,564 - Thread-98 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:55:59] "GET /api/v1.0/logs HTTP/1.0" 200 -
  191. 2018-07-12 23:56:01,583 - Thread-99 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:01] "GET /api/v1.0/logs HTTP/1.0" 200 -
  192. 2018-07-12 23:56:03,602 - Thread-100 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:03] "GET /api/v1.0/logs HTTP/1.0" 200 -
  193. 2018-07-12 23:56:05,620 - Thread-101 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:05] "GET /api/v1.0/logs HTTP/1.0" 200 -
  194. 2018-07-12 23:56:07,637 - Thread-102 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:07] "GET /api/v1.0/logs HTTP/1.0" 200 -
  195. 2018-07-12 23:56:09,655 - Thread-103 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:09] "GET /api/v1.0/logs HTTP/1.0" 200 -
  196. 2018-07-12 23:56:11,680 - Thread-104 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:11] "GET /api/v1.0/logs HTTP/1.0" 200 -
  197. 2018-07-12 23:56:13,706 - Thread-105 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:13] "GET /api/v1.0/logs HTTP/1.0" 200 -
  198. 2018-07-12 23:56:15,724 - Thread-106 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:15] "GET /api/v1.0/logs HTTP/1.0" 200 -
  199. 2018-07-12 23:56:17,746 - Thread-107 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:17] "GET /api/v1.0/logs HTTP/1.0" 200 -
  200. 2018-07-12 23:56:19,764 - Thread-108 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:19] "GET /api/v1.0/logs HTTP/1.0" 200 -
  201. 2018-07-12 23:56:21,781 - Thread-109 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:21] "GET /api/v1.0/logs HTTP/1.0" 200 -
  202. 2018-07-12 23:56:23,799 - Thread-110 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:23] "GET /api/v1.0/logs HTTP/1.0" 200 -
  203. 2018-07-12 23:56:25,819 - Thread-111 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:25] "GET /api/v1.0/logs HTTP/1.0" 200 -
  204. 2018-07-12 23:56:27,836 - Thread-112 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:27] "GET /api/v1.0/logs HTTP/1.0" 200 -
  205. 2018-07-12 23:56:29,858 - Thread-113 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:29] "GET /api/v1.0/logs HTTP/1.0" 200 -
  206. 2018-07-12 23:56:31,876 - Thread-114 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:31] "GET /api/v1.0/logs HTTP/1.0" 200 -
  207. 2018-07-12 23:56:33,893 - Thread-115 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:33] "GET /api/v1.0/logs HTTP/1.0" 200 -
  208. 2018-07-12 23:56:35,915 - Thread-116 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:35] "GET /api/v1.0/logs HTTP/1.0" 200 -
  209. 2018-07-12 23:56:37,933 - Thread-117 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:37] "GET /api/v1.0/logs HTTP/1.0" 200 -
  210. 2018-07-12 23:56:39,951 - Thread-118 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:39] "GET /api/v1.0/logs HTTP/1.0" 200 -
  211. 2018-07-12 23:56:41,969 - Thread-119 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:41] "GET /api/v1.0/logs HTTP/1.0" 200 -
  212. 2018-07-12 23:56:43,986 - Thread-120 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:43] "GET /api/v1.0/logs HTTP/1.0" 200 -
  213. 2018-07-12 23:56:46,005 - Thread-121 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:46] "GET /api/v1.0/logs HTTP/1.0" 200 -
  214. 2018-07-12 23:56:48,023 - Thread-122 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:48] "GET /api/v1.0/logs HTTP/1.0" 200 -
  215. 2018-07-12 23:56:50,042 - Thread-123 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:50] "GET /api/v1.0/logs HTTP/1.0" 200 -
  216. 2018-07-12 23:56:52,061 - Thread-124 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:52] "GET /api/v1.0/logs HTTP/1.0" 200 -
  217. 2018-07-12 23:56:52,736 - Thread-87 - create_stack - INFO - Stack is now active with name - StorPerfAgentGroup
  218. 2018-07-12 23:56:53,214 - Thread-87 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  219. 2018-07-12 23:56:54,082 - Thread-125 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:54] "GET /api/v1.0/logs HTTP/1.0" 200 -
  220. 2018-07-12 23:56:56,105 - Thread-126 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:56] "GET /api/v1.0/logs HTTP/1.0" 200 -
  221. 2018-07-12 23:56:58,122 - Thread-127 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:56:58] "GET /api/v1.0/logs HTTP/1.0" 200 -
  222. 2018-07-12 23:56:58,520 - Thread-19 - create_instance - INFO - Found existing machine with name - storperf-agent
  223. 2018-07-12 23:56:58,579 - Thread-87 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.169
  224. 2018-07-12 23:56:59,931 - Thread-87 - storperf.storperf_master - DEBUG - Volume id 5da0082c-e3a8-46e3-b71d-23e657166919, size=4, type=tripleo-ceph
  225. 2018-07-12 23:57:00,143 - Thread-128 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:00] "GET /api/v1.0/logs HTTP/1.0" 200 -
  226. 2018-07-12 23:57:00,310 - Thread-87 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:00] "POST /api/v1.0/configurations HTTP/1.0" 200 -
  227. 2018-07-12 23:57:00,430 - Thread-129 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:00] "GET /api/v1.0/configurations HTTP/1.0" 200 -
  228. 2018-07-12 23:57:02,162 - Thread-130 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:02] "GET /api/v1.0/logs HTTP/1.0" 200 -
  229. 2018-07-12 23:57:10,450 - Thread-131 - __main__ - INFO - {u'queue_depths': u'4', u'block_sizes': u'4096k', u'workloads': {u'rs': {u'rw': u'read'}}, u'metadata': {u'build_tag': u'latest', u'test_case': u'opnfv_yardstick_tc074'}}
  230. 2018-07-12 23:57:10,451 - Thread-131 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  231. 2018-07-12 23:57:10,452 - Thread-131 - storperf.db.job_db - DEBUG - Job table exists
  232. 2018-07-12 23:57:10,453 - Thread-131 - storperf.db.job_db - DEBUG - Job params table exists
  233. 2018-07-12 23:57:10,453 - Thread-131 - storperf.db.job_db - DEBUG - Job summary table exists
  234. 2018-07-12 23:57:10,453 - Thread-131 - storperf.test_executor - ERROR - Cannot determine installer
  235. 2018-07-12 23:57:10,455 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  236. 2018-07-12 23:57:10,455 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  237. 2018-07-12 23:57:10,455 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  238. 2018-07-12 23:57:10,456 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  239. 2018-07-12 23:57:10,456 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  240. 2018-07-12 23:57:10,457 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  241. 2018-07-12 23:57:10,457 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  242. 2018-07-12 23:57:10,458 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  243. 2018-07-12 23:57:10,458 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  244. 2018-07-12 23:57:10,459 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  245. 2018-07-12 23:57:10,459 - Thread-131 - storperf.storperf_master - INFO - Custom workloads = None
  246. 2018-07-12 23:57:10,459 - Thread-131 - storperf.storperf_master - INFO - Custom workloads = {u'rs': {u'rw': u'read'}}
  247. 2018-07-12 23:57:10,460 - Thread-131 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  248. 2018-07-12 23:57:10,460 - Thread-131 - storperf.db.job_db - DEBUG - Job table exists
  249. 2018-07-12 23:57:10,461 - Thread-131 - storperf.db.job_db - DEBUG - Job params table exists
  250. 2018-07-12 23:57:10,461 - Thread-131 - storperf.db.job_db - DEBUG - Job summary table exists
  251. 2018-07-12 23:57:10,461 - Thread-131 - storperf.test_executor - ERROR - Cannot determine installer
  252. 2018-07-12 23:57:10,462 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  253. 2018-07-12 23:57:10,463 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  254. 2018-07-12 23:57:10,463 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  255. 2018-07-12 23:57:10,463 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  256. 2018-07-12 23:57:10,464 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  257. 2018-07-12 23:57:10,464 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  258. 2018-07-12 23:57:10,465 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  259. 2018-07-12 23:57:10,465 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  260. 2018-07-12 23:57:10,466 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  261. 2018-07-12 23:57:10,466 - Thread-131 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  262. 2018-07-12 23:57:10,466 - Thread-131 - storperf.test_executor - DEBUG - Set custom workloads to: {u'rs': {u'rw': u'read'}}
  263. 2018-07-12 23:57:10,467 - Thread-131 - storperf.test_executor - DEBUG - Set block_sizes to: 4096k
  264. 2018-07-12 23:57:10,467 - Thread-131 - storperf.test_executor - DEBUG - Set queue_depths to: 4
  265. 2018-07-12 23:57:10,468 - Thread-132 - storperf.storperf_master:10.5.149.169 - INFO - Initializing slave at 10.5.149.169
  266. 2018-07-12 23:57:10,468 - Thread-132 - storperf.storperf_master:10.5.149.169 - DEBUG - Checking if slave 10.5.149.169 is alive
  267. 2018-07-12 23:57:10,472 - Thread-132 - storperf.storperf_master:10.5.149.169 - DEBUG - Slave 10.5.149.169 is alive and ready
  268. 2018-07-12 23:57:10,487 - Thread-133 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  269. 2018-07-12 23:57:10,626 - Thread-133 - paramiko.transport - INFO - Authentication (publickey) successful!
  270. 2018-07-12 23:57:10,922 - Thread-132 - storperf.storperf_master:10.5.149.169 - DEBUG - Available space on / is 2778924
  271. 2018-07-12 23:57:10,923 - Thread-132 - storperf.storperf_master:10.5.149.169 - DEBUG - Transferring fio to 10.5.149.169
  272. 2018-07-12 23:57:11,144 - Thread-131 - storperf.test_executor - DEBUG - Set slaves to: [u'10.5.149.169']
  273. 2018-07-12 23:57:11,145 - Thread-131 - storperf.test_executor - DEBUG - Set volume count to: 1
  274. 2018-07-12 23:57:11,146 - Thread-131 - storperf.db.job_db - DEBUG - Reserved job id 51d912a8-e39c-4d0c-a46d-1e1621d0b33b
  275. 2018-07-12 23:57:11,168 - Thread-131 - storperf.test_executor - INFO - Workload 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  276. 2018-07-12 23:57:11,168 - Thread-131 - storperf.test_executor - INFO - Testing FIO parameters for {'status': 'Pending', 'workload': <storperf.workloads._custom_workload._custom_workload object at 0x7ff6725b0590>, 'name': u'51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k', 'workload_name': u'rs', 'blocksize': u'4096k', 'queue-depth': u'4'}
  277. 2018-07-12 23:57:11,170 - Thread-131 - storperf.test_executor - DEBUG - Device to profile on 10.5.149.169: /dev/vdb
  278. 2018-07-12 23:57:11,171 - Thread-131 - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7ff6727fa890>
  279. 2018-07-12 23:57:11,171 - Thread-135 - storperf.test_executor - INFO - Starting 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k.10-5-149-169--dev-vdb
  280. 2018-07-12 23:57:11,181 - Thread-139 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  281. 2018-07-12 23:57:11,294 - Thread-139 - paramiko.transport - INFO - Authentication (publickey) successful!
  282. 2018-07-12 23:57:11,308 - Thread-135 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60 --parse-only
  283. 2018-07-12 23:57:11,443 - 10.5.149.169 stdout - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Started
  284. 2018-07-12 23:57:11,444 - 10.5.149.169 stderr - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Started
  285. 2018-07-12 23:57:11,444 - Thread-135 - storperf.fio.fio_invoker:10.5.149.169 - INFO - Started fio on 10.5.149.169
  286. 2018-07-12 23:57:11,933 - Thread-135 - storperf.fio.fio_invoker:10.5.149.169 - INFO - Finished fio on 10.5.149.169 with exit code 0
  287. 2018-07-12 23:57:11,934 - 10.5.149.169 stdout - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Finished
  288. 2018-07-12 23:57:11,934 - Thread-135 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Joining stderr handler
  289. 2018-07-12 23:57:11,935 - 10.5.149.169 stderr - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Finished
  290. 2018-07-12 23:57:11,936 - Thread-135 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Joining stdout handler
  291. 2018-07-12 23:57:11,937 - Thread-135 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Ended
  292. 2018-07-12 23:57:12,051 - Thread-135 - storperf.test_executor - INFO - Ended 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k.10-5-149-169--dev-vdb
  293. 2018-07-12 23:57:12,052 - Thread-131 - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7ff6727fa890>, exit status None
  294. 2018-07-12 23:57:12,074 - Thread-131 - storperf.test_executor - ERROR - Cannot determine installer
  295. 2018-07-12 23:57:12,075 - Workload thread - storperf.test_executor - INFO - Starting job 51d912a8-e39c-4d0c-a46d-1e1621d0b33b
  296. 2018-07-12 23:57:12,076 - Thread-131 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:12] "POST /api/v2.0/jobs HTTP/1.0" 200 -
  297. 2018-07-12 23:57:12,077 - Workload thread - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  298. 2018-07-12 23:57:12,078 - Workload thread - storperf.db.job_db - DEBUG - Job table exists
  299. 2018-07-12 23:57:12,080 - Workload thread - storperf.db.job_db - DEBUG - Job params table exists
  300. 2018-07-12 23:57:12,080 - Workload thread - storperf.db.job_db - DEBUG - Job summary table exists
  301. 2018-07-12 23:57:12,081 - Workload thread - storperf.test_executor - INFO - Workload 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  302. 2018-07-12 23:57:12,082 - Workload thread - storperf.test_executor - INFO - Starting run 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k
  303. 2018-07-12 23:57:12,085 - Workload thread - storperf.test_executor - DEBUG - Device to profile on 10.5.149.169: /dev/vdb
  304. 2018-07-12 23:57:12,085 - Workload thread - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7ff672681290>
  305. 2018-07-12 23:57:12,086 - Thread-141 - storperf.test_executor - INFO - Starting 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k.10-5-149-169--dev-vdb
  306. 2018-07-12 23:57:12,088 - Thread-142 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:12] "GET /api/v2.0/jobs?type=status&id=51d912a8-e39c-4d0c-a46d-1e1621d0b33b HTTP/1.0" 404 -
  307. 2018-07-12 23:57:12,128 - Thread-146 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  308. 2018-07-12 23:57:12,245 - Thread-146 - paramiko.transport - INFO - Authentication (publickey) successful!
  309. 2018-07-12 23:57:12,255 - Thread-141 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60
  310. 2018-07-12 23:57:12,396 - 10.5.149.169 stdout - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Started
  311. 2018-07-12 23:57:12,397 - 10.5.149.169 stderr - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Started
  312. 2018-07-12 23:57:12,397 - Thread-141 - storperf.fio.fio_invoker:10.5.149.169 - INFO - Started fio on 10.5.149.169
  313. 2018-07-12 23:57:13,099 - Thread-148 - storperf.test_executor - INFO - Terminating current run
  314. 2018-07-12 23:57:13,099 - Thread-148 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Terminating fio on 10.5.149.169
  315. 2018-07-12 23:57:13,110 - Thread-149 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  316. 2018-07-12 23:57:13,223 - Thread-149 - paramiko.transport - INFO - Authentication (publickey) successful!
  317. 2018-07-12 23:57:13,235 - Thread-148 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Executing on 10.5.149.169: sudo killall fio
  318. 2018-07-12 23:57:14,140 - Thread-148 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Terminating fio on 10.5.149.169
  319. 2018-07-12 23:57:14,148 - Thread-151 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  320. 2018-07-12 23:57:14,223 - Thread-141 - storperf.fio.fio_invoker:10.5.149.169 - INFO - Finished fio on 10.5.149.169 with exit code 128
  321. 2018-07-12 23:57:14,223 - 10.5.149.169 stderr - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Finished
  322. 2018-07-12 23:57:14,224 - 10.5.149.169 stdout - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Finished
  323. 2018-07-12 23:57:14,224 - Thread-141 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Joining stderr handler
  324. 2018-07-12 23:57:14,226 - Thread-141 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Joining stdout handler
  325. 2018-07-12 23:57:14,226 - Thread-141 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Ended
  326. 2018-07-12 23:57:14,270 - Thread-151 - paramiko.transport - INFO - Authentication (publickey) successful!
  327. 2018-07-12 23:57:14,282 - Thread-148 - storperf.fio.fio_invoker:10.5.149.169 - DEBUG - Executing on 10.5.149.169: sudo killall fio
  328. 2018-07-12 23:57:14,427 - Thread-148 - storperf.fio.fio_invoker:10.5.149.169 - ERROR - fio: no process found
  329. 2018-07-12 23:57:14,543 - Thread-148 - create_stack - INFO - Deleting stack - StorPerfAgentGroup
  330. 2018-07-12 23:57:17,801 - Thread-141 - storperf.test_executor - INFO - Ended 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k.10-5-149-169--dev-vdb
  331. 2018-07-12 23:57:17,801 - Workload thread - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7ff672681290>, exit status []
  332. 2018-07-12 23:57:17,802 - Workload thread - storperf.test_executor - INFO - Completed run 51d912a8-e39c-4d0c-a46d-1e1621d0b33b.rs.queue-depth.4.block-size.4096k
  333. 2018-07-12 23:57:17,802 - Workload thread - storperf.test_executor - INFO - Completed job 51d912a8-e39c-4d0c-a46d-1e1621d0b33b
  334. 2018-07-12 23:57:17,803 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method DataHandler.data_event of <storperf.utilities.data_handler.DataHandler object at 0x7ff6727fa890>>
  335. 2018-07-12 23:57:17,803 - Workload thread - storperf.utilities.data_handler - DEBUG - Event received
  336. 2018-07-12 23:57:17,803 - Workload thread - storperf.test_executor - ERROR - While notifying listener 'steady_state'
  337. Traceback (most recent call last):
  338. File "/storperf/storperf/test_executor.py", line 170, in broadcast_event
  339. event_listener(self)
  340. File "/storperf/storperf/utilities/data_handler.py", line 35, in data_event
  341. self._push_to_db(executor)
  342. File "/storperf/storperf/utilities/data_handler.py", line 140, in _push_to_db
  343. criteria = self._determine_criteria(executor.metadata)
  344. File "/storperf/storperf/utilities/data_handler.py", line 163, in _determine_criteria
  345. for workload, value in metadata['details']['steady_state'].items():
  346. KeyError: 'steady_state'
  347. 2018-07-12 23:57:17,804 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method StorPerfMaster.executor_event of <storperf.storperf_master.StorPerfMaster object at 0x7ff67374d810>>
  348. 2018-07-12 23:57:36,663 - Thread-148 - create_stack - INFO - Attempting to delete again stack - StorPerfAgentGroup
  349. 2018-07-12 23:57:36,742 - Thread-148 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:36] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  350. 2018-07-12 23:57:40,849 - Thread-153 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:40] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  351. 2018-07-12 23:57:40,984 - Thread-154 - werkzeug - INFO - 172.18.0.6 - - [12/Jul/2018 23:57:40] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  352. 2018-07-13 00:02:39,602 - Thread-155 - create_stack - INFO - Created stack with name - StorPerfAgentGroup
  353. 2018-07-13 00:04:01,908 - Thread-155 - create_stack - INFO - Stack is now active with name - StorPerfAgentGroup
  354. 2018-07-13 00:04:02,455 - Thread-155 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  355. 2018-07-13 00:04:07,716 - Thread-4 - create_instance - INFO - Found existing machine with name - storperf-agent
  356. 2018-07-13 00:04:07,778 - Thread-155 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.169
  357. 2018-07-13 00:04:08,979 - Thread-155 - storperf.storperf_master - DEBUG - Volume id b5461938-f8ec-497b-ab0f-8a22ded4005a, size=4, type=tripleo-ceph
  358. 2018-07-13 00:04:09,150 - Thread-155 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:04:09] "POST /api/v1.0/configurations HTTP/1.0" 200 -
  359. 2018-07-13 00:13:01,102 - Thread-156 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  360. 2018-07-13 00:13:06,876 - Thread-9 - create_instance - INFO - Found existing machine with name - storperf-agent
  361. 2018-07-13 00:13:06,954 - Thread-156 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.169
  362. 2018-07-13 00:13:08,183 - Thread-156 - storperf.storperf_master - DEBUG - Volume id b5461938-f8ec-497b-ab0f-8a22ded4005a, size=4, type=tripleo-ceph
  363. 2018-07-13 00:13:08,222 - Thread-156 - create_stack - INFO - Deleting stack - StorPerfAgentGroup
  364. 2018-07-13 00:13:30,585 - Thread-156 - create_stack - INFO - Attempting to delete again stack - StorPerfAgentGroup
  365. 2018-07-13 00:13:30,666 - Thread-156 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:13:30] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  366. 2018-07-13 00:14:13,798 - Thread-157 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:13] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  367. 2018-07-13 00:14:19,893 - Thread-159 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:19] "GET /api/v1.0/logs HTTP/1.0" 200 -
  368. 2018-07-13 00:14:21,916 - Thread-160 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:21] "GET /api/v1.0/logs HTTP/1.0" 200 -
  369. 2018-07-13 00:14:23,934 - Thread-161 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:23] "GET /api/v1.0/logs HTTP/1.0" 200 -
  370. 2018-07-13 00:14:24,674 - Thread-158 - create_stack - INFO - Created stack with name - StorPerfAgentGroup
  371. 2018-07-13 00:14:25,951 - Thread-162 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:25] "GET /api/v1.0/logs HTTP/1.0" 200 -
  372. 2018-07-13 00:14:27,975 - Thread-163 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:27] "GET /api/v1.0/logs HTTP/1.0" 200 -
  373. 2018-07-13 00:14:29,994 - Thread-164 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:29] "GET /api/v1.0/logs HTTP/1.0" 200 -
  374. 2018-07-13 00:14:32,010 - Thread-165 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:32] "GET /api/v1.0/logs HTTP/1.0" 200 -
  375. 2018-07-13 00:14:34,029 - Thread-166 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:34] "GET /api/v1.0/logs HTTP/1.0" 200 -
  376. 2018-07-13 00:14:36,052 - Thread-167 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:36] "GET /api/v1.0/logs HTTP/1.0" 200 -
  377. 2018-07-13 00:14:38,070 - Thread-168 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:38] "GET /api/v1.0/logs HTTP/1.0" 200 -
  378. 2018-07-13 00:14:40,089 - Thread-169 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:40] "GET /api/v1.0/logs HTTP/1.0" 200 -
  379. 2018-07-13 00:14:42,108 - Thread-170 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:42] "GET /api/v1.0/logs HTTP/1.0" 200 -
  380. 2018-07-13 00:14:44,131 - Thread-171 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:44] "GET /api/v1.0/logs HTTP/1.0" 200 -
  381. 2018-07-13 00:14:46,152 - Thread-172 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:46] "GET /api/v1.0/logs HTTP/1.0" 200 -
  382. 2018-07-13 00:14:48,168 - Thread-173 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:48] "GET /api/v1.0/logs HTTP/1.0" 200 -
  383. 2018-07-13 00:14:50,188 - Thread-174 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:50] "GET /api/v1.0/logs HTTP/1.0" 200 -
  384. 2018-07-13 00:14:52,210 - Thread-175 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:52] "GET /api/v1.0/logs HTTP/1.0" 200 -
  385. 2018-07-13 00:14:54,228 - Thread-176 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:54] "GET /api/v1.0/logs HTTP/1.0" 200 -
  386. 2018-07-13 00:14:56,249 - Thread-177 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:56] "GET /api/v1.0/logs HTTP/1.0" 200 -
  387. 2018-07-13 00:14:58,271 - Thread-178 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:14:58] "GET /api/v1.0/logs HTTP/1.0" 200 -
  388. 2018-07-13 00:15:00,289 - Thread-179 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:00] "GET /api/v1.0/logs HTTP/1.0" 200 -
  389. 2018-07-13 00:15:02,312 - Thread-180 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:02] "GET /api/v1.0/logs HTTP/1.0" 200 -
  390. 2018-07-13 00:15:04,329 - Thread-181 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:04] "GET /api/v1.0/logs HTTP/1.0" 200 -
  391. 2018-07-13 00:15:06,347 - Thread-182 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:06] "GET /api/v1.0/logs HTTP/1.0" 200 -
  392. 2018-07-13 00:15:08,365 - Thread-183 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:08] "GET /api/v1.0/logs HTTP/1.0" 200 -
  393. 2018-07-13 00:15:10,381 - Thread-184 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:10] "GET /api/v1.0/logs HTTP/1.0" 200 -
  394. 2018-07-13 00:15:12,401 - Thread-185 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:12] "GET /api/v1.0/logs HTTP/1.0" 200 -
  395. 2018-07-13 00:15:14,420 - Thread-186 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:14] "GET /api/v1.0/logs HTTP/1.0" 200 -
  396. 2018-07-13 00:15:16,438 - Thread-187 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:16] "GET /api/v1.0/logs HTTP/1.0" 200 -
  397. 2018-07-13 00:15:18,456 - Thread-188 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:18] "GET /api/v1.0/logs HTTP/1.0" 200 -
  398. 2018-07-13 00:15:20,172 - Thread-158 - create_stack - INFO - Stack is now active with name - StorPerfAgentGroup
  399. 2018-07-13 00:15:20,475 - Thread-189 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:20] "GET /api/v1.0/logs HTTP/1.0" 200 -
  400. 2018-07-13 00:15:21,047 - Thread-158 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  401. 2018-07-13 00:15:22,492 - Thread-190 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:22] "GET /api/v1.0/logs HTTP/1.0" 200 -
  402. 2018-07-13 00:15:24,511 - Thread-191 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:24] "GET /api/v1.0/logs HTTP/1.0" 200 -
  403. 2018-07-13 00:15:26,527 - Thread-192 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:26] "GET /api/v1.0/logs HTTP/1.0" 200 -
  404. 2018-07-13 00:15:28,115 - Thread-11 - create_instance - INFO - Found existing machine with name - storperf-agent
  405. 2018-07-13 00:15:28,196 - Thread-158 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.168
  406. 2018-07-13 00:15:28,546 - Thread-193 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:28] "GET /api/v1.0/logs HTTP/1.0" 200 -
  407. 2018-07-13 00:15:29,377 - Thread-158 - storperf.storperf_master - DEBUG - Volume id 6e2e3ed3-c595-402d-a356-401da01040fa, size=4, type=tripleo-ceph
  408. 2018-07-13 00:15:29,504 - Thread-158 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:29] "POST /api/v1.0/configurations HTTP/1.0" 200 -
  409. 2018-07-13 00:15:29,659 - Thread-194 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:29] "GET /api/v1.0/configurations HTTP/1.0" 200 -
  410. 2018-07-13 00:15:30,568 - Thread-195 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:30] "GET /api/v1.0/logs HTTP/1.0" 200 -
  411. 2018-07-13 00:15:39,678 - Thread-196 - __main__ - INFO - {u'queue_depths': u'4', u'block_sizes': u'4096k', u'workloads': {u'rs': {u'rw': u'read'}}, u'metadata': {u'build_tag': u'latest', u'test_case': u'opnfv_yardstick_tc074'}}
  412. 2018-07-13 00:15:39,678 - Thread-196 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  413. 2018-07-13 00:15:39,679 - Thread-196 - storperf.db.job_db - DEBUG - Job table exists
  414. 2018-07-13 00:15:39,680 - Thread-196 - storperf.db.job_db - DEBUG - Job params table exists
  415. 2018-07-13 00:15:39,680 - Thread-196 - storperf.db.job_db - DEBUG - Job summary table exists
  416. 2018-07-13 00:15:39,680 - Thread-196 - storperf.test_executor - ERROR - Cannot determine installer
  417. 2018-07-13 00:15:39,681 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  418. 2018-07-13 00:15:39,682 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  419. 2018-07-13 00:15:39,682 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  420. 2018-07-13 00:15:39,683 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  421. 2018-07-13 00:15:39,683 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  422. 2018-07-13 00:15:39,684 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  423. 2018-07-13 00:15:39,684 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  424. 2018-07-13 00:15:39,685 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  425. 2018-07-13 00:15:39,685 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  426. 2018-07-13 00:15:39,685 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  427. 2018-07-13 00:15:39,686 - Thread-196 - storperf.storperf_master - INFO - Custom workloads = None
  428. 2018-07-13 00:15:39,686 - Thread-196 - storperf.storperf_master - INFO - Custom workloads = {u'rs': {u'rw': u'read'}}
  429. 2018-07-13 00:15:39,687 - Thread-196 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  430. 2018-07-13 00:15:39,687 - Thread-196 - storperf.db.job_db - DEBUG - Job table exists
  431. 2018-07-13 00:15:39,687 - Thread-196 - storperf.db.job_db - DEBUG - Job params table exists
  432. 2018-07-13 00:15:39,688 - Thread-196 - storperf.db.job_db - DEBUG - Job summary table exists
  433. 2018-07-13 00:15:39,688 - Thread-196 - storperf.test_executor - ERROR - Cannot determine installer
  434. 2018-07-13 00:15:39,689 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  435. 2018-07-13 00:15:39,689 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  436. 2018-07-13 00:15:39,690 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  437. 2018-07-13 00:15:39,690 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  438. 2018-07-13 00:15:39,691 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  439. 2018-07-13 00:15:39,691 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  440. 2018-07-13 00:15:39,691 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  441. 2018-07-13 00:15:39,692 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  442. 2018-07-13 00:15:39,692 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  443. 2018-07-13 00:15:39,693 - Thread-196 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  444. 2018-07-13 00:15:39,693 - Thread-196 - storperf.test_executor - DEBUG - Set custom workloads to: {u'rs': {u'rw': u'read'}}
  445. 2018-07-13 00:15:39,693 - Thread-196 - storperf.test_executor - DEBUG - Set block_sizes to: 4096k
  446. 2018-07-13 00:15:39,694 - Thread-196 - storperf.test_executor - DEBUG - Set queue_depths to: 4
  447. 2018-07-13 00:15:39,694 - Thread-197 - storperf.storperf_master:10.5.149.168 - INFO - Initializing slave at 10.5.149.168
  448. 2018-07-13 00:15:39,695 - Thread-197 - storperf.storperf_master:10.5.149.168 - DEBUG - Checking if slave 10.5.149.168 is alive
  449. 2018-07-13 00:15:39,698 - Thread-197 - storperf.storperf_master:10.5.149.168 - DEBUG - Slave 10.5.149.168 is alive and ready
  450. 2018-07-13 00:15:39,713 - Thread-198 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  451. 2018-07-13 00:15:39,856 - Thread-198 - paramiko.transport - INFO - Authentication (publickey) successful!
  452. 2018-07-13 00:15:40,116 - Thread-197 - storperf.storperf_master:10.5.149.168 - DEBUG - Available space on / is 2778944
  453. 2018-07-13 00:15:40,117 - Thread-197 - storperf.storperf_master:10.5.149.168 - DEBUG - Transferring fio to 10.5.149.168
  454. 2018-07-13 00:15:40,289 - Thread-196 - storperf.test_executor - DEBUG - Set slaves to: [u'10.5.149.168']
  455. 2018-07-13 00:15:40,290 - Thread-196 - storperf.test_executor - DEBUG - Set volume count to: 1
  456. 2018-07-13 00:15:40,291 - Thread-196 - storperf.db.job_db - DEBUG - Reserved job id 4b4212e2-e408-4d78-bc4a-8bc303d8190b
  457. 2018-07-13 00:15:40,313 - Thread-196 - storperf.test_executor - INFO - Workload 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  458. 2018-07-13 00:15:40,313 - Thread-196 - storperf.test_executor - INFO - Testing FIO parameters for {'status': 'Pending', 'workload': <storperf.workloads._custom_workload._custom_workload object at 0x7ff6727ebc10>, 'name': u'4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k', 'workload_name': u'rs', 'blocksize': u'4096k', 'queue-depth': u'4'}
  459. 2018-07-13 00:15:40,315 - Thread-196 - storperf.test_executor - DEBUG - Device to profile on 10.5.149.168: /dev/vdb
  460. 2018-07-13 00:15:40,316 - Thread-196 - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7ff6725c87d0>
  461. 2018-07-13 00:15:40,316 - Thread-200 - storperf.test_executor - INFO - Starting 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  462. 2018-07-13 00:15:40,325 - Thread-204 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  463. 2018-07-13 00:15:40,443 - Thread-204 - paramiko.transport - INFO - Authentication (publickey) successful!
  464. 2018-07-13 00:15:40,452 - Thread-200 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60 --parse-only
  465. 2018-07-13 00:15:40,619 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  466. 2018-07-13 00:15:40,620 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  467. 2018-07-13 00:15:40,623 - Thread-200 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Started fio on 10.5.149.168
  468. 2018-07-13 00:15:41,271 - Thread-200 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Finished fio on 10.5.149.168 with exit code 0
  469. 2018-07-13 00:15:41,272 - Thread-200 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stderr handler
  470. 2018-07-13 00:15:41,272 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  471. 2018-07-13 00:15:41,273 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  472. 2018-07-13 00:15:41,275 - Thread-200 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stdout handler
  473. 2018-07-13 00:15:41,275 - Thread-200 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Ended
  474. 2018-07-13 00:15:41,390 - Thread-200 - storperf.test_executor - INFO - Ended 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  475. 2018-07-13 00:15:41,390 - Thread-196 - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7ff6725c87d0>, exit status None
  476. 2018-07-13 00:15:41,440 - Thread-196 - storperf.test_executor - ERROR - Cannot determine installer
  477. 2018-07-13 00:15:41,441 - Workload thread - storperf.test_executor - INFO - Starting job 4b4212e2-e408-4d78-bc4a-8bc303d8190b
  478. 2018-07-13 00:15:41,442 - Thread-196 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:41] "POST /api/v2.0/jobs HTTP/1.0" 200 -
  479. 2018-07-13 00:15:41,443 - Workload thread - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  480. 2018-07-13 00:15:41,443 - Workload thread - storperf.db.job_db - DEBUG - Job table exists
  481. 2018-07-13 00:15:41,444 - Workload thread - storperf.db.job_db - DEBUG - Job params table exists
  482. 2018-07-13 00:15:41,448 - Workload thread - storperf.db.job_db - DEBUG - Job summary table exists
  483. 2018-07-13 00:15:41,449 - Workload thread - storperf.test_executor - INFO - Workload 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  484. 2018-07-13 00:15:41,451 - Thread-206 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:15:41] "GET /api/v2.0/jobs?type=status&id=4b4212e2-e408-4d78-bc4a-8bc303d8190b HTTP/1.0" 404 -
  485. 2018-07-13 00:15:41,452 - Workload thread - storperf.test_executor - INFO - Starting run 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k
  486. 2018-07-13 00:15:41,457 - Workload thread - storperf.test_executor - DEBUG - Device to profile on 10.5.149.168: /dev/vdb
  487. 2018-07-13 00:15:41,457 - Workload thread - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7ff672858f90>
  488. 2018-07-13 00:15:41,458 - Thread-207 - storperf.test_executor - INFO - Starting 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  489. 2018-07-13 00:15:41,508 - Thread-211 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  490. 2018-07-13 00:15:41,622 - Thread-211 - paramiko.transport - INFO - Authentication (publickey) successful!
  491. 2018-07-13 00:15:41,635 - Thread-207 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60
  492. 2018-07-13 00:15:41,770 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  493. 2018-07-13 00:15:41,771 - Thread-207 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Started fio on 10.5.149.168
  494. 2018-07-13 00:15:41,771 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Started
  495. 2018-07-13 00:15:42,464 - Thread-213 - storperf.test_executor - INFO - Terminating current run
  496. 2018-07-13 00:15:42,464 - Thread-213 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Terminating fio on 10.5.149.168
  497. 2018-07-13 00:15:42,473 - Thread-214 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  498. 2018-07-13 00:15:42,587 - Thread-214 - paramiko.transport - INFO - Authentication (publickey) successful!
  499. 2018-07-13 00:15:42,599 - Thread-213 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Executing on 10.5.149.168: sudo killall fio
  500. 2018-07-13 00:15:42,829 - Thread-213 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Terminating fio on 10.5.149.168
  501. 2018-07-13 00:15:42,838 - Thread-216 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  502. 2018-07-13 00:15:42,910 - Thread-207 - storperf.fio.fio_invoker:10.5.149.168 - INFO - Finished fio on 10.5.149.168 with exit code 128
  503. 2018-07-13 00:15:42,911 - Thread-207 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stderr handler
  504. 2018-07-13 00:15:42,912 - 10.5.149.168 stdout - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  505. 2018-07-13 00:15:42,913 - 10.5.149.168 stderr - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Finished
  506. 2018-07-13 00:15:42,916 - Thread-207 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Joining stdout handler
  507. 2018-07-13 00:15:42,916 - Thread-207 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Ended
  508. 2018-07-13 00:15:42,958 - Thread-216 - paramiko.transport - INFO - Authentication (publickey) successful!
  509. 2018-07-13 00:15:42,965 - Thread-213 - storperf.fio.fio_invoker:10.5.149.168 - DEBUG - Executing on 10.5.149.168: sudo killall fio
  510. 2018-07-13 00:15:43,110 - Thread-213 - storperf.fio.fio_invoker:10.5.149.168 - ERROR - fio: no process found
  511. 2018-07-13 00:15:43,113 - Thread-213 - create_stack - INFO - Deleting stack - StorPerfAgentGroup
  512. 2018-07-13 00:15:43,118 - Thread-207 - storperf.test_executor - INFO - Ended 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k.10-5-149-168--dev-vdb
  513. 2018-07-13 00:15:43,121 - Workload thread - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7ff672858f90>, exit status []
  514. 2018-07-13 00:15:43,121 - Workload thread - storperf.test_executor - INFO - Completed run 4b4212e2-e408-4d78-bc4a-8bc303d8190b.rs.queue-depth.4.block-size.4096k
  515. 2018-07-13 00:15:43,122 - Workload thread - storperf.test_executor - INFO - Completed job 4b4212e2-e408-4d78-bc4a-8bc303d8190b
  516. 2018-07-13 00:15:43,122 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method DataHandler.data_event of <storperf.utilities.data_handler.DataHandler object at 0x7ff6727eb910>>
  517. 2018-07-13 00:15:43,123 - Workload thread - storperf.utilities.data_handler - DEBUG - Event received
  518. 2018-07-13 00:15:43,123 - Workload thread - storperf.test_executor - ERROR - While notifying listener 'steady_state'
  519. Traceback (most recent call last):
  520. File "/storperf/storperf/test_executor.py", line 170, in broadcast_event
  521. event_listener(self)
  522. File "/storperf/storperf/utilities/data_handler.py", line 35, in data_event
  523. self._push_to_db(executor)
  524. File "/storperf/storperf/utilities/data_handler.py", line 140, in _push_to_db
  525. criteria = self._determine_criteria(executor.metadata)
  526. File "/storperf/storperf/utilities/data_handler.py", line 163, in _determine_criteria
  527. for workload, value in metadata['details']['steady_state'].items():
  528. KeyError: 'steady_state'
  529. 2018-07-13 00:15:43,124 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method StorPerfMaster.executor_event of <storperf.storperf_master.StorPerfMaster object at 0x7ff67374d810>>
  530. 2018-07-13 00:16:05,084 - Thread-213 - create_stack - INFO - Attempting to delete again stack - StorPerfAgentGroup
  531. 2018-07-13 00:16:05,179 - Thread-213 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:16:05] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  532. 2018-07-13 00:16:09,223 - Thread-218 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:16:09] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  533. 2018-07-13 00:16:09,327 - Thread-219 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 00:16:09] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  534. 2018-07-13 12:48:30,425 - MainThread - werkzeug - INFO - * Detected change in '/storperf/storperf/fio/fio_invoker.py', reloading
  535. 2018-07-13 12:48:31,094 - MainThread - werkzeug - INFO - * Restarting with stat
  536. 2018-07-13 12:48:32,203 - MainThread - werkzeug - WARNING - * Debugger is active!
  537. 2018-07-13 12:48:32,204 - MainThread - werkzeug - INFO - * Debugger PIN: 116-097-559
  538. 2018-07-13 12:49:38,536 - Thread-25 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:38] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  539. 2018-07-13 12:49:44,628 - Thread-27 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:44] "GET /api/v1.0/logs HTTP/1.0" 200 -
  540. 2018-07-13 12:49:46,646 - Thread-28 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:46] "GET /api/v1.0/logs HTTP/1.0" 200 -
  541. 2018-07-13 12:49:48,661 - Thread-29 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:48] "GET /api/v1.0/logs HTTP/1.0" 200 -
  542. 2018-07-13 12:49:50,424 - Thread-26 - create_stack - INFO - Created stack with name - StorPerfAgentGroup
  543. 2018-07-13 12:49:50,679 - Thread-30 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:50] "GET /api/v1.0/logs HTTP/1.0" 200 -
  544. 2018-07-13 12:49:52,699 - Thread-31 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:52] "GET /api/v1.0/logs HTTP/1.0" 200 -
  545. 2018-07-13 12:49:54,727 - Thread-32 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:54] "GET /api/v1.0/logs HTTP/1.0" 200 -
  546. 2018-07-13 12:49:56,747 - Thread-33 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:56] "GET /api/v1.0/logs HTTP/1.0" 200 -
  547. 2018-07-13 12:49:58,767 - Thread-34 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:49:58] "GET /api/v1.0/logs HTTP/1.0" 200 -
  548. 2018-07-13 12:50:00,789 - Thread-35 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:00] "GET /api/v1.0/logs HTTP/1.0" 200 -
  549. 2018-07-13 12:50:02,811 - Thread-36 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:02] "GET /api/v1.0/logs HTTP/1.0" 200 -
  550. 2018-07-13 12:50:04,830 - Thread-37 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:04] "GET /api/v1.0/logs HTTP/1.0" 200 -
  551. 2018-07-13 12:50:06,847 - Thread-38 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:06] "GET /api/v1.0/logs HTTP/1.0" 200 -
  552. 2018-07-13 12:50:08,867 - Thread-39 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:08] "GET /api/v1.0/logs HTTP/1.0" 200 -
  553. 2018-07-13 12:50:10,886 - Thread-40 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:10] "GET /api/v1.0/logs HTTP/1.0" 200 -
  554. 2018-07-13 12:50:12,904 - Thread-41 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:12] "GET /api/v1.0/logs HTTP/1.0" 200 -
  555. 2018-07-13 12:50:14,922 - Thread-42 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:14] "GET /api/v1.0/logs HTTP/1.0" 200 -
  556. 2018-07-13 12:50:16,941 - Thread-43 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:16] "GET /api/v1.0/logs HTTP/1.0" 200 -
  557. 2018-07-13 12:50:18,968 - Thread-44 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:18] "GET /api/v1.0/logs HTTP/1.0" 200 -
  558. 2018-07-13 12:50:20,985 - Thread-45 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:20] "GET /api/v1.0/logs HTTP/1.0" 200 -
  559. 2018-07-13 12:50:23,004 - Thread-46 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:23] "GET /api/v1.0/logs HTTP/1.0" 200 -
  560. 2018-07-13 12:50:25,025 - Thread-47 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:25] "GET /api/v1.0/logs HTTP/1.0" 200 -
  561. 2018-07-13 12:50:27,047 - Thread-48 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:27] "GET /api/v1.0/logs HTTP/1.0" 200 -
  562. 2018-07-13 12:50:29,066 - Thread-49 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:29] "GET /api/v1.0/logs HTTP/1.0" 200 -
  563. 2018-07-13 12:50:31,084 - Thread-50 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:31] "GET /api/v1.0/logs HTTP/1.0" 200 -
  564. 2018-07-13 12:50:33,101 - Thread-51 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:33] "GET /api/v1.0/logs HTTP/1.0" 200 -
  565. 2018-07-13 12:50:35,122 - Thread-52 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:35] "GET /api/v1.0/logs HTTP/1.0" 200 -
  566. 2018-07-13 12:50:37,140 - Thread-53 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:37] "GET /api/v1.0/logs HTTP/1.0" 200 -
  567. 2018-07-13 12:50:39,161 - Thread-54 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:39] "GET /api/v1.0/logs HTTP/1.0" 200 -
  568. 2018-07-13 12:50:41,184 - Thread-55 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:41] "GET /api/v1.0/logs HTTP/1.0" 200 -
  569. 2018-07-13 12:50:43,204 - Thread-56 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:43] "GET /api/v1.0/logs HTTP/1.0" 200 -
  570. 2018-07-13 12:50:45,223 - Thread-57 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:45] "GET /api/v1.0/logs HTTP/1.0" 200 -
  571. 2018-07-13 12:50:47,240 - Thread-58 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:47] "GET /api/v1.0/logs HTTP/1.0" 200 -
  572. 2018-07-13 12:50:49,264 - Thread-59 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:49] "GET /api/v1.0/logs HTTP/1.0" 200 -
  573. 2018-07-13 12:50:51,282 - Thread-60 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:51] "GET /api/v1.0/logs HTTP/1.0" 200 -
  574. 2018-07-13 12:50:53,300 - Thread-61 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:53] "GET /api/v1.0/logs HTTP/1.0" 200 -
  575. 2018-07-13 12:50:55,318 - Thread-62 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:55] "GET /api/v1.0/logs HTTP/1.0" 200 -
  576. 2018-07-13 12:50:57,338 - Thread-63 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:57] "GET /api/v1.0/logs HTTP/1.0" 200 -
  577. 2018-07-13 12:50:59,354 - Thread-64 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:50:59] "GET /api/v1.0/logs HTTP/1.0" 200 -
  578. 2018-07-13 12:51:01,370 - Thread-65 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:01] "GET /api/v1.0/logs HTTP/1.0" 200 -
  579. 2018-07-13 12:51:03,391 - Thread-66 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:03] "GET /api/v1.0/logs HTTP/1.0" 200 -
  580. 2018-07-13 12:51:05,408 - Thread-67 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:05] "GET /api/v1.0/logs HTTP/1.0" 200 -
  581. 2018-07-13 12:51:07,426 - Thread-68 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:07] "GET /api/v1.0/logs HTTP/1.0" 200 -
  582. 2018-07-13 12:51:09,446 - Thread-69 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:09] "GET /api/v1.0/logs HTTP/1.0" 200 -
  583. 2018-07-13 12:51:11,464 - Thread-70 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:11] "GET /api/v1.0/logs HTTP/1.0" 200 -
  584. 2018-07-13 12:51:13,484 - Thread-71 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:13] "GET /api/v1.0/logs HTTP/1.0" 200 -
  585. 2018-07-13 12:51:15,503 - Thread-72 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:15] "GET /api/v1.0/logs HTTP/1.0" 200 -
  586. 2018-07-13 12:51:17,520 - Thread-73 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:17] "GET /api/v1.0/logs HTTP/1.0" 200 -
  587. 2018-07-13 12:51:19,544 - Thread-74 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:19] "GET /api/v1.0/logs HTTP/1.0" 200 -
  588. 2018-07-13 12:51:21,562 - Thread-75 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:21] "GET /api/v1.0/logs HTTP/1.0" 200 -
  589. 2018-07-13 12:51:23,579 - Thread-76 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:23] "GET /api/v1.0/logs HTTP/1.0" 200 -
  590. 2018-07-13 12:51:25,598 - Thread-77 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:25] "GET /api/v1.0/logs HTTP/1.0" 200 -
  591. 2018-07-13 12:51:27,615 - Thread-78 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:27] "GET /api/v1.0/logs HTTP/1.0" 200 -
  592. 2018-07-13 12:51:28,757 - Thread-26 - create_stack - INFO - Stack is now active with name - StorPerfAgentGroup
  593. 2018-07-13 12:51:29,272 - Thread-26 - create_stack - INFO - Found stack with name - StorPerfAgentGroup
  594. 2018-07-13 12:51:29,634 - Thread-79 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:29] "GET /api/v1.0/logs HTTP/1.0" 200 -
  595. 2018-07-13 12:51:31,652 - Thread-80 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:31] "GET /api/v1.0/logs HTTP/1.0" 200 -
  596. 2018-07-13 12:51:33,669 - Thread-81 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:33] "GET /api/v1.0/logs HTTP/1.0" 200 -
  597. 2018-07-13 12:51:34,638 - Thread-4 - create_instance - INFO - Found existing machine with name - storperf-agent
  598. 2018-07-13 12:51:34,721 - Thread-26 - storperf.storperf_master - DEBUG - Found VM at 10.5.149.166
  599. 2018-07-13 12:51:35,687 - Thread-82 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:35] "GET /api/v1.0/logs HTTP/1.0" 200 -
  600. 2018-07-13 12:51:36,477 - Thread-26 - storperf.storperf_master - DEBUG - Volume id 03756cfd-43ef-4ea8-91fc-8794c5a1b6a5, size=4, type=tripleo-ceph
  601. 2018-07-13 12:51:36,587 - Thread-26 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:36] "POST /api/v1.0/configurations HTTP/1.0" 200 -
  602. 2018-07-13 12:51:36,706 - Thread-83 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:36] "GET /api/v1.0/configurations HTTP/1.0" 200 -
  603. 2018-07-13 12:51:37,705 - Thread-84 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:37] "GET /api/v1.0/logs HTTP/1.0" 200 -
  604. 2018-07-13 12:51:46,727 - Thread-85 - __main__ - INFO - {u'queue_depths': u'4', u'block_sizes': u'4096k', u'workloads': {u'rs': {u'rw': u'read'}}, u'metadata': {u'build_tag': u'latest', u'test_case': u'opnfv_yardstick_tc074'}}
  605. 2018-07-13 12:51:46,728 - Thread-85 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  606. 2018-07-13 12:51:46,730 - Thread-85 - storperf.db.job_db - DEBUG - Job table exists
  607. 2018-07-13 12:51:46,730 - Thread-85 - storperf.db.job_db - DEBUG - Job params table exists
  608. 2018-07-13 12:51:46,731 - Thread-85 - storperf.db.job_db - DEBUG - Job summary table exists
  609. 2018-07-13 12:51:46,731 - Thread-85 - storperf.test_executor - ERROR - Cannot determine installer
  610. 2018-07-13 12:51:46,733 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  611. 2018-07-13 12:51:46,733 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  612. 2018-07-13 12:51:46,734 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  613. 2018-07-13 12:51:46,734 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  614. 2018-07-13 12:51:46,735 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  615. 2018-07-13 12:51:46,736 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  616. 2018-07-13 12:51:46,736 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  617. 2018-07-13 12:51:46,737 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  618. 2018-07-13 12:51:46,737 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  619. 2018-07-13 12:51:46,738 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  620. 2018-07-13 12:51:46,738 - Thread-85 - storperf.storperf_master - INFO - Custom workloads = None
  621. 2018-07-13 12:51:46,739 - Thread-85 - storperf.storperf_master - INFO - Custom workloads = {u'rs': {u'rw': u'read'}}
  622. 2018-07-13 12:51:46,739 - Thread-85 - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  623. 2018-07-13 12:51:46,740 - Thread-85 - storperf.db.job_db - DEBUG - Job table exists
  624. 2018-07-13 12:51:46,741 - Thread-85 - storperf.db.job_db - DEBUG - Job params table exists
  625. 2018-07-13 12:51:46,741 - Thread-85 - storperf.db.job_db - DEBUG - Job summary table exists
  626. 2018-07-13 12:51:46,742 - Thread-85 - storperf.test_executor - ERROR - Cannot determine installer
  627. 2018-07-13 12:51:46,743 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  628. 2018-07-13 12:51:46,743 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  629. 2018-07-13 12:51:46,744 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  630. 2018-07-13 12:51:46,744 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  631. 2018-07-13 12:51:46,745 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  632. 2018-07-13 12:51:46,746 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'ws' from '/storperf/storperf/workloads/ws.pyc'>
  633. 2018-07-13 12:51:46,746 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rs' from '/storperf/storperf/workloads/rs.pyc'>
  634. 2018-07-13 12:51:46,747 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'wr' from '/storperf/storperf/workloads/wr.pyc'>
  635. 2018-07-13 12:51:46,747 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rr' from '/storperf/storperf/workloads/rr.pyc'>
  636. 2018-07-13 12:51:46,748 - Thread-85 - storperf.test_executor - DEBUG - Found: <module 'rw' from '/storperf/storperf/workloads/rw.pyc'>
  637. 2018-07-13 12:51:46,748 - Thread-85 - storperf.test_executor - DEBUG - Set custom workloads to: {u'rs': {u'rw': u'read'}}
  638. 2018-07-13 12:51:46,749 - Thread-85 - storperf.test_executor - DEBUG - Set block_sizes to: 4096k
  639. 2018-07-13 12:51:46,749 - Thread-85 - storperf.test_executor - DEBUG - Set queue_depths to: 4
  640. 2018-07-13 12:51:46,750 - Thread-86 - storperf.storperf_master:10.5.149.166 - INFO - Initializing slave at 10.5.149.166
  641. 2018-07-13 12:51:46,751 - Thread-86 - storperf.storperf_master:10.5.149.166 - DEBUG - Checking if slave 10.5.149.166 is alive
  642. 2018-07-13 12:51:46,755 - Thread-86 - storperf.storperf_master:10.5.149.166 - DEBUG - Slave 10.5.149.166 is alive and ready
  643. 2018-07-13 12:51:46,778 - Thread-87 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  644. 2018-07-13 12:51:46,924 - Thread-87 - paramiko.transport - INFO - Authentication (publickey) successful!
  645. 2018-07-13 12:51:47,205 - Thread-86 - storperf.storperf_master:10.5.149.166 - DEBUG - Available space on / is 2778944
  646. 2018-07-13 12:51:47,206 - Thread-86 - storperf.storperf_master:10.5.149.166 - DEBUG - Transferring fio to 10.5.149.166
  647. 2018-07-13 12:51:47,437 - Thread-85 - storperf.test_executor - DEBUG - Set slaves to: [u'10.5.149.166']
  648. 2018-07-13 12:51:47,437 - Thread-85 - storperf.test_executor - DEBUG - Set volume count to: 1
  649. 2018-07-13 12:51:47,439 - Thread-85 - storperf.db.job_db - DEBUG - Reserved job id 98202a20-9004-4301-a8e4-0dca1e366b7c
  650. 2018-07-13 12:51:47,502 - Thread-85 - storperf.test_executor - INFO - Workload 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  651. 2018-07-13 12:51:47,503 - Thread-85 - storperf.test_executor - INFO - Testing FIO parameters for {'status': 'Pending', 'workload': <storperf.workloads._custom_workload._custom_workload object at 0x7fb90b7c95d0>, 'name': u'98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k', 'workload_name': u'rs', 'blocksize': u'4096k', 'queue-depth': u'4'}
  652. 2018-07-13 12:51:47,505 - Thread-85 - storperf.test_executor - DEBUG - Device to profile on 10.5.149.166: /dev/vdb
  653. 2018-07-13 12:51:47,505 - Thread-85 - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7fb9077234d0>
  654. 2018-07-13 12:51:47,506 - Thread-89 - storperf.test_executor - INFO - Starting 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k.10-5-149-166--dev-vdb
  655. 2018-07-13 12:51:47,508 - Thread-93 - paramiko.transport - DEBUG - starting thread (client mode): 0x76b6ed0L
  656. 2018-07-13 12:51:47,509 - Thread-93 - paramiko.transport - DEBUG - Local version/idstring: SSH-2.0-paramiko_2.0.2
  657. 2018-07-13 12:51:47,515 - Thread-93 - paramiko.transport - DEBUG - Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
  658. 2018-07-13 12:51:47,515 - Thread-93 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  659. 2018-07-13 12:51:47,517 - Thread-93 - paramiko.transport - DEBUG - kex algos:[u'curve25519-sha256@libssh.org', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group14-sha1'] server key:[u'ssh-rsa', u'rsa-sha2-512', u'rsa-sha2-256', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] server encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] client mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] server mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] client compress:[u'none', u'zlib@openssh.com'] server compress:[u'none', u'zlib@openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
  660. 2018-07-13 12:51:47,517 - Thread-93 - paramiko.transport - DEBUG - Kex agreed: diffie-hellman-group14-sha1
  661. 2018-07-13 12:51:47,518 - Thread-93 - paramiko.transport - DEBUG - Cipher agreed: aes128-ctr
  662. 2018-07-13 12:51:47,518 - Thread-93 - paramiko.transport - DEBUG - MAC agreed: hmac-sha2-256
  663. 2018-07-13 12:51:47,519 - Thread-93 - paramiko.transport - DEBUG - Compression agreed: none
  664. 2018-07-13 12:51:47,581 - Thread-93 - paramiko.transport - DEBUG - kex engine KexGroup14 specified hash_algo <built-in function openssl_sha1>
  665. 2018-07-13 12:51:47,583 - Thread-93 - paramiko.transport - DEBUG - Switch to new keys ...
  666. 2018-07-13 12:51:47,609 - Thread-89 - paramiko.transport - DEBUG - Adding ssh-rsa host key for 10.5.149.166: e45f802fc3a07a27d0cbe775b9bbb76f
  667. 2018-07-13 12:51:47,610 - Thread-89 - paramiko.transport - DEBUG - Trying key b9927bb6d18101c38ffdbcd779fabff8 from storperf/resources/ssh/storperf_rsa
  668. 2018-07-13 12:51:47,620 - Thread-93 - paramiko.transport - DEBUG - userauth is OK
  669. 2018-07-13 12:51:47,629 - Thread-93 - paramiko.transport - INFO - Authentication (publickey) successful!
  670. 2018-07-13 12:51:47,642 - Thread-89 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60 --parse-only
  671. 2018-07-13 12:51:47,643 - Thread-89 - paramiko.transport - DEBUG - [chan 0] Max packet in: 32768 bytes
  672. 2018-07-13 12:51:47,729 - Thread-93 - paramiko.transport - DEBUG - Received global request "hostkeys-00@openssh.com"
  673. 2018-07-13 12:51:47,730 - Thread-93 - paramiko.transport - DEBUG - Rejecting "hostkeys-00@openssh.com" global request from server.
  674. 2018-07-13 12:51:47,769 - Thread-93 - paramiko.transport - DEBUG - [chan 0] Max packet out: 32768 bytes
  675. 2018-07-13 12:51:47,769 - Thread-93 - paramiko.transport - DEBUG - Secsh channel 0 opened.
  676. 2018-07-13 12:51:47,777 - Thread-93 - paramiko.transport - DEBUG - [chan 0] Sesch channel 0 request ok
  677. 2018-07-13 12:51:47,780 - 10.5.149.166 stdout - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Started
  678. 2018-07-13 12:51:47,781 - Thread-89 - storperf.fio.fio_invoker:10.5.149.166 - INFO - Started fio on 10.5.149.166
  679. 2018-07-13 12:51:47,780 - 10.5.149.166 stderr - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Started
  680. 2018-07-13 12:51:48,249 - Thread-89 - storperf.fio.fio_invoker:10.5.149.166 - INFO - Finished fio on 10.5.149.166 with exit code 0
  681. 2018-07-13 12:51:48,250 - 10.5.149.166 stdout - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Finished
  682. 2018-07-13 12:51:48,249 - Thread-93 - paramiko.transport - DEBUG - [chan 0] EOF received (0)
  683. 2018-07-13 12:51:48,250 - Thread-89 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Joining stderr handler
  684. 2018-07-13 12:51:48,251 - 10.5.149.166 stderr - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Finished
  685. 2018-07-13 12:51:48,252 - Thread-93 - paramiko.transport - DEBUG - [chan 0] EOF sent (0)
  686. 2018-07-13 12:51:48,253 - Thread-89 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Joining stdout handler
  687. 2018-07-13 12:51:48,254 - Thread-89 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Ended
  688. 2018-07-13 12:51:48,355 - Thread-93 - paramiko.transport - DEBUG - EOF in transport thread
  689. 2018-07-13 12:51:48,368 - Thread-89 - storperf.test_executor - INFO - Ended 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k.10-5-149-166--dev-vdb
  690. 2018-07-13 12:51:48,369 - Thread-85 - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7fb9077234d0>, exit status None
  691. 2018-07-13 12:51:48,455 - Thread-85 - storperf.test_executor - ERROR - Cannot determine installer
  692. 2018-07-13 12:51:48,457 - Workload thread - storperf.test_executor - INFO - Starting job 98202a20-9004-4301-a8e4-0dca1e366b7c
  693. 2018-07-13 12:51:48,458 - Thread-85 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:48] "POST /api/v2.0/jobs HTTP/1.0" 200 -
  694. 2018-07-13 12:51:48,459 - Workload thread - storperf.db.job_db - DEBUG - Connecting to StorPerfJob.db
  695. 2018-07-13 12:51:48,462 - Workload thread - storperf.db.job_db - DEBUG - Job table exists
  696. 2018-07-13 12:51:48,463 - Workload thread - storperf.db.job_db - DEBUG - Job params table exists
  697. 2018-07-13 12:51:48,464 - Workload thread - storperf.db.job_db - DEBUG - Job summary table exists
  698. 2018-07-13 12:51:48,466 - Workload thread - storperf.test_executor - INFO - Workload 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k={'rw': u'read', 'name': u'rs', 'output-format': 'json', 'numjobs': '1', 'direct': '1', 'bs': u'4096k', 'loops': '200', 'ioengine': 'libaio', 'iodepth': u'4', 'status-interval': '60'}
  699. 2018-07-13 12:51:48,467 - Workload thread - storperf.test_executor - INFO - Starting run 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k
  700. 2018-07-13 12:51:48,480 - Thread-95 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:51:48] "GET /api/v2.0/jobs?type=status&id=98202a20-9004-4301-a8e4-0dca1e366b7c HTTP/1.0" 404 -
  701. 2018-07-13 12:51:48,480 - Workload thread - storperf.test_executor - DEBUG - Device to profile on 10.5.149.166: /dev/vdb
  702. 2018-07-13 12:51:48,481 - Workload thread - storperf.test_executor - DEBUG - Waiting on <multiprocessing.pool.ApplyResult object at 0x7fb9076adb10>
  703. 2018-07-13 12:51:48,482 - Thread-96 - storperf.test_executor - INFO - Starting 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k.10-5-149-166--dev-vdb
  704. 2018-07-13 12:51:48,596 - Thread-100 - paramiko.transport - DEBUG - starting thread (client mode): 0x76665d0L
  705. 2018-07-13 12:51:48,597 - Thread-100 - paramiko.transport - DEBUG - Local version/idstring: SSH-2.0-paramiko_2.0.2
  706. 2018-07-13 12:51:48,603 - Thread-100 - paramiko.transport - DEBUG - Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
  707. 2018-07-13 12:51:48,603 - Thread-100 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  708. 2018-07-13 12:51:48,604 - Thread-100 - paramiko.transport - DEBUG - kex algos:[u'curve25519-sha256@libssh.org', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group14-sha1'] server key:[u'ssh-rsa', u'rsa-sha2-512', u'rsa-sha2-256', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] server encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] client mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] server mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] client compress:[u'none', u'zlib@openssh.com'] server compress:[u'none', u'zlib@openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
  709. 2018-07-13 12:51:48,605 - Thread-100 - paramiko.transport - DEBUG - Kex agreed: diffie-hellman-group14-sha1
  710. 2018-07-13 12:51:48,605 - Thread-100 - paramiko.transport - DEBUG - Cipher agreed: aes128-ctr
  711. 2018-07-13 12:51:48,606 - Thread-100 - paramiko.transport - DEBUG - MAC agreed: hmac-sha2-256
  712. 2018-07-13 12:51:48,606 - Thread-100 - paramiko.transport - DEBUG - Compression agreed: none
  713. 2018-07-13 12:51:48,670 - Thread-100 - paramiko.transport - DEBUG - kex engine KexGroup14 specified hash_algo <built-in function openssl_sha1>
  714. 2018-07-13 12:51:48,671 - Thread-100 - paramiko.transport - DEBUG - Switch to new keys ...
  715. 2018-07-13 12:51:48,697 - Thread-96 - paramiko.transport - DEBUG - Adding ssh-rsa host key for 10.5.149.166: e45f802fc3a07a27d0cbe775b9bbb76f
  716. 2018-07-13 12:51:48,699 - Thread-96 - paramiko.transport - DEBUG - Trying key b9927bb6d18101c38ffdbcd779fabff8 from storperf/resources/ssh/storperf_rsa
  717. 2018-07-13 12:51:48,708 - Thread-100 - paramiko.transport - DEBUG - userauth is OK
  718. 2018-07-13 12:51:48,718 - Thread-100 - paramiko.transport - INFO - Authentication (publickey) successful!
  719. 2018-07-13 12:51:48,731 - Thread-96 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Remote command: sudo ./fio --rw=read --name=rs --output-format=json --numjobs=1 --direct=1 --filename=/dev/vdb --bs=4096k --loops=200 --ioengine=libaio --size=100% --iodepth=4 --status-interval=60
  720. 2018-07-13 12:51:48,732 - Thread-96 - paramiko.transport - DEBUG - [chan 0] Max packet in: 32768 bytes
  721. 2018-07-13 12:51:48,812 - Thread-100 - paramiko.transport - DEBUG - Received global request "hostkeys-00@openssh.com"
  722. 2018-07-13 12:51:48,812 - Thread-100 - paramiko.transport - DEBUG - Rejecting "hostkeys-00@openssh.com" global request from server.
  723. 2018-07-13 12:51:48,849 - Thread-100 - paramiko.transport - DEBUG - [chan 0] Max packet out: 32768 bytes
  724. 2018-07-13 12:51:48,849 - Thread-100 - paramiko.transport - DEBUG - Secsh channel 0 opened.
  725. 2018-07-13 12:51:48,866 - Thread-100 - paramiko.transport - DEBUG - [chan 0] Sesch channel 0 request ok
  726. 2018-07-13 12:51:48,867 - 10.5.149.166 stdout - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Started
  727. 2018-07-13 12:51:48,868 - 10.5.149.166 stderr - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Started
  728. 2018-07-13 12:51:48,868 - Thread-96 - storperf.fio.fio_invoker:10.5.149.166 - INFO - Started fio on 10.5.149.166
  729. 2018-07-13 12:51:49,495 - Thread-102 - storperf.test_executor - INFO - Terminating current run
  730. 2018-07-13 12:51:49,496 - Thread-102 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Terminating fio on 10.5.149.166
  731. 2018-07-13 12:51:49,499 - Thread-103 - paramiko.transport - DEBUG - starting thread (client mode): 0x76db0d0L
  732. 2018-07-13 12:51:49,500 - Thread-103 - paramiko.transport - DEBUG - Local version/idstring: SSH-2.0-paramiko_2.0.2
  733. 2018-07-13 12:51:49,506 - Thread-103 - paramiko.transport - DEBUG - Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
  734. 2018-07-13 12:51:49,507 - Thread-103 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  735. 2018-07-13 12:51:49,509 - Thread-103 - paramiko.transport - DEBUG - kex algos:[u'curve25519-sha256@libssh.org', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group14-sha1'] server key:[u'ssh-rsa', u'rsa-sha2-512', u'rsa-sha2-256', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] server encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] client mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] server mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] client compress:[u'none', u'zlib@openssh.com'] server compress:[u'none', u'zlib@openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
  736. 2018-07-13 12:51:49,510 - Thread-103 - paramiko.transport - DEBUG - Kex agreed: diffie-hellman-group14-sha1
  737. 2018-07-13 12:51:49,511 - Thread-103 - paramiko.transport - DEBUG - Cipher agreed: aes128-ctr
  738. 2018-07-13 12:51:49,512 - Thread-103 - paramiko.transport - DEBUG - MAC agreed: hmac-sha2-256
  739. 2018-07-13 12:51:49,513 - Thread-103 - paramiko.transport - DEBUG - Compression agreed: none
  740. 2018-07-13 12:51:49,574 - Thread-103 - paramiko.transport - DEBUG - kex engine KexGroup14 specified hash_algo <built-in function openssl_sha1>
  741. 2018-07-13 12:51:49,575 - Thread-103 - paramiko.transport - DEBUG - Switch to new keys ...
  742. 2018-07-13 12:51:49,578 - Thread-102 - paramiko.transport - DEBUG - Adding ssh-rsa host key for 10.5.149.166: e45f802fc3a07a27d0cbe775b9bbb76f
  743. 2018-07-13 12:51:49,581 - Thread-102 - paramiko.transport - DEBUG - Trying key b9927bb6d18101c38ffdbcd779fabff8 from storperf/resources/ssh/storperf_rsa
  744. 2018-07-13 12:51:49,612 - Thread-103 - paramiko.transport - DEBUG - userauth is OK
  745. 2018-07-13 12:51:49,622 - Thread-103 - paramiko.transport - INFO - Authentication (publickey) successful!
  746. 2018-07-13 12:51:49,646 - Thread-102 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Executing on 10.5.149.166: sudo killall fio
  747. 2018-07-13 12:51:49,647 - Thread-102 - paramiko.transport - DEBUG - [chan 0] Max packet in: 32768 bytes
  748. 2018-07-13 12:51:49,733 - Thread-103 - paramiko.transport - DEBUG - Received global request "hostkeys-00@openssh.com"
  749. 2018-07-13 12:51:49,733 - Thread-103 - paramiko.transport - DEBUG - Rejecting "hostkeys-00@openssh.com" global request from server.
  750. 2018-07-13 12:51:49,773 - Thread-103 - paramiko.transport - DEBUG - [chan 0] Max packet out: 32768 bytes
  751. 2018-07-13 12:51:49,773 - Thread-103 - paramiko.transport - DEBUG - Secsh channel 0 opened.
  752. 2018-07-13 12:51:49,781 - Thread-103 - paramiko.transport - DEBUG - [chan 0] Sesch channel 0 request ok
  753. 2018-07-13 12:51:50,096 - Thread-103 - paramiko.transport - DEBUG - [chan 0] EOF received (0)
  754. 2018-07-13 12:51:50,098 - Thread-102 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Terminating fio on 10.5.149.166
  755. 2018-07-13 12:51:50,100 - Thread-105 - paramiko.transport - DEBUG - starting thread (client mode): 0x76db910L
  756. 2018-07-13 12:51:50,101 - Thread-105 - paramiko.transport - DEBUG - Local version/idstring: SSH-2.0-paramiko_2.0.2
  757. 2018-07-13 12:51:50,107 - Thread-105 - paramiko.transport - DEBUG - Remote version/idstring: SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
  758. 2018-07-13 12:51:50,108 - Thread-105 - paramiko.transport - INFO - Connected (version 2.0, client OpenSSH_7.2p2)
  759. 2018-07-13 12:51:50,109 - Thread-105 - paramiko.transport - DEBUG - kex algos:[u'curve25519-sha256@libssh.org', u'ecdh-sha2-nistp256', u'ecdh-sha2-nistp384', u'ecdh-sha2-nistp521', u'diffie-hellman-group-exchange-sha256', u'diffie-hellman-group14-sha1'] server key:[u'ssh-rsa', u'rsa-sha2-512', u'rsa-sha2-256', u'ecdsa-sha2-nistp256', u'ssh-ed25519'] client encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] server encrypt:[u'chacha20-poly1305@openssh.com', u'aes128-ctr', u'aes192-ctr', u'aes256-ctr', u'aes128-gcm@openssh.com', u'aes256-gcm@openssh.com'] client mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] server mac:[u'umac-64-etm@openssh.com', u'umac-128-etm@openssh.com', u'hmac-sha2-256-etm@openssh.com', u'hmac-sha2-512-etm@openssh.com', u'hmac-sha1-etm@openssh.com', u'umac-64@openssh.com', u'umac-128@openssh.com', u'hmac-sha2-256', u'hmac-sha2-512', u'hmac-sha1'] client compress:[u'none', u'zlib@openssh.com'] server compress:[u'none', u'zlib@openssh.com'] client lang:[u''] server lang:[u''] kex follows?False
  760. 2018-07-13 12:51:50,109 - Thread-105 - paramiko.transport - DEBUG - Kex agreed: diffie-hellman-group14-sha1
  761. 2018-07-13 12:51:50,110 - Thread-105 - paramiko.transport - DEBUG - Cipher agreed: aes128-ctr
  762. 2018-07-13 12:51:50,110 - Thread-105 - paramiko.transport - DEBUG - MAC agreed: hmac-sha2-256
  763. 2018-07-13 12:51:50,111 - Thread-105 - paramiko.transport - DEBUG - Compression agreed: none
  764. 2018-07-13 12:51:50,176 - Thread-96 - storperf.fio.fio_invoker:10.5.149.166 - INFO - Finished fio on 10.5.149.166 with exit code 128
  765. 2018-07-13 12:51:50,177 - Thread-96 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Joining stderr handler
  766. 2018-07-13 12:51:50,177 - 10.5.149.166 stderr - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Finished
  767. 2018-07-13 12:51:50,178 - 10.5.149.166 stdout - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Finished
  768. 2018-07-13 12:51:50,177 - Thread-100 - paramiko.transport - DEBUG - [chan 0] EOF received (0)
  769. 2018-07-13 12:51:50,180 - Thread-96 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Joining stdout handler
  770. 2018-07-13 12:51:50,182 - Thread-96 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Ended
  771. 2018-07-13 12:51:50,182 - Thread-100 - paramiko.transport - DEBUG - [chan 0] EOF sent (0)
  772. 2018-07-13 12:51:50,183 - Thread-100 - paramiko.transport - DEBUG - Dropping user packet because connection is dead.
  773. 2018-07-13 12:51:50,183 - Thread-100 - paramiko.transport - DEBUG - Dropping user packet because connection is dead.
  774. 2018-07-13 12:51:50,184 - Thread-105 - paramiko.transport - DEBUG - kex engine KexGroup14 specified hash_algo <built-in function openssl_sha1>
  775. 2018-07-13 12:51:50,185 - Thread-105 - paramiko.transport - DEBUG - Switch to new keys ...
  776. 2018-07-13 12:51:50,201 - Thread-102 - paramiko.transport - DEBUG - Adding ssh-rsa host key for 10.5.149.166: e45f802fc3a07a27d0cbe775b9bbb76f
  777. 2018-07-13 12:51:50,202 - Thread-102 - paramiko.transport - DEBUG - Trying key b9927bb6d18101c38ffdbcd779fabff8 from storperf/resources/ssh/storperf_rsa
  778. 2018-07-13 12:51:50,224 - Thread-105 - paramiko.transport - DEBUG - userauth is OK
  779. 2018-07-13 12:51:50,233 - Thread-105 - paramiko.transport - INFO - Authentication (publickey) successful!
  780. 2018-07-13 12:51:50,235 - Thread-102 - storperf.fio.fio_invoker:10.5.149.166 - DEBUG - Executing on 10.5.149.166: sudo killall fio
  781. 2018-07-13 12:51:50,235 - Thread-102 - paramiko.transport - DEBUG - [chan 0] Max packet in: 32768 bytes
  782. 2018-07-13 12:51:50,328 - Thread-105 - paramiko.transport - DEBUG - Received global request "hostkeys-00@openssh.com"
  783. 2018-07-13 12:51:50,328 - Thread-105 - paramiko.transport - DEBUG - Rejecting "hostkeys-00@openssh.com" global request from server.
  784. 2018-07-13 12:51:50,350 - Thread-96 - storperf.test_executor - INFO - Ended 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k.10-5-149-166--dev-vdb
  785. 2018-07-13 12:51:50,351 - Workload thread - storperf.test_executor - DEBUG - Done waiting for <multiprocessing.pool.ApplyResult object at 0x7fb9076adb10>, exit status []
  786. 2018-07-13 12:51:50,351 - Workload thread - storperf.test_executor - INFO - Completed run 98202a20-9004-4301-a8e4-0dca1e366b7c.rs.queue-depth.4.block-size.4096k
  787. 2018-07-13 12:51:50,352 - Workload thread - storperf.test_executor - INFO - Completed job 98202a20-9004-4301-a8e4-0dca1e366b7c
  788. 2018-07-13 12:51:50,352 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method StorPerfMaster.executor_event of <storperf.storperf_master.StorPerfMaster object at 0x7fb908584810>>
  789. 2018-07-13 12:51:50,353 - Workload thread - storperf.test_executor - DEBUG - Notifying event listener <bound method DataHandler.data_event of <storperf.utilities.data_handler.DataHandler object at 0x7fb907666210>>
  790. 2018-07-13 12:51:50,353 - Workload thread - storperf.utilities.data_handler - DEBUG - Event received
  791. 2018-07-13 12:51:50,354 - Workload thread - storperf.test_executor - ERROR - While notifying listener 'steady_state'
  792. Traceback (most recent call last):
  793. File "/storperf/storperf/test_executor.py", line 170, in broadcast_event
  794. event_listener(self)
  795. File "/storperf/storperf/utilities/data_handler.py", line 35, in data_event
  796. self._push_to_db(executor)
  797. File "/storperf/storperf/utilities/data_handler.py", line 140, in _push_to_db
  798. criteria = self._determine_criteria(executor.metadata)
  799. File "/storperf/storperf/utilities/data_handler.py", line 163, in _determine_criteria
  800. for workload, value in metadata['details']['steady_state'].items():
  801. KeyError: 'steady_state'
  802. 2018-07-13 12:51:50,369 - Thread-105 - paramiko.transport - DEBUG - [chan 0] Max packet out: 32768 bytes
  803. 2018-07-13 12:51:50,372 - Thread-105 - paramiko.transport - DEBUG - Secsh channel 0 opened.
  804. 2018-07-13 12:51:50,403 - Thread-105 - paramiko.transport - DEBUG - [chan 0] Sesch channel 0 request ok
  805. 2018-07-13 12:51:50,414 - Thread-105 - paramiko.transport - DEBUG - [chan 0] EOF received (0)
  806. 2018-07-13 12:51:50,414 - Thread-102 - storperf.fio.fio_invoker:10.5.149.166 - ERROR - fio: no process found
  807. 2018-07-13 12:51:50,415 - Thread-105 - paramiko.transport - DEBUG - [chan 0] EOF sent (0)
  808. 2018-07-13 12:51:50,416 - Thread-105 - paramiko.transport - DEBUG - Dropping user packet because connection is dead.
  809. 2018-07-13 12:51:50,417 - Thread-105 - paramiko.transport - DEBUG - Dropping user packet because connection is dead.
  810. 2018-07-13 12:51:50,417 - Thread-102 - create_stack - INFO - Deleting stack - StorPerfAgentGroup
  811. 2018-07-13 12:52:15,695 - Thread-102 - create_stack - INFO - Attempting to delete again stack - StorPerfAgentGroup
  812. 2018-07-13 12:52:15,773 - Thread-102 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:52:15] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  813. 2018-07-13 12:52:18,454 - Thread-107 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:52:18] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
  814. 2018-07-13 12:52:18,570 - Thread-108 - werkzeug - INFO - 172.18.0.6 - - [13/Jul/2018 12:52:18] "DELETE /api/v1.0/configurations HTTP/1.0" 200 -
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!