Heketi v4.0.0-8-g9372c22-release-4 [heketi] INFO 2017/07/19 09:09:10 Loaded kubernetes executor [heketi] INFO 2017/07/19 09:09:10 Loaded simple allocator [heketi] INFO 2017/07/19 09:09:10 GlusterFS Application Loaded Listening on port 8080 [negroni] Started GET /clusters [negroni] Completed 200 OK in 187.909µs [negroni] Started POST /clusters [heketi] INFO 2017/07/19 09:09:16 Backup successful [negroni] Completed 201 Created in 178.349519ms [negroni] Started POST /nodes [heketi] INFO 2017/07/19 09:09:17 Adding node black1 [asynchttp] INFO 2017/07/19 09:09:17 Started job 59c81e2e22adf873476efc7c984af659 [heketi] INFO 2017/07/19 09:09:17 Added node 63d405e57106556d7303eeb7f452308d [asynchttp] INFO 2017/07/19 09:09:17 Completed job 59c81e2e22adf873476efc7c984af659 in 49.62021ms [heketi] INFO 2017/07/19 09:09:17 Backup successful [negroni] Completed 202 Accepted in 359.185501ms [negroni] Started GET /queue/59c81e2e22adf873476efc7c984af659 [heketi] INFO 2017/07/19 09:09:17 Backup successful [negroni] Completed 303 See Other in 140.972176ms [negroni] Started GET /nodes/63d405e57106556d7303eeb7f452308d [negroni] Completed 200 OK in 546.566µs [negroni] Started POST /devices [heketi] INFO 2017/07/19 09:09:17 Adding device /dev/sdd to node 63d405e57106556d7303eeb7f452308d [asynchttp] INFO 2017/07/19 09:09:17 Started job e68edd369dba800d4395c3aea29314f9 [heketi] INFO 2017/07/19 09:09:17 Backup successful [negroni] Completed 202 Accepted in 186.137314ms [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 81.741µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.965µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.51µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 55.48µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.035µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 76.76µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 80.96µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 76.89µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 90.912µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.115µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.24µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.515µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 76.16µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 57.433µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 46.398µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.204µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.931µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.796µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.72µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.795µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 59.973µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 81.916µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 61.415µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 82.775µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.666µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.346µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.432µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 112.635µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 86.867µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 107.596µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.741µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.176µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.135µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.453µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 84.24µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.191µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 63.236µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.468µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.305µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.905µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.7µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.375µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 139.691µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.41µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 53.481µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.223µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.52µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.721µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 48.158µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 76.045µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 108.716µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.315µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.87µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 60.168µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.755µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.466µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 59.468µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.386µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.063µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 62.048µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 77.13µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.729µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.635µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.863µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 74.24µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 231.076µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.79µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.07µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.82µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 73.14µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 73.18µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 77.995µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 75.52µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 75.245µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 91.131µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.226µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.139µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.445µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 80.269µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 51.316µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.621µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.929µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.42µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.39µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.253µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.711µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 75.106µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.305µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 74.369µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 45.81µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 79.76µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 130.986µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.846µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.195µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.968µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 98.641µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.352µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.307µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.835µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 78.203µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 73.406µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.525µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 82.738µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.71µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 37.453µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.875µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 75.74µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.28µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.63µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.169µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 75.02µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 93.732µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.066µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.775µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.521µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.147µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.245µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 78.461µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 78.62µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.115µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.64µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 78.34µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.161µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.885µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.525µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.103µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 78.67µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.105µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.626µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.973µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 47.32µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.745µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.16µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.415µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 77.54µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.84µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 74.065µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.205µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 96.499µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 77.635µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.45µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 79.94µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.857µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 86.047µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.276µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.916µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.51µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 88.365µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.245µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.626µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.31µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.933µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 61.903µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.053µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.115µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 82.396µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.893µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 67.369µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.182µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.875µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.693µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 63.973µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.245µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.065µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 62.437µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 76.55µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.425µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 75.185µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.631µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 79.435µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.175µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.66µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.75µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.515µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 61.832µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.31µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 55.833µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.515µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.445µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 114.805µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 41.745µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 79.54µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.548µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 64.195µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.906µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 347.471µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 75.331µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.23µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.26µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.9µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 74.46µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 116.265µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.954µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.55µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 73.875µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.827µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.846µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.77µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.386µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.76µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 83.156µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 66.868µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.035µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 69.82µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 62.116µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 68.14µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.293µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 72.83µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 65.618µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 71.56µs [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 70.331µs [kubeexec] DEBUG 2017/07/19 09:12:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: pvcreate --metadatasize=128M --dataalignment=256K /dev/sdd Result: Physical volume "/dev/sdd" successfully created. [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [negroni] Completed 200 OK in 59.804µs [kubeexec] DEBUG 2017/07/19 09:12:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: vgcreate vg_284e8a4d9dd8a261e0b7d6616a43ad6b /dev/sdd Result: Volume group "vg_284e8a4d9dd8a261e0b7d6616a43ad6b" successfully created [kubeexec] DEBUG 2017/07/19 09:12:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: vgdisplay -c vg_284e8a4d9dd8a261e0b7d6616a43ad6b Result: vg_284e8a4d9dd8a261e0b7d6616a43ad6b:r/w:772:-1:0:0:0:-1:0:1:1:1953382400:4096:476900:0:476900:gDoBGY-9Hw3-A3IY-Mc8n-qgkw-ZzAw-OAP2vL [sshexec] DEBUG 2017/07/19 09:12:49 /src/github.com/heketi/heketi/executors/sshexec/device.go:119: Size of /dev/sdd in black1 is 1953382400 [heketi] INFO 2017/07/19 09:12:49 Added device /dev/sdd [asynchttp] INFO 2017/07/19 09:12:49 Completed job e68edd369dba800d4395c3aea29314f9 in 3m31.559092934s [negroni] Started GET /queue/e68edd369dba800d4395c3aea29314f9 [heketi] INFO 2017/07/19 09:12:49 Backup successful [negroni] Completed 204 No Content in 140.165724ms [negroni] Started POST /nodes [heketi] INFO 2017/07/19 09:12:50 Adding node black2 [asynchttp] INFO 2017/07/19 09:12:50 Started job 8ccc57530b6dfae33bb19dc18142524d [sshexec] INFO 2017/07/19 09:12:50 Probing: black1 -> 172.100.100.2 [heketi] INFO 2017/07/19 09:12:50 Backup successful [negroni] Completed 202 Accepted in 189.383268ms [negroni] Started GET /queue/8ccc57530b6dfae33bb19dc18142524d [negroni] Completed 200 OK in 30.809µs [kubeexec] DEBUG 2017/07/19 09:12:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: gluster peer probe 172.100.100.2 Result: peer probe: success. Host 172.100.100.2 port 24007 already in peer list [sshexec] INFO 2017/07/19 09:12:50 Setting snapshot limit [negroni] Started GET /queue/8ccc57530b6dfae33bb19dc18142524d [negroni] Completed 200 OK in 73.728µs [negroni] Started GET /queue/8ccc57530b6dfae33bb19dc18142524d [negroni] Completed 200 OK in 78.58µs [kubeexec] DEBUG 2017/07/19 09:12:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: gluster --mode=script snapshot config snap-max-hard-limit 14 Result: snapshot config: snap-max-hard-limit for System set successfully [heketi] INFO 2017/07/19 09:12:50 Added node fdcad2b8b99b12ee3f0b5b029bbf2524 [asynchttp] INFO 2017/07/19 09:12:50 Completed job 8ccc57530b6dfae33bb19dc18142524d in 783.205277ms [negroni] Started GET /queue/8ccc57530b6dfae33bb19dc18142524d [heketi] INFO 2017/07/19 09:12:51 Backup successful [negroni] Completed 303 See Other in 213.34888ms [negroni] Started GET /nodes/fdcad2b8b99b12ee3f0b5b029bbf2524 [negroni] Completed 200 OK in 389.656µs [negroni] Started POST /devices [heketi] INFO 2017/07/19 09:12:51 Adding device /dev/sdd to node fdcad2b8b99b12ee3f0b5b029bbf2524 [asynchttp] INFO 2017/07/19 09:12:51 Started job ac188f5bc0acf71d3d4ff49d185dfec7 [heketi] INFO 2017/07/19 09:12:51 Backup successful [negroni] Completed 202 Accepted in 275.393988ms [negroni] Started GET /queue/ac188f5bc0acf71d3d4ff49d185dfec7 [negroni] Completed 200 OK in 37.824µs [kubeexec] DEBUG 2017/07/19 09:12:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black2 Pod: glusterfs-s9f4g Command: pvcreate --metadatasize=128M --dataalignment=256K /dev/sdd Result: Physical volume "/dev/sdd" successfully created. [kubeexec] DEBUG 2017/07/19 09:12:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black2 Pod: glusterfs-s9f4g Command: vgcreate vg_79068a61269ee2a564a5a6e0cc1037d6 /dev/sdd Result: Volume group "vg_79068a61269ee2a564a5a6e0cc1037d6" successfully created [kubeexec] DEBUG 2017/07/19 09:12:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black2 Pod: glusterfs-s9f4g Command: vgdisplay -c vg_79068a61269ee2a564a5a6e0cc1037d6 Result: vg_79068a61269ee2a564a5a6e0cc1037d6:r/w:772:-1:0:0:0:-1:0:1:1:1953382400:4096:476900:0:476900:mHAj30-pO4x-Es1V-V1nI-GK9c-u30H-2WzXC0 [sshexec] DEBUG 2017/07/19 09:12:52 /src/github.com/heketi/heketi/executors/sshexec/device.go:119: Size of /dev/sdd in black2 is 1953382400 [negroni] Started GET /queue/ac188f5bc0acf71d3d4ff49d185dfec7 [negroni] Completed 200 OK in 67.13µs [heketi] INFO 2017/07/19 09:12:52 Added device /dev/sdd [asynchttp] INFO 2017/07/19 09:12:52 Completed job ac188f5bc0acf71d3d4ff49d185dfec7 in 1.274874194s [negroni] Started GET /queue/ac188f5bc0acf71d3d4ff49d185dfec7 [heketi] INFO 2017/07/19 09:12:53 Backup successful [negroni] Completed 204 No Content in 212.389191ms [negroni] Started POST /nodes [heketi] INFO 2017/07/19 09:12:53 Adding node black3 [asynchttp] INFO 2017/07/19 09:12:53 Started job 60323a384a736447dc22eea78cb01354 [sshexec] INFO 2017/07/19 09:12:53 Probing: black1 -> 172.100.100.3 [heketi] INFO 2017/07/19 09:12:53 Backup successful [negroni] Completed 202 Accepted in 182.998702ms [negroni] Started GET /queue/60323a384a736447dc22eea78cb01354 [negroni] Completed 200 OK in 45.564µs [kubeexec] DEBUG 2017/07/19 09:12:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: gluster peer probe 172.100.100.3 Result: peer probe: success. Host 172.100.100.3 port 24007 already in peer list [sshexec] INFO 2017/07/19 09:12:53 Setting snapshot limit [negroni] Started GET /queue/60323a384a736447dc22eea78cb01354 [negroni] Completed 200 OK in 37.073µs [negroni] Started GET /queue/60323a384a736447dc22eea78cb01354 [negroni] Completed 200 OK in 74.84µs [kubeexec] DEBUG 2017/07/19 09:12:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: gluster --mode=script snapshot config snap-max-hard-limit 14 Result: snapshot config: snap-max-hard-limit for System set successfully [heketi] INFO 2017/07/19 09:12:54 Added node 816c25b6b17bca355901320f783bc773 [asynchttp] INFO 2017/07/19 09:12:54 Completed job 60323a384a736447dc22eea78cb01354 in 733.066663ms [negroni] Started GET /queue/60323a384a736447dc22eea78cb01354 [heketi] INFO 2017/07/19 09:12:54 Backup successful [negroni] Completed 303 See Other in 79.255745ms [negroni] Started GET /nodes/816c25b6b17bca355901320f783bc773 [negroni] Completed 200 OK in 287.555µs [negroni] Started POST /devices [heketi] INFO 2017/07/19 09:12:54 Adding device /dev/sdd to node 816c25b6b17bca355901320f783bc773 [asynchttp] INFO 2017/07/19 09:12:54 Started job 1e08b5c269ee757541a0f2163d8c0e76 [heketi] INFO 2017/07/19 09:12:54 Backup successful [negroni] Completed 202 Accepted in 159.704639ms [negroni] Started GET /queue/1e08b5c269ee757541a0f2163d8c0e76 [negroni] Completed 200 OK in 64.595µs [kubeexec] DEBUG 2017/07/19 09:12:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black3 Pod: glusterfs-4c2sn Command: pvcreate --metadatasize=128M --dataalignment=256K /dev/sdd Result: Physical volume "/dev/sdd" successfully created. [kubeexec] DEBUG 2017/07/19 09:12:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black3 Pod: glusterfs-4c2sn Command: vgcreate vg_d386fc09ee51092f5e7ecd647dc1ec45 /dev/sdd Result: Volume group "vg_d386fc09ee51092f5e7ecd647dc1ec45" successfully created [kubeexec] DEBUG 2017/07/19 09:12:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black3 Pod: glusterfs-4c2sn Command: vgdisplay -c vg_d386fc09ee51092f5e7ecd647dc1ec45 Result: vg_d386fc09ee51092f5e7ecd647dc1ec45:r/w:772:-1:0:0:0:-1:0:1:1:1953382400:4096:476900:0:476900:65OntN-0nX7-ri5X-uj3a-QEI7-oeyl-r3tmvp [sshexec] DEBUG 2017/07/19 09:12:55 /src/github.com/heketi/heketi/executors/sshexec/device.go:119: Size of /dev/sdd in black3 is 1953382400 [heketi] INFO 2017/07/19 09:12:55 Added device /dev/sdd [asynchttp] INFO 2017/07/19 09:12:55 Completed job 1e08b5c269ee757541a0f2163d8c0e76 in 1.083240116s [negroni] Started GET /queue/1e08b5c269ee757541a0f2163d8c0e76 [heketi] INFO 2017/07/19 09:12:56 Backup successful [negroni] Completed 204 No Content in 434.686015ms [negroni] Started GET /clusters/f0b4adf0e561c64f68db72eaff411207 [negroni] Completed 200 OK in 302.016µs [negroni] Started GET /clusters [negroni] Completed 200 OK in 144.373µs [negroni] Started GET /clusters/f0b4adf0e561c64f68db72eaff411207 [negroni] Completed 200 OK in 339.636µs [negroni] Started POST /volumes [asynchttp] INFO 2017/07/19 09:12:56 Started job 9a482306dc68528394849604c621bea0 [heketi] INFO 2017/07/19 09:12:56 Creating volume a87c7055e269655c2fee0790516fdcf2 [heketi] INFO 2017/07/19 09:12:56 brick_num: 0 [heketi] INFO 2017/07/19 09:12:56 Backup successful [negroni] Completed 202 Accepted in 161.050194ms [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 64.544µs [heketi] INFO 2017/07/19 09:12:56 Creating brick 0b967c98a211caf6a2f2ebc2af26b3cc [heketi] INFO 2017/07/19 09:12:56 Creating brick 444b6811d86b1f2c4ab9eb196a495239 [heketi] INFO 2017/07/19 09:12:56 Creating brick 7b3f12de7e65ee2adc284be36db732e4 [kubeexec] DEBUG 2017/07/19 09:12:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: mkdir -p /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc Result: [kubeexec] DEBUG 2017/07/19 09:12:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black3 Pod: glusterfs-4c2sn Command: mkdir -p /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239 Result: [kubeexec] DEBUG 2017/07/19 09:12:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black2 Pod: glusterfs-s9f4g Command: mkdir -p /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4 Result: [kubeexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvcreate --poolmetadatasize 12288K -c 256K -L 2097152K -T vg_284e8a4d9dd8a261e0b7d6616a43ad6b/tp_0b967c98a211caf6a2f2ebc2af26b3cc -V 2097152K -n brick_0b967c98a211caf6a2f2ebc2af26b3cc] on glusterfs-hhtwh: Err[command terminated with exit code 5]: Stdout [ Using default stripesize 64.00 KiB. ]: Stderr [ /dev/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/lvol0: not found: device not cleared Aborting. Failed to wipe start of new LV. ] [kubeexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvcreate --poolmetadatasize 12288K -c 256K -L 2097152K -T vg_d386fc09ee51092f5e7ecd647dc1ec45/tp_444b6811d86b1f2c4ab9eb196a495239 -V 2097152K -n brick_444b6811d86b1f2c4ab9eb196a495239] on glusterfs-4c2sn: Err[command terminated with exit code 5]: Stdout [ Using default stripesize 64.00 KiB. ]: Stderr [ /dev/vg_d386fc09ee51092f5e7ecd647dc1ec45/lvol0: not found: device not cleared Aborting. Failed to wipe start of new LV. ] [kubeexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvcreate --poolmetadatasize 12288K -c 256K -L 2097152K -T vg_79068a61269ee2a564a5a6e0cc1037d6/tp_7b3f12de7e65ee2adc284be36db732e4 -V 2097152K -n brick_7b3f12de7e65ee2adc284be36db732e4] on glusterfs-s9f4g: Err[command terminated with exit code 5]: Stdout [ Using default stripesize 64.00 KiB. ]: Stderr [ /dev/vg_79068a61269ee2a564a5a6e0cc1037d6/lvol0: not found: device not cleared Aborting. Failed to wipe start of new LV. ] [kubeexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc] on glusterfs-hhtwh: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc: not mounted ] [sshexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-hhtwh: umount: /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc: not mounted [kubeexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239] on glusterfs-4c2sn: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239: not mounted ] [sshexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-4c2sn: umount: /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239: not mounted [kubeexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4] on glusterfs-s9f4g: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4: not mounted ] [sshexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-s9f4g: umount: /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4: not mounted [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 73.13µs [kubeexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_284e8a4d9dd8a261e0b7d6616a43ad6b/tp_0b967c98a211caf6a2f2ebc2af26b3cc] on glusterfs-hhtwh: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_284e8a4d9dd8a261e0b7d6616a43ad6b/tp_0b967c98a211caf6a2f2ebc2af26b3cc" ] [sshexec] ERROR 2017/07/19 09:12:57 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-hhtwh: Failed to find logical volume "vg_284e8a4d9dd8a261e0b7d6616a43ad6b/tp_0b967c98a211caf6a2f2ebc2af26b3cc" [kubeexec] ERROR 2017/07/19 09:12:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_d386fc09ee51092f5e7ecd647dc1ec45/tp_444b6811d86b1f2c4ab9eb196a495239] on glusterfs-4c2sn: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_d386fc09ee51092f5e7ecd647dc1ec45/tp_444b6811d86b1f2c4ab9eb196a495239" ] [sshexec] ERROR 2017/07/19 09:12:58 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-4c2sn: Failed to find logical volume "vg_d386fc09ee51092f5e7ecd647dc1ec45/tp_444b6811d86b1f2c4ab9eb196a495239" [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 60.494µs [kubeexec] ERROR 2017/07/19 09:12:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_79068a61269ee2a564a5a6e0cc1037d6/tp_7b3f12de7e65ee2adc284be36db732e4] on glusterfs-s9f4g: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_79068a61269ee2a564a5a6e0cc1037d6/tp_7b3f12de7e65ee2adc284be36db732e4" ] [sshexec] ERROR 2017/07/19 09:12:58 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-s9f4g: Failed to find logical volume "vg_79068a61269ee2a564a5a6e0cc1037d6/tp_7b3f12de7e65ee2adc284be36db732e4" [kubeexec] DEBUG 2017/07/19 09:12:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black1 Pod: glusterfs-hhtwh Command: rmdir /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc Result: [kubeexec] DEBUG 2017/07/19 09:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black3 Pod: glusterfs-4c2sn Command: rmdir /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239 Result: [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 72.625µs [kubeexec] DEBUG 2017/07/19 09:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:250: Host: black2 Pod: glusterfs-s9f4g Command: rmdir /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4 Result: [kubeexec] ERROR 2017/07/19 09:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [sed -i.save "/brick_0b967c98a211caf6a2f2ebc2af26b3cc/d" /var/lib/heketi/fstab] on glusterfs-hhtwh: Err[command terminated with exit code 2]: Stdout []: Stderr [sed: can't read /var/lib/heketi/fstab: No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:00 /src/github.com/heketi/heketi/executors/sshexec/brick.go:163: Unable to execute command on glusterfs-hhtwh: sed: can't read /var/lib/heketi/fstab: No such file or directory [kubeexec] ERROR 2017/07/19 09:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [sed -i.save "/brick_444b6811d86b1f2c4ab9eb196a495239/d" /var/lib/heketi/fstab] on glusterfs-4c2sn: Err[command terminated with exit code 2]: Stdout []: Stderr [sed: can't read /var/lib/heketi/fstab: No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:00 /src/github.com/heketi/heketi/executors/sshexec/brick.go:163: Unable to execute command on glusterfs-4c2sn: sed: can't read /var/lib/heketi/fstab: No such file or directory [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 68.045µs [kubeexec] ERROR 2017/07/19 09:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [sed -i.save "/brick_7b3f12de7e65ee2adc284be36db732e4/d" /var/lib/heketi/fstab] on glusterfs-s9f4g: Err[command terminated with exit code 2]: Stdout []: Stderr [sed: can't read /var/lib/heketi/fstab: No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:00 /src/github.com/heketi/heketi/executors/sshexec/brick.go:163: Unable to execute command on glusterfs-s9f4g: sed: can't read /var/lib/heketi/fstab: No such file or directory [heketi] ERROR 2017/07/19 09:13:00 /src/github.com/heketi/heketi/apps/glusterfs/brick_create.go:49: Unable to execute command on glusterfs-s9f4g: /dev/vg_79068a61269ee2a564a5a6e0cc1037d6/lvol0: not found: device not cleared Aborting. Failed to wipe start of new LV. [heketi] INFO 2017/07/19 09:13:00 Deleting brick 0b967c98a211caf6a2f2ebc2af26b3cc [heketi] INFO 2017/07/19 09:13:00 Deleting brick 444b6811d86b1f2c4ab9eb196a495239 [heketi] INFO 2017/07/19 09:13:00 Deleting brick 7b3f12de7e65ee2adc284be36db732e4 [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 58.106µs [kubeexec] ERROR 2017/07/19 09:13:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc] on glusterfs-hhtwh: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc: mountpoint not found ] [sshexec] ERROR 2017/07/19 09:13:01 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-hhtwh: umount: /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc: mountpoint not found [kubeexec] ERROR 2017/07/19 09:13:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239] on glusterfs-4c2sn: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239: mountpoint not found ] [sshexec] ERROR 2017/07/19 09:13:01 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-4c2sn: umount: /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239: mountpoint not found [kubeexec] ERROR 2017/07/19 09:13:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [umount /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4] on glusterfs-s9f4g: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4: mountpoint not found ] [sshexec] ERROR 2017/07/19 09:13:02 /src/github.com/heketi/heketi/executors/sshexec/brick.go:134: Unable to execute command on glusterfs-s9f4g: umount: /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4: mountpoint not found [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 44.794µs [kubeexec] ERROR 2017/07/19 09:13:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_284e8a4d9dd8a261e0b7d6616a43ad6b/tp_0b967c98a211caf6a2f2ebc2af26b3cc] on glusterfs-hhtwh: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_284e8a4d9dd8a261e0b7d6616a43ad6b/tp_0b967c98a211caf6a2f2ebc2af26b3cc" ] [sshexec] ERROR 2017/07/19 09:13:02 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-hhtwh: Failed to find logical volume "vg_284e8a4d9dd8a261e0b7d6616a43ad6b/tp_0b967c98a211caf6a2f2ebc2af26b3cc" [kubeexec] ERROR 2017/07/19 09:13:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_d386fc09ee51092f5e7ecd647dc1ec45/tp_444b6811d86b1f2c4ab9eb196a495239] on glusterfs-4c2sn: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_d386fc09ee51092f5e7ecd647dc1ec45/tp_444b6811d86b1f2c4ab9eb196a495239" ] [sshexec] ERROR 2017/07/19 09:13:03 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-4c2sn: Failed to find logical volume "vg_d386fc09ee51092f5e7ecd647dc1ec45/tp_444b6811d86b1f2c4ab9eb196a495239" [kubeexec] ERROR 2017/07/19 09:13:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [lvremove -f vg_79068a61269ee2a564a5a6e0cc1037d6/tp_7b3f12de7e65ee2adc284be36db732e4] on glusterfs-s9f4g: Err[command terminated with exit code 5]: Stdout []: Stderr [ Failed to find logical volume "vg_79068a61269ee2a564a5a6e0cc1037d6/tp_7b3f12de7e65ee2adc284be36db732e4" ] [sshexec] ERROR 2017/07/19 09:13:03 /src/github.com/heketi/heketi/executors/sshexec/brick.go:143: Unable to execute command on glusterfs-s9f4g: Failed to find logical volume "vg_79068a61269ee2a564a5a6e0cc1037d6/tp_7b3f12de7e65ee2adc284be36db732e4" [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 66.29µs [kubeexec] ERROR 2017/07/19 09:13:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [rmdir /var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc] on glusterfs-hhtwh: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc': No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:152: Unable to execute command on glusterfs-hhtwh: rmdir: failed to remove '/var/lib/heketi/mounts/vg_284e8a4d9dd8a261e0b7d6616a43ad6b/brick_0b967c98a211caf6a2f2ebc2af26b3cc': No such file or directory [kubeexec] ERROR 2017/07/19 09:13:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [rmdir /var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239] on glusterfs-4c2sn: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239': No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:152: Unable to execute command on glusterfs-4c2sn: rmdir: failed to remove '/var/lib/heketi/mounts/vg_d386fc09ee51092f5e7ecd647dc1ec45/brick_444b6811d86b1f2c4ab9eb196a495239': No such file or directory [kubeexec] ERROR 2017/07/19 09:13:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [rmdir /var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4] on glusterfs-s9f4g: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4': No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:04 /src/github.com/heketi/heketi/executors/sshexec/brick.go:152: Unable to execute command on glusterfs-s9f4g: rmdir: failed to remove '/var/lib/heketi/mounts/vg_79068a61269ee2a564a5a6e0cc1037d6/brick_7b3f12de7e65ee2adc284be36db732e4': No such file or directory [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 65.705µs [kubeexec] ERROR 2017/07/19 09:13:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [sed -i.save "/brick_0b967c98a211caf6a2f2ebc2af26b3cc/d" /var/lib/heketi/fstab] on glusterfs-hhtwh: Err[command terminated with exit code 2]: Stdout []: Stderr [sed: can't read /var/lib/heketi/fstab: No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:05 /src/github.com/heketi/heketi/executors/sshexec/brick.go:163: Unable to execute command on glusterfs-hhtwh: sed: can't read /var/lib/heketi/fstab: No such file or directory [kubeexec] ERROR 2017/07/19 09:13:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [sed -i.save "/brick_444b6811d86b1f2c4ab9eb196a495239/d" /var/lib/heketi/fstab] on glusterfs-4c2sn: Err[command terminated with exit code 2]: Stdout []: Stderr [sed: can't read /var/lib/heketi/fstab: No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:05 /src/github.com/heketi/heketi/executors/sshexec/brick.go:163: Unable to execute command on glusterfs-4c2sn: sed: can't read /var/lib/heketi/fstab: No such file or directory [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 200 OK in 68.013µs [kubeexec] ERROR 2017/07/19 09:13:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:247: Failed to run command [sed -i.save "/brick_7b3f12de7e65ee2adc284be36db732e4/d" /var/lib/heketi/fstab] on glusterfs-s9f4g: Err[command terminated with exit code 2]: Stdout []: Stderr [sed: can't read /var/lib/heketi/fstab: No such file or directory ] [sshexec] ERROR 2017/07/19 09:13:05 /src/github.com/heketi/heketi/executors/sshexec/brick.go:163: Unable to execute command on glusterfs-s9f4g: sed: can't read /var/lib/heketi/fstab: No such file or directory [asynchttp] INFO 2017/07/19 09:13:05 Completed job 9a482306dc68528394849604c621bea0 in 9.33985091s [heketi] ERROR 2017/07/19 09:13:05 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:154: Failed to create volume: Unable to execute command on glusterfs-s9f4g: /dev/vg_79068a61269ee2a564a5a6e0cc1037d6/lvol0: not found: device not cleared Aborting. Failed to wipe start of new LV. [negroni] Started GET /queue/9a482306dc68528394849604c621bea0 [negroni] Completed 500 Internal Server Error in 76.611µs