osa-debug-utility-container-pip-packages

Did a fresh utility container install, failed on install of certain pip packages
* python-openstackclient
* python-glanceclient
---------

TASK: [Install pip packages] **************************************************
failed: [aio1_utility_container-3a6437f8] => (item=python-openstackclient) => {"attempts": 1, "cmd": "/usr/local/bin/pip install python-openstackclient", "failed": true, "item": "python-openstackclient"}
msg: Task failed as maximum retries was encountered
changed: [aio1_utility_container-3a6437f8] => (item=python-ceilometerclient)
changed: [aio1_utility_container-3a6437f8] => (item=python-cinderclient)
failed: [aio1_utility_container-3a6437f8] => (item=python-glanceclient) => {"attempts": 1, "cmd": "/usr/local/bin/pip install python-glanceclient", "failed": true, "item": "python-glanceclient"}
msg: Task failed as maximum retries was encountered
changed: [aio1_utility_container-3a6437f8] => (item=python-heatclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-keystoneclient)
changed: [aio1_utility_container-3a6437f8] => (item=python-neutronclient)
changed: [aio1_utility_container-3a6437f8] => (item=python-novaclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-swiftclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-memcached)
ok: [aio1_utility_container-3a6437f8] => (item=pycrypto)

FATAL: all hosts have already failed -- aborting

--------------------

Then reran the utility install in full, everything is green except for the same packages fail to install.

--------------------

TASK: [Install pip packages] **************************************************
failed: [aio1_utility_container-3a6437f8] => (item=python-openstackclient) => {"attempts": 1, "cmd": "/usr/local/bin/pip install python-openstackclient", "failed": true, "item": "python-openstackclient"}
msg: Task failed as maximum retries was encountered
ok: [aio1_utility_container-3a6437f8] => (item=python-ceilometerclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-cinderclient)
failed: [aio1_utility_container-3a6437f8] => (item=python-glanceclient) => {"attempts": 1, "cmd": "/usr/local/bin/pip install python-glanceclient", "failed": true, "item": "python-glanceclient"}
msg: Task failed as maximum retries was encountered
ok: [aio1_utility_container-3a6437f8] => (item=python-heatclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-keystoneclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-neutronclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-novaclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-swiftclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-memcached)
ok: [aio1_utility_container-3a6437f8] => (item=pycrypto)

----------------------

Attach to the container and check what is installed with pip

$ lxc-attach [container]
$ sudo pip list
Babel (2.2.0)
cliff (1.10.1)
cmd2 (0.6.8)
futures (3.0.4)
iso8601 (0.1.11)
msgpack-python (0.4.7)
MySQL-python (1.2.5)
netaddr (0.7.18)
netifaces (0.10.4)
oslo.config (1.9.3)
oslo.i18n (1.5.0)
oslo.serialization (1.4.0)
oslo.utils (1.4.2)
pbr (0.11.1)
pip (7.1.2)
prettytable (0.7.2)
pycrypto (2.6.1)
pycurl (7.19.3)
pyparsing (2.1.0)
python-apt (0.9.3.5ubuntu2)
python-ceilometerclient (1.1.2)
python-cinderclient (1.1.2)
python-heatclient (0.4.0.post2)
python-keystoneclient (1.3.4)
python-memcached (1.57)
python-neutronclient (2.4.0)
python-novaclient (2.23.3)
python-swiftclient (2.4.0)
pytz (2015.7)
PyYAML (3.11)
requests (2.7.0)
setuptools (20.0)
simplejson (3.8.1)
six (1.10.0)
stevedore (1.3.0)
wheel (0.29.0)

The failed clients are missing, as expected.

Try installing them manually with pip.

$ sudo pip install python-openstackclient

This works, and also installs the missing glanceclient

$ sudo pip list
appdirs (1.4.0)
Babel (2.2.0)
cliff (1.16.0)
cmd2 (0.6.8)
debtcollector (1.3.0)
funcsigs (0.4)
functools32 (3.2.3.post2)
futures (3.0.4)
iso8601 (0.1.11)
jsonpatch (1.13)
jsonpointer (1.10)
jsonschema (2.5.1)
keystoneauth1 (2.2.0)
monotonic (0.6)
msgpack-python (0.4.7)
MySQL-python (1.2.5)
netaddr (0.7.18)
netifaces (0.10.4)
openstacksdk (0.7.4)
os-client-config (1.14.0)
oslo.config (3.6.0)
oslo.i18n (3.3.0)
oslo.serialization (2.3.0)
oslo.utils (3.5.0)
pbr (1.8.1)
pip (7.1.2)
prettytable (0.7.2)
pycrypto (2.6.1)
pycurl (7.19.3)
pyparsing (2.1.0)
python-apt (0.9.3.5ubuntu2)
python-ceilometerclient (1.1.2)
python-cinderclient (1.5.0)
python-glanceclient (1.2.0)
python-heatclient (0.4.0.post2)
python-keystoneclient (2.1.2)
python-memcached (1.57)
python-neutronclient (2.4.0)
python-novaclient (3.2.0)
python-openstackclient (2.1.0)
python-swiftclient (2.4.0)
pytz (2015.7)
PyYAML (3.11)
requests (2.9.1)
requestsexceptions (1.1.3)
setuptools (20.0)
simplejson (3.8.1)
six (1.10.0)
stevedore (1.11.0)
unicodecsv (0.14.1)
warlock (1.2.0)
wheel (0.29.0)
wrapt (1.10.6)

Not only does this install a few new packages, but it also updates a lot of versions.
https://www.diffchecker.com/1li2rqti

$ sudo pip install python-glanceclient

As expected, this doesn't install anything as the previous command already installed it.

Back on the host, try to run utility container playbook again.
Fails again, but this time with different set of packages failing.

Fails:
* python-cinderclient
* python-glanceclient
* python-novaclient

So openstackclient installs ok, glance still fails and now cinder/nova show up as failed. Also ceilometer and neutron clients show us changed instead of ok.

ok: [aio1_utility_container-3a6437f8] => (item=python-openstackclient)
changed: [aio1_utility_container-3a6437f8] => (item=python-ceilometerclient)
failed: [aio1_utility_container-3a6437f8] => (item=python-cinderclient) => {"attempts": 1, "cmd": "/usr/local/bin/pip install python-cinderclient", "failed": true, "item": "python-cinderclient"}
msg: Task failed as maximum retries was encountered
failed: [aio1_utility_container-3a6437f8] => (item=python-glanceclient) => {"attempts": 1, "cmd": "/usr/local/bin/pip install python-glanceclient", "failed": true, "item": "python-glanceclient"}
msg: Task failed as maximum retries was encountered
ok: [aio1_utility_container-3a6437f8] => (item=python-heatclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-keystoneclient)
changed: [aio1_utility_container-3a6437f8] => (item=python-neutronclient)
failed: [aio1_utility_container-3a6437f8] => (item=python-novaclient) => {"attempts": 1, "cmd": "/usr/local/bin/pip install python-novaclient", "failed": true, "item": "python-novaclient"}
msg: Task failed as maximum retries was encountered
ok: [aio1_utility_container-3a6437f8] => (item=python-swiftclient)
ok: [aio1_utility_container-3a6437f8] => (item=python-memcached)
ok: [aio1_utility_container-3a6437f8] => (item=pycrypto)