Intel NFV CI consists of:
1. networking CI (commenting to Neutron) - https://wiki.openstack.org/wiki/ThirdPartySystems/Intel-Networking-CI
2. pci/sriov CI (commenting to Nova) - https://wiki.openstack.org/wiki/ThirdPartySystems/Intel-PCI-CI
3. compute (numa) CI (in works, to comment to Nova)

Networking CI is the topic today.

1. Purpose:
To test networking-ovs-dpdk (OpenvSwitch on Intel's DPDK, running in userspace) with latest upstream Openstack code proposals from gerrit.openstack.org. Triggered on each new patchset.

2. Flow:
gerrit.openstack.org -> Jenkins (Gerrit trigger plugin) -> baremetal node -> docker (lxc) container -> devstack -> tempest -> gerrit.openstack.org (comment), http://intel-openstack-ci-logs.ovh/networking-ci/ (artifacts)

3. containers pros:
1. Easy to work with multiple openstack instances on the same baremetal - container is designed to isolate its filesystem and processes from the host machine and other containers.
- Cleaner filesystem. Proposed patchsets can sometimes mess what packages are being installed to the virtuaenv/operating system (install new python (pip) packages/system packages). Operating system and/or packages may be left unusuable/not (easily) downgradable. This problem does not affect containers as each of them is spawned from a base image having a fixed shape of an Operating System/packages.
- Another layer of protection (compared to running on baremetal) in case of intrusion or process frenzy. Also, containers are brought down after a Jenkins build is done so any security breach (intentionous or accidental) is shortlived .
2. Faster than a Virtual Machine due to lack of hypervisor/hardware emulating layers. Prepping, starting a container and stacking (devstack) in our Networking CI takes 5-7mins.
3. Access to new and hot features in hardware (HW offloads, SRIOV etc) from inside a Virtual Machine can be/usually is limited. Containers can have access to all hardware features as there's no hypervisor (middle layer) that would have to emulate these in case of a VM.

4. containers cons:
1. Some resources that are used from withing multiple containers at the same time can be a bottleneck/prone to race condition.
- not advised to load/unload kernel modules from within a container - containers share kernel (the one host was started with)
- huge pages (allocated on the host) are shared amongst all containers, sometimes that needs a locking/monitoring mechanism to overcome some limitations
2. Sometimes buggy/limited container/docker operation - fairly young technology.

5. workarounds, hints, scripts:
1. Use ‘--privileged’ container allows access to hosts /dev from inside the container - less secure but gives you access to hardware capabilities.

2. Use as few connections (attachments) to container as possible to execute different parts of the build. i.e.: 1. attach and save environment files, 2. attach and stack, 3. attach to do cleanup. Reason for that is when you attach to the container and run commands, they may not finish before you detach from the container - leaving the processes with no parents = zombies = children of PID 0 = sometimes hanging. You want to limit the amount of these Z processes as they may (but not limited to) affect future processing and affect your build.

3. Mount a volume/fileshare from the host to be used as logs directory inside the container. It makes the log files available for troubleshooting even when the container is stopped or deleted (i.e.: /var/lib/jenkins/build-X/logs is mounted as /opt/stack/logs inside the container).

4. Do some clean up pre-stacking, i.e: remove mariadb/rabbitmq logs/locks/database. While it's not needed with a completely fresh image (as you should not have these logs/locks/data) as you develop and change/save your image you may have artifacts of previous stacking left on the filesystem. Do 'shutdown -h now' inside the container if you plan to save it as a base image for future builds - it stops services = removes above and other lock files so next time you stack you won't end up with 'lock file exists but service not running' situation.

5. lvm inside the container needs to be repointed to look for lvm configured devices not in /dev - we’ve used /lvm - to prevent openstack/devstack running in a container colliding with host’s/other container’s LVM configuration

6. When starting a container mount /var/cache from your host as /var/cache inside a container - it may speed up installing prereqs (yum/apt/pip) for Openstack.