OpenStack Kilo and Docker: CentOS 7 Integration

Posted on Posted in OpenStack, Teknocratica No Comments Tagged with , ,

Found a number of references but here’s the recipe from zero to hero.

References

Native Docker Install

  1. Install Docker on Nova Compute host: 
    yum install -y docker
  2. Optional: On my Nova Compute host, the root partition (where /var/lib lives) is only 50GB. So I move Docker to a larger partition and soft-link: 
    
mkdir -p /home/rootdir/var/lib
mv /var/lib/docker /home/rootdir/var/lib/
ln -fs /home/rootdir/var/lib/docker /var/lib/docker
  3. Setup Docker to support 4TB filesystem (default is 100GB limit). Note we create sparse file; unsure if this is required but remembering that OpenStack will be looking at available disk space before scheduling host. Note that this destroys any Docker images / running containers. That’s why we do it now, before we’ve run anything. 
    rm -fR /var/lib/docker/*
mkdir -p /var/lib/docker/devicemapper/devicemapper
dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=4096
  4. There are terrible things discussed about the default devicemapper driver with Docker. Many suggest to use overlayfs. Too bad it doesn’t work with XFS file systems on CentOS 7…and that XFS is the file system used by default when creating new partitions during Anaconda install. So…if you want to use overlayfs, follow these steps. I have not. 
    # all this commented out because CentOS 7 xfs doesn't work with overlayfs (which is "tech preview" in CentOS 7):
# sed -i -e "s#^\(OPTIONS='--selinux-enabled\)\('\)#\1=false\2#" /etc/sysconfig/docker
# sed -i -e 's#^\(DOCKER_STORAGE_OPTIONS\)\([^=]*\).*#\1 = -s overlay#' /etc/sysconfig/docker-storage
  5. Now we can start Docker: 
    systemctl start docker
systemctl enable docker
  6. Verify we have 4TB disk: 
    [root@lposhostx076 proj]# docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
 Pool Name: docker-253:2-26843546688-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 307.2 MB
 Data Space Total: 4.398 TB
 Data Space Available: 4.398 TB
 Metadata Space Used: 17.36 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.13 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /home/rootdir/var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /home/rootdir/var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 32
Total Memory: 251.9 GiB
Name: lposhostx076.hlsdev.local

Nova Compute Integration

  1. Stop Docker so we can continue with Nova Compute: 
    systemctl stop docker
  2. Add Nova Compute user to dockerroot group for Docker unixsocket access. Note that this is deprecated by the Atomic Project article above, but that OpenStack Nova Compute does actually need to run Docker 🙂 
    usermod -a -G dockerroot nova
  3. Install driver on Nova Compute, then restart Docker. Note these instructions are specific to OpenStack Kilo and will certainly change for other releases: 
    yum install -y python-pip
mkdir -p /root/proj
cd /root/proj
git clone https://github.com/stackforge/nova-docker.git
git checkout -b kilo origin/stable/kilo
git branch -v -a
python setup.py install
systemctl start docker
  4. Permit Nova to access docker via socket. Unfortunately, this must be performed each time Docker restarted (I have not yet automated it): 
    chgrp dockerroot /var/run/docker.sock
  5. Modify /etc/nova/nova.conf to tell Nova Compute to use the Docker hypervisor driver: 
    [DEFAULT]
compute_driver=novadocker.virt.docker.DockerDriver
  6. Setup rootwrap so that Nova Compute user can execute Docker commands: 
    [ ! -d /etc/nova/rootwrap.d ] && mkdir -p /etc/nova/rootwrap.d
cat > /etc/nova/rootwrap.d/docker.filters <<EOF
# nova-rootwrap command filters for setting up network in the docker driver
# This file should be owned by (and only-writeable by) the root user

[Filters] 
# nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'
ln: CommandFilter, /bin/ln, root
EOF
  7. Patch Kilo docker driver; by default it only permits Linux Containers (lxc). Added “docker” to the list below for x86 and x64: 
    # /usr/lib/python2.7/site-packages/novadocker/virt/docker/driver.py
...
            'supported_instances': jsonutils.dumps([
                ('i686', 'docker', 'lxc'), 
                ('x86_64', 'docker', 'lxc')
            ])
...
  8. Restart Compute: 
    systemctl restart openstack-nova-compute

Glance Integration

On your Glance host, run the following:

  1. Modify /etc/glance/glance-api.conf: 
    [DEFAULT]
container_formats=ami,ari,aki,bare,ovf,ova,docker
  2. Restart Glance: 
    systemctl restart openstack-glance-api

Nova Scheduler Integration

On your Nova Controller (running openstack-nova-scheduler), run the following steps. These modify the Nova Scheduler to place Docker instances only on Docker-enabled Nova Compute hosts. The steps also assume you have sourced in the appropriate OpenStack credentials file:

  1. Assuming you are using default KVM (also displayed as QEMU although – as you well know – they are not the same thing), modify all existing flavors to have a new attribute indicating they are KVM: 
    for i in $(openstack flavor list | tail -n +4 | head -n -1 | awk -F'|' '{print $3}' | sed -e 's# ##g'); do
  if echo $i | grep --quiet -v '^docker\.'; then
    openstack flavor set --property aggregate_instance_extra_specs:virt_type=kvm $i
  fi
done

    2. Create a new flavor, and set attribute for Docker:

  2. if ! openstack flavor show docker.m1.small >/dev/null 2>&1; then
  openstack flavor create --ram 2048 --disk 20 --vcpus 1 docker.m1.small
fi
openstack flavor set --property aggregate_instance_extra_specs:virt_type=docker docker.m1.small
  3. We will use OpenStack Host Aggregates to provide our filtering. We create two aggregates; one for Docker and one for KVM. If you have additional hypervisors, modify accordingly: 
    if ! openstack aggregate show docker-host >/dev/null 2>&1; then
  openstack aggregate create --zone nova docker-host
fi
openstack aggregate set --property virt_type=docker docker-host
if ! openstack aggregate show kvm-host >/dev/null 2>&1; then
  openstack aggregate create --zone nova kvm-host
fi
openstack aggregate set --property virt_type=kvm kvm-host
  4. Add hosts to each aggregate: 
    for i in $(openstack hypervisor list | tail -n +4 | head -n -1 | awk -F'|' '{print $2}' | sed -e 's# ##g'); do
  l_aggregate_name_add='kvm-host'
  l_aggregate_name_remove='docker-host'
  l_hypervisor_hostname=$(openstack hypervisor show $i | grep -e '^| \+hypervisor_hostname \+|' | awk -F'|' '{print $3}' | sed -e 's#^ \+##' -e 's# \+$##')
  l_hypervisor_type=$(openstack hypervisor show $i | grep -e '^| \+hypervisor_type \+|' | awk -F'|' '{print $3}' | sed -e 's#^ \+##' -e 's# \+$##')
  if [ "x$l_hypervisor_type" = "xdocker" ]; then
    l_aggregate_name_add='docker-host'
    l_aggregate_name_remove='kvm-host'
  fi
  if openstack aggregate show "$l_aggregate_name_remove" | grep -e '^| \+hosts \+|' | grep --quiet "'$l_hypervisor_hostname'"; then
    echo openstack aggregate remove host "$l_aggregate_name_remove" $l_hypervisor_hostname
  fi
  if ! openstack aggregate show "$l_aggregate_name_add" | grep -e '^| \+hosts \+|' | grep --quiet "'$l_hypervisor_hostname'"; then
    echo openstack aggregate add host "$l_aggregate_name_add" $l_hypervisor_hostname
  fi
done
  5. Modify Nova Scheduler via /etc/nova/nova.conf to use the AggregateInstanceExtraSpecsFilter so that the virt_type property we set above will be used to filter out hosts that have incompatible hypervisors: 
    [DEFAULT]
scheduler_default_filters=AggregateInstanceExtraSpecsFilter,RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
  6. Restart Nova Scheduler: 
    systemctl restart openstack-nova-scheduler

Proof of the Pudding is in the Tasting

We first integrated a simple Ubuntu SSHD Docker image:

docker pull rastasheep/ubuntu-sshd:14.04
docker save rastasheep/ubuntu-sshd:14.04 > rastasheep_ubuntu-sshd:14.04
glance image-create --architecture x86_64 --name 'rastasheep/ubuntu-sshd:14.04' \
  --visibility public --os-version 14.04 --disk-format raw --os-distro ubuntu \
  --container-format docker --file ./rastasheep_ubuntu-sshd\:14.04 \
  --progress

Verify it’s on file:

[root@lposhostx076 nova(lvosksclu120-rc-admin)]# glance image-show $(glance image-list | grep rastasheep | awk -F'|' '{print $2}')
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| architecture     | x86_64                               |
| checksum         | d78a9ff8d12f9e91cd5387e40d36fa9f     |
| container_format | docker                               |
| created_at       | 2015-09-21T21:33:58Z                 |
| disk_format      | raw                                  |
| id               | ba0ba9e0-630f-42b6-8f73-fca19f3c7ee6 |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | rastasheep/ubuntu-sshd:14.04         |
| os_distro        | ubuntu                               |
| os_version       | 14.04                                |
| owner            | f1397ff9930c4c20bc0712eba2555e43     |
| protected        | False                                |
| size             | 264800256                            |
| status           | active                               |
| tags             | []                                   |
| updated_at       | 2015-09-21T21:34:05Z                 |
| virtual_size     | None                                 |
| visibility       | public                               |
+------------------+--------------------------------------+

Now boot an instance from the image. I won’t show the nova boot commands because it uses lots of ID values; just use Horizon for this. When you are done, you can see the instance (I’ve named it dockertest and I used a project named sab-sadmin-dev. This also demonstrates a funny OpenStack one-liner:

[root@lposhostx076 nova(lvosksclu120-rc-admin)]# openstack server show $(openstack --os-project-id="$(openstack project list | grep sab-sadmin-dev | awk -F'|' '{print $2}' | sed -e 's# ##g')" server list | grep dockertest | awk -F'|' '{print $2}')
+--------------------------------------+---------------------------------------------------------------------+
| Field                                | Value                                                               |
+--------------------------------------+---------------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                                |
| OS-EXT-AZ:availability_zone          | nova                                                                |
| OS-EXT-SRV-ATTR:host                 | lposhostx076.hlsdev.local                                           |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | lposhostx076.hlsdev.local                                           |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000129                                                   |
| OS-EXT-STS:power_state               | 1                                                                   |
| OS-EXT-STS:task_state                | None                                                                |
| OS-EXT-STS:vm_state                  | active                                                              |
| OS-SRV-USG:launched_at               | 2015-09-23T13:57:09.000000                                          |
| OS-SRV-USG:terminated_at             | None                                                                |
| accessIPv4                           |                                                                     |
| accessIPv6                           |                                                                     |
| addresses                            | sab-sadmin-dev-net=10.0.0.144, 172.20.142.21                       |
| config_drive                         |                                                                     |
| created                              | 2015-09-23T13:57:04Z                                                |
| flavor                               | docker.m1.small (c7d7df07-ff66-46e5-8cd9-a98136cfaa2a)              |
| hostId                               | 67cbd0497e9600dad5b8b630297e4bb8e2e805a87170a730cf7bf98f            |
| id                                   | b1a5d738-a7c4-4009-a712-7514517b931d                                |
| image                                | rastasheep/ubuntu-sshd:14.04 (ba0ba9e0-630f-42b6-8f73-fca19f3c7ee6) |
| key_name                             | new-admin                                                           |
| name                                 | dockertest                                                          |
| os-extended-volumes:volumes_attached | []                                                                  |
| progress                             | 0                                                                   |
| project_id                           | 23a18e23522e4f55a08828da48e2b547                                    |
| properties                           |                                                                     |
| security_groups                      | [{u'name': u'sab-sadmin-dev-icmp-ssh'}, {u'name': u'default'}]     |
| status                               | ACTIVE                                                              |
| updated                              | 2015-09-23T13:57:09Z                                                |
| user_id                              | b0ebea8f7597b11f32b9cd71748c73b602d8ac7722b1e5bfe1fe42b32b7ead20    |
+--------------------------------------+---------------------------------------------------------------------+

Yup, we are using the docker flavor defined above, and the Ubuntu SSHD image we imported to Glance.

Here’s what it looks like on the Nova Compute hypervisor:

[root@lposhostx076 nova(lvosksclu120-rc-admin)]# docker ps
CONTAINER ID        IMAGE                          COMMAND               CREATED             STATUS              PORTS               NAMES
6fe07a32a58f        rastasheep/ubuntu-sshd:14.04   "/usr/sbin/sshd -D"   About an hour ago   Up About an hour                        nova-b1a5d738-a7c4-4009-a712-7514517b931d   
[root@lposhostx076 nova(lvosksclu120-rc-admin)]# docker inspect 6fe07a32a58f
[
{
    "Id": "6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6",
    "Created": "2015-09-23T13:57:06.522408383Z",
    "Path": "/usr/sbin/sshd",
    "Args": [
        "-D"
    ],
    "State": {
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 38562,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2015-09-23T13:57:06.861698662Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "2069c778a8155d55245e1b37e382841b5e78d296d2c5a2d5b4f9714e2859417b",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "NetworkID": "",
        "PortMapping": null,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "ResolvConfPath": "",
    "HostnamePath": "/home/rootdir/var/lib/docker/containers/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6/hostname",
    "HostsPath": "",
    "LogPath": "/home/rootdir/var/lib/docker/containers/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6-json.log",
    "Name": "/nova-b1a5d738-a7c4-4009-a712-7514517b931d",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c138,c900",
    "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c138,c900",
    "Volumes": {},
    "VolumesRW": {},
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemorySwap": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "Privileged": false,
        "PortBindings": null,
        "Links": null,
        "PublishAllPorts": false,
        "Dns": [
            "192.168.1.2"
        ],
        "DnsSearch": null,
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": null,
        "NetworkMode": "bridge",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": ""
    },
    "Config": {
        "Hostname": "instance-00000129",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": true,
        "AttachStderr": true,
        "PortSpecs": null,
        "ExposedPorts": {
            "22/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/usr/sbin/sshd",
            "-D"
        ],
        "Image": "rastasheep/ubuntu-sshd:14.04",
        "Volumes": null,
        "VolumeDriver": "",
        "WorkingDir": "",
        "Entrypoint": null,
        "NetworkDisabled": true,
        "MacAddress": "",
        "OnBuild": null,
        "Labels": {}
    }
}
]

That is all. Enjoy!

About

Team-oriented systems mentor with deep knowledge of numerous software methodologies, technologies, languages, and operating systems. Excited about turning emerging technology into working production-ready systems. Focused on moving software teams to a higher level of world-class application development. Specialties:Software analysis and development...Product management through the entire lifecycle...Discrete product integration specialist!

Leave a Reply

Your email address will not be published. Required fields are marked *

*