OpenStack Kilo and Docker: CentOS 7 Integration

Found a number of references but here’s the recipe from zero to hero.

References

Native Docker Install

  1. Install Docker on Nova Compute host:
    yum install -y docker
  2. Optional: On my Nova Compute host, the root partition (where /var/lib lives) is only 50GB. So I move Docker to a larger partition and soft-link:
    
    mkdir -p /home/rootdir/var/lib
    mv /var/lib/docker /home/rootdir/var/lib/
    ln -fs /home/rootdir/var/lib/docker /var/lib/docker
  3. Setup Docker to support 4TB filesystem (default is 100GB limit). Note we create sparse file; unsure if this is required but remembering that OpenStack will be looking at available disk space before scheduling host. Note that this destroys any Docker images / running containers. That’s why we do it now, before we’ve run anything.
    rm -fR /var/lib/docker/*
    mkdir -p /var/lib/docker/devicemapper/devicemapper
    dd if=/dev/zero of=/var/lib/docker/devicemapper/devicemapper/data bs=1G count=0 seek=4096
  4. There are terrible things discussed about the default devicemapper driver with Docker. Many suggest to use overlayfs. Too bad it doesn’t work with XFS file systems on CentOS 7…and that XFS is the file system used by default when creating new partitions during Anaconda install. So…if you want to use overlayfs, follow these steps. I have not.
    # all this commented out because CentOS 7 xfs doesn't work with overlayfs (which is "tech preview" in CentOS 7):
    # sed -i -e "s#^\(OPTIONS='--selinux-enabled\)\('\)#\1=false\2#" /etc/sysconfig/docker
    # sed -i -e 's#^\(DOCKER_STORAGE_OPTIONS\)\([^=]*\).*#\1 = -s overlay#' /etc/sysconfig/docker-storage
  5. Now we can start Docker:
    systemctl start docker
    systemctl enable docker
  6. Verify we have 4TB disk:
    [root@lposhostx076 proj]# docker info
    Containers: 0
    Images: 0
    Storage Driver: devicemapper
     Pool Name: docker-253:2-26843546688-pool
     Pool Blocksize: 65.54 kB
     Backing Filesystem: xfs
     Data file: /dev/loop0
     Metadata file: /dev/loop1
     Data Space Used: 307.2 MB
     Data Space Total: 4.398 TB
     Data Space Available: 4.398 TB
     Metadata Space Used: 17.36 MB
     Metadata Space Total: 2.147 GB
     Metadata Space Available: 2.13 GB
     Udev Sync Supported: true
     Deferred Removal Enabled: false
     Data loop file: /home/rootdir/var/lib/docker/devicemapper/devicemapper/data
     Metadata loop file: /home/rootdir/var/lib/docker/devicemapper/devicemapper/metadata
     Library Version: 1.02.93-RHEL7 (2015-01-28)
    Execution Driver: native-0.2
    Logging Driver: json-file
    Kernel Version: 3.10.0-229.14.1.el7.x86_64
    Operating System: CentOS Linux 7 (Core)
    CPUs: 32
    Total Memory: 251.9 GiB
    Name: lposhostx076.hlsdev.local

Nova Compute Integration

  1. Stop Docker so we can continue with Nova Compute:
    systemctl stop docker
  2. Add Nova Compute user to dockerroot group for Docker unixsocket access. Note that this is deprecated by the Atomic Project article above, but that OpenStack Nova Compute does actually need to run Docker 🙂
    usermod -a -G dockerroot nova
  3. Install driver on Nova Compute, then restart Docker. Note these instructions are specific to OpenStack Kilo and will certainly change for other releases:
    yum install -y python-pip
    mkdir -p /root/proj
    cd /root/proj
    git clone https://github.com/stackforge/nova-docker.git
    git checkout -b kilo origin/stable/kilo
    git branch -v -a
    python setup.py install
    systemctl start docker
  4. Permit Nova to access docker via socket. Unfortunately, this must be performed each time Docker restarted (I have not yet automated it):
    chgrp dockerroot /var/run/docker.sock
  5. Modify /etc/nova/nova.conf to tell Nova Compute to use the Docker hypervisor driver:
    [DEFAULT]
    compute_driver=novadocker.virt.docker.DockerDriver
  6. Setup rootwrap so that Nova Compute user can execute Docker commands:
    [ ! -d /etc/nova/rootwrap.d ] && mkdir -p /etc/nova/rootwrap.d
    cat > /etc/nova/rootwrap.d/docker.filters <<EOF
    # nova-rootwrap command filters for setting up network in the docker driver
    # This file should be owned by (and only-writeable by) the root user
    
    [Filters] 
    # nova/virt/docker/driver.py: 'ln', '-sf', '/var/run/netns/.*'
    ln: CommandFilter, /bin/ln, root
    EOF
  7. Patch Kilo docker driver; by default it only permits Linux Containers (lxc). Added “docker” to the list below for x86 and x64:
    # /usr/lib/python2.7/site-packages/novadocker/virt/docker/driver.py
    ...
                'supported_instances': jsonutils.dumps([
                    ('i686', 'docker', 'lxc'), 
                    ('x86_64', 'docker', 'lxc')
                ])
    ...
  8. Restart Compute:
    systemctl restart openstack-nova-compute

Glance Integration

On your Glance host, run the following:

  1. Modify /etc/glance/glance-api.conf:
    [DEFAULT]
    container_formats=ami,ari,aki,bare,ovf,ova,docker
  2. Restart Glance:
    systemctl restart openstack-glance-api

Nova Scheduler Integration

On your Nova Controller (running openstack-nova-scheduler), run the following steps. These modify the Nova Scheduler to place Docker instances only on Docker-enabled Nova Compute hosts. The steps also assume you have sourced in the appropriate OpenStack credentials file:

  1. Assuming you are using default KVM (also displayed as QEMU although – as you well know – they are not the same thing), modify all existing flavors to have a new attribute indicating they are KVM:
    for i in $(openstack flavor list | tail -n +4 | head -n -1 | awk -F'|' '{print $3}' | sed -e 's# ##g'); do
      if echo $i | grep --quiet -v '^docker\.'; then
        openstack flavor set --property aggregate_instance_extra_specs:virt_type=kvm $i
      fi
    done
  2. Create a new flavor, and set attribute for Docker:

  3. if ! openstack flavor show docker.m1.small >/dev/null 2>&1; then
      openstack flavor create --ram 2048 --disk 20 --vcpus 1 docker.m1.small
    fi
    openstack flavor set --property aggregate_instance_extra_specs:virt_type=docker docker.m1.small
  4. We will use OpenStack Host Aggregates to provide our filtering. We create two aggregates; one for Docker and one for KVM. If you have additional hypervisors, modify accordingly:
    if ! openstack aggregate show docker-host >/dev/null 2>&1; then
      openstack aggregate create --zone nova docker-host
    fi
    openstack aggregate set --property virt_type=docker docker-host
    if ! openstack aggregate show kvm-host >/dev/null 2>&1; then
      openstack aggregate create --zone nova kvm-host
    fi
    openstack aggregate set --property virt_type=kvm kvm-host
  5. Add hosts to each aggregate:
    for i in $(openstack hypervisor list | tail -n +4 | head -n -1 | awk -F'|' '{print $2}' | sed -e 's# ##g'); do
      l_aggregate_name_add='kvm-host'
      l_aggregate_name_remove='docker-host'
      l_hypervisor_hostname=$(openstack hypervisor show $i | grep -e '^| \+hypervisor_hostname \+|' | awk -F'|' '{print $3}' | sed -e 's#^ \+##' -e 's# \+$##')
      l_hypervisor_type=$(openstack hypervisor show $i | grep -e '^| \+hypervisor_type \+|' | awk -F'|' '{print $3}' | sed -e 's#^ \+##' -e 's# \+$##')
      if [ "x$l_hypervisor_type" = "xdocker" ]; then
        l_aggregate_name_add='docker-host'
        l_aggregate_name_remove='kvm-host'
      fi
      if openstack aggregate show "$l_aggregate_name_remove" | grep -e '^| \+hosts \+|' | grep --quiet "'$l_hypervisor_hostname'"; then
        echo openstack aggregate remove host "$l_aggregate_name_remove" $l_hypervisor_hostname
      fi
      if ! openstack aggregate show "$l_aggregate_name_add" | grep -e '^| \+hosts \+|' | grep --quiet "'$l_hypervisor_hostname'"; then
        echo openstack aggregate add host "$l_aggregate_name_add" $l_hypervisor_hostname
      fi
    done
  6. Modify Nova Scheduler via /etc/nova/nova.conf to use the AggregateInstanceExtraSpecsFilter so that the virt_type property we set above will be used to filter out hosts that have incompatible hypervisors:
    [DEFAULT]
    scheduler_default_filters=AggregateInstanceExtraSpecsFilter,RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
  7. Restart Nova Scheduler:
    systemctl restart openstack-nova-scheduler

Proof of the Pudding is in the Tasting

We first integrated a simple Ubuntu SSHD Docker image:

docker pull rastasheep/ubuntu-sshd:14.04
docker save rastasheep/ubuntu-sshd:14.04 > rastasheep_ubuntu-sshd:14.04
glance image-create --architecture x86_64 --name 'rastasheep/ubuntu-sshd:14.04' \
  --visibility public --os-version 14.04 --disk-format raw --os-distro ubuntu \
  --container-format docker --file ./rastasheep_ubuntu-sshd\:14.04 \
  --progress

Verify it’s on file:

[root@lposhostx076 nova(lvosksclu120-rc-admin)]# glance image-show $(glance image-list | grep rastasheep | awk -F'|' '{print $2}')
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| architecture     | x86_64                               |
| checksum         | d78a9ff8d12f9e91cd5387e40d36fa9f     |
| container_format | docker                               |
| created_at       | 2015-09-21T21:33:58Z                 |
| disk_format      | raw                                  |
| id               | ba0ba9e0-630f-42b6-8f73-fca19f3c7ee6 |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | rastasheep/ubuntu-sshd:14.04         |
| os_distro        | ubuntu                               |
| os_version       | 14.04                                |
| owner            | f1397ff9930c4c20bc0712eba2555e43     |
| protected        | False                                |
| size             | 264800256                            |
| status           | active                               |
| tags             | []                                   |
| updated_at       | 2015-09-21T21:34:05Z                 |
| virtual_size     | None                                 |
| visibility       | public                               |
+------------------+--------------------------------------+

Now boot an instance from the image. I won’t show the nova boot commands because it uses lots of ID values; just use Horizon for this. When you are done, you can see the instance (I’ve named it dockertest and I used a project named sab-sadmin-dev. This also demonstrates a funny OpenStack one-liner:

[root@lposhostx076 nova(lvosksclu120-rc-admin)]# openstack server show $(openstack --os-project-id="$(openstack project list | grep sab-sadmin-dev | awk -F'|' '{print $2}' | sed -e 's# ##g')" server list | grep dockertest | awk -F'|' '{print $2}')
+--------------------------------------+---------------------------------------------------------------------+
| Field                                | Value                                                               |
+--------------------------------------+---------------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                                |
| OS-EXT-AZ:availability_zone          | nova                                                                |
| OS-EXT-SRV-ATTR:host                 | lposhostx076.hlsdev.local                                           |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | lposhostx076.hlsdev.local                                           |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000129                                                   |
| OS-EXT-STS:power_state               | 1                                                                   |
| OS-EXT-STS:task_state                | None                                                                |
| OS-EXT-STS:vm_state                  | active                                                              |
| OS-SRV-USG:launched_at               | 2015-09-23T13:57:09.000000                                          |
| OS-SRV-USG:terminated_at             | None                                                                |
| accessIPv4                           |                                                                     |
| accessIPv6                           |                                                                     |
| addresses                            | sab-sadmin-dev-net=10.0.0.144, 172.20.142.21                       |
| config_drive                         |                                                                     |
| created                              | 2015-09-23T13:57:04Z                                                |
| flavor                               | docker.m1.small (c7d7df07-ff66-46e5-8cd9-a98136cfaa2a)              |
| hostId                               | 67cbd0497e9600dad5b8b630297e4bb8e2e805a87170a730cf7bf98f            |
| id                                   | b1a5d738-a7c4-4009-a712-7514517b931d                                |
| image                                | rastasheep/ubuntu-sshd:14.04 (ba0ba9e0-630f-42b6-8f73-fca19f3c7ee6) |
| key_name                             | new-admin                                                           |
| name                                 | dockertest                                                          |
| os-extended-volumes:volumes_attached | []                                                                  |
| progress                             | 0                                                                   |
| project_id                           | 23a18e23522e4f55a08828da48e2b547                                    |
| properties                           |                                                                     |
| security_groups                      | [{u'name': u'sab-sadmin-dev-icmp-ssh'}, {u'name': u'default'}]     |
| status                               | ACTIVE                                                              |
| updated                              | 2015-09-23T13:57:09Z                                                |
| user_id                              | b0ebea8f7597b11f32b9cd71748c73b602d8ac7722b1e5bfe1fe42b32b7ead20    |
+--------------------------------------+---------------------------------------------------------------------+

Yup, we are using the docker flavor defined above, and the Ubuntu SSHD image we imported to Glance.

Here’s what it looks like on the Nova Compute hypervisor:

[root@lposhostx076 nova(lvosksclu120-rc-admin)]# docker ps
CONTAINER ID        IMAGE                          COMMAND               CREATED             STATUS              PORTS               NAMES
6fe07a32a58f        rastasheep/ubuntu-sshd:14.04   "/usr/sbin/sshd -D"   About an hour ago   Up About an hour                        nova-b1a5d738-a7c4-4009-a712-7514517b931d   
[root@lposhostx076 nova(lvosksclu120-rc-admin)]# docker inspect 6fe07a32a58f
[
{
    "Id": "6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6",
    "Created": "2015-09-23T13:57:06.522408383Z",
    "Path": "/usr/sbin/sshd",
    "Args": [
        "-D"
    ],
    "State": {
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 38562,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2015-09-23T13:57:06.861698662Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "2069c778a8155d55245e1b37e382841b5e78d296d2c5a2d5b4f9714e2859417b",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "NetworkID": "",
        "PortMapping": null,
        "Ports": null,
        "SandboxKey": "",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "ResolvConfPath": "",
    "HostnamePath": "/home/rootdir/var/lib/docker/containers/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6/hostname",
    "HostsPath": "",
    "LogPath": "/home/rootdir/var/lib/docker/containers/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6-json.log",
    "Name": "/nova-b1a5d738-a7c4-4009-a712-7514517b931d",
    "RestartCount": 0,
    "Driver": "devicemapper",
    "ExecDriver": "native-0.2",
    "MountLabel": "system_u:object_r:svirt_sandbox_file_t:s0:c138,c900",
    "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c138,c900",
    "Volumes": {},
    "VolumesRW": {},
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LxcConf": null,
        "Memory": 0,
        "MemorySwap": 0,
        "CpuShares": 0,
        "CpuPeriod": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "CpuQuota": 0,
        "BlkioWeight": 0,
        "OomKillDisable": false,
        "Privileged": false,
        "PortBindings": null,
        "Links": null,
        "PublishAllPorts": false,
        "Dns": [
            "192.168.1.2"
        ],
        "DnsSearch": null,
        "ExtraHosts": null,
        "VolumesFrom": null,
        "Devices": null,
        "NetworkMode": "bridge",
        "IpcMode": "",
        "PidMode": "",
        "UTSMode": "",
        "CapAdd": null,
        "CapDrop": null,
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "SecurityOpt": null,
        "ReadonlyRootfs": false,
        "Ulimits": null,
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "CgroupParent": ""
    },
    "Config": {
        "Hostname": "instance-00000129",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": true,
        "AttachStderr": true,
        "PortSpecs": null,
        "ExposedPorts": {
            "22/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/usr/sbin/sshd",
            "-D"
        ],
        "Image": "rastasheep/ubuntu-sshd:14.04",
        "Volumes": null,
        "VolumeDriver": "",
        "WorkingDir": "",
        "Entrypoint": null,
        "NetworkDisabled": true,
        "MacAddress": "",
        "OnBuild": null,
        "Labels": {}
    }
}
]

That is all. Enjoy!

Team-oriented systems mentor with deep knowledge of numerous software methodologies, technologies, languages, and operating systems. Excited about turning emerging technology into working production-ready systems. Focused on moving software teams to a higher level of world-class application development. Specialties:Software analysis and development...Product management through the entire lifecycle...Discrete product integration specialist!

Tagged with: , ,

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Human Verification: In order to verify that you are a human and not a spam bot, please enter the answer into the following box below based on the instructions contained in the graphic.