Found a number of references but here’s the recipe from zero to hero.<\/p>\n
yum install -y docker<\/code><\/pre>\n<\/li>\n- Optional: On my Nova Compute host, the root partition (where
\/var\/lib<\/code> lives) is only 50GB. So I move Docker to a larger partition and soft-link:\n\r\nmkdir -p \/home\/rootdir\/var\/lib\r\nmv \/var\/lib\/docker \/home\/rootdir\/var\/lib\/\r\nln -fs \/home\/rootdir\/var\/lib\/docker \/var\/lib\/docker<\/code><\/pre>\n<\/li>\n- Setup Docker to support 4TB filesystem (default is 100GB limit). Note we create sparse file; unsure if this is required but remembering that OpenStack will be looking at available disk space before scheduling host. Note that this destroys<\/strong> any Docker images \/ running containers. That’s why we do it now, before we’ve run anything.\n
rm -fR \/var\/lib\/docker\/*\r\nmkdir -p \/var\/lib\/docker\/devicemapper\/devicemapper\r\ndd if=\/dev\/zero of=\/var\/lib\/docker\/devicemapper\/devicemapper\/data bs=1G count=0 seek=4096<\/code><\/pre>\n<\/li>\n- There are terrible things discussed about the default
devicemapper<\/code> driver with Docker. Many suggest to use overlayfs<\/code>. Too bad it doesn’t work with XFS file systems on CentOS 7…and that XFS is the file system used by default when creating new partitions during Anaconda install. So…if you want to use overlayfs, follow these steps. I have not.\n# all this commented out because CentOS 7 xfs doesn't work with overlayfs (which is \"tech preview\" in CentOS 7):\r\n# sed -i -e \"s#^\\(OPTIONS='--selinux-enabled\\)\\('\\)#\\1=false\\2#\" \/etc\/sysconfig\/docker\r\n# sed -i -e 's#^\\(DOCKER_STORAGE_OPTIONS\\)\\([^=]*\\).*#\\1 = -s overlay#' \/etc\/sysconfig\/docker-storage<\/code><\/pre>\n<\/li>\n- Now we can start Docker:\n
systemctl start docker\r\nsystemctl enable docker<\/code><\/pre>\n<\/li>\n- Verify we have 4TB disk:\n
[root@lposhostx076 proj]# docker info<\/strong>\r\nContainers: 0\r\nImages: 0\r\nStorage Driver: devicemapper\r\n Pool Name: docker-253:2-26843546688-pool\r\n Pool Blocksize: 65.54 kB\r\n Backing Filesystem: xfs\r\n Data file: \/dev\/loop0\r\n Metadata file: \/dev\/loop1\r\n Data Space Used: 307.2 MB\r\n Data Space Total: 4.398 TB\r\n Data Space Available: 4.398 TB\r\n Metadata Space Used: 17.36 MB\r\n Metadata Space Total: 2.147 GB\r\n Metadata Space Available: 2.13 GB\r\n Udev Sync Supported: true\r\n Deferred Removal Enabled: false\r\n Data loop file: \/home\/rootdir\/var\/lib\/docker\/devicemapper\/devicemapper\/data\r\n Metadata loop file: \/home\/rootdir\/var\/lib\/docker\/devicemapper\/devicemapper\/metadata\r\n Library Version: 1.02.93-RHEL7 (2015-01-28)\r\nExecution Driver: native-0.2\r\nLogging Driver: json-file\r\nKernel Version: 3.10.0-229.14.1.el7.x86_64\r\nOperating System: CentOS Linux 7 (Core)\r\nCPUs: 32\r\nTotal Memory: 251.9 GiB\r\nName: lposhostx076.hlsdev.local<\/em><\/code><\/pre>\n<\/li>\n<\/ol>\nNova Compute Integration<\/h3>\n\n- Stop Docker so we can continue with Nova Compute:\n
systemctl stop docker<\/code><\/pre>\n<\/li>\n- Add Nova Compute user to
dockerroot<\/code> group for Docker unixsocket access. Note that this is deprecated by the Atomic Project article above, but that OpenStack Nova Compute does<\/em> actually need to run Docker \ud83d\ude42\nusermod -a -G dockerroot nova<\/code><\/pre>\n<\/li>\n- Install driver on Nova Compute, then restart Docker. Note these instructions are specific to OpenStack Kilo and will certainly change for other releases:\n
yum install -y python-pip\r\nmkdir -p \/root\/proj\r\ncd \/root\/proj\r\ngit clone https:\/\/github.com\/stackforge\/nova-docker.git\r\ngit checkout -b kilo origin\/stable\/kilo\r\ngit branch -v -a\r\npython setup.py install\r\nsystemctl start docker<\/code><\/pre>\n<\/li>\n- Permit Nova to access docker via socket. Unfortunately, this must be performed each time Docker restarted<\/strong> (I have not yet automated it):\n
chgrp dockerroot \/var\/run\/docker.sock<\/code><\/pre>\n<\/li>\n- Modify
\/etc\/nova\/nova.conf<\/code> to tell Nova Compute to use the Docker hypervisor driver:\n[DEFAULT]\r\ncompute_driver=novadocker.virt.docker.DockerDriver<\/code><\/pre>\n<\/li>\n- Setup rootwrap so that Nova Compute user can execute Docker commands:\n
[ ! -d \/etc\/nova\/rootwrap.d ] && mkdir -p \/etc\/nova\/rootwrap.d\r\ncat > \/etc\/nova\/rootwrap.d\/docker.filters <<EOF\r\n# nova-rootwrap command filters for setting up network in the docker driver\r\n# This file should be owned by (and only-writeable by) the root user\r\n\r\n[Filters] \r\n# nova\/virt\/docker\/driver.py: 'ln', '-sf', '\/var\/run\/netns\/.*'\r\nln: CommandFilter, \/bin\/ln, root\r\nEOF<\/code><\/pre>\n<\/li>\n- Patch Kilo docker driver; by default it only permits Linux Containers (lxc). Added “docker” to the list below for x86 and x64:\n
# \/usr\/lib\/python2.7\/site-packages\/novadocker\/virt\/docker\/driver.py\r\n...\r\n 'supported_instances': jsonutils.dumps([\r\n ('i686', 'docker', 'lxc'), \r\n ('x86_64', 'docker', 'lxc')\r\n ])\r\n...<\/code><\/pre>\n<\/li>\n- Restart Compute:\n
systemctl restart openstack-nova-compute<\/code><\/pre>\n<\/li>\n<\/ol>\nGlance Integration<\/h3>\n
On your Glance host, run the following:<\/p>\n
\n- Modify
\/etc\/glance\/glance-api.conf<\/code>:\n[DEFAULT]\r\ncontainer_formats=ami,ari,aki,bare,ovf,ova,docker<\/code><\/pre>\n<\/li>\n- Restart Glance:\n
systemctl restart openstack-glance-api<\/code><\/pre>\n<\/li>\n<\/ol>\nNova Scheduler Integration<\/h3>\n
On your Nova Controller (running openstack-nova-scheduler<\/code>), run the following steps. These modify the Nova Scheduler to place Docker instances only on Docker-enabled Nova Compute hosts. The steps also assume you have sourced in the appropriate OpenStack credentials file:<\/p>\n\n- Assuming you are using default KVM (also displayed as QEMU although – as you well know – they are not the same thing), modify all existing flavors to have a new attribute indicating they are KVM:\n
for i in $(openstack flavor list | tail -n +4 | head -n -1 | awk -F'|' '{print $3}' | sed -e 's# ##g'); do\r\n if echo $i | grep --quiet -v '^docker\\.'; then\r\n openstack flavor set --property aggregate_instance_extra_specs:virt_type=kvm $i\r\n fi\r\ndone<\/code><\/pre>\n<\/li>\nCreate a new flavor, and set attribute for Docker:<\/p>\n
- \n
if ! openstack flavor show docker.m1.small >\/dev\/null 2>&1; then\r\n openstack flavor create --ram 2048 --disk 20 --vcpus 1 docker.m1.small\r\nfi\r\nopenstack flavor set --property aggregate_instance_extra_specs:virt_type=docker docker.m1.small<\/code><\/pre>\n<\/li>\n- We will use OpenStack Host Aggregates to provide our filtering. We create two aggregates; one for Docker and one for KVM. If you have additional hypervisors, modify accordingly:\n
if ! openstack aggregate show docker-host >\/dev\/null 2>&1; then\r\n openstack aggregate create --zone nova docker-host\r\nfi\r\nopenstack aggregate set --property virt_type=docker docker-host\r\nif ! openstack aggregate show kvm-host >\/dev\/null 2>&1; then\r\n openstack aggregate create --zone nova kvm-host\r\nfi\r\nopenstack aggregate set --property virt_type=kvm kvm-host<\/code><\/pre>\n<\/li>\n- Add hosts to each aggregate:\n
for i in $(openstack hypervisor list | tail -n +4 | head -n -1 | awk -F'|' '{print $2}' | sed -e 's# ##g'); do\r\n l_aggregate_name_add='kvm-host'\r\n l_aggregate_name_remove='docker-host'\r\n l_hypervisor_hostname=$(openstack hypervisor show $i | grep -e '^| \\+hypervisor_hostname \\+|' | awk -F'|' '{print $3}' | sed -e 's#^ \\+##' -e 's# \\+$##')\r\n l_hypervisor_type=$(openstack hypervisor show $i | grep -e '^| \\+hypervisor_type \\+|' | awk -F'|' '{print $3}' | sed -e 's#^ \\+##' -e 's# \\+$##')\r\n if [ \"x$l_hypervisor_type\" = \"xdocker\" ]; then\r\n l_aggregate_name_add='docker-host'\r\n l_aggregate_name_remove='kvm-host'\r\n fi\r\n if openstack aggregate show \"$l_aggregate_name_remove\" | grep -e '^| \\+hosts \\+|' | grep --quiet \"'$l_hypervisor_hostname'\"; then\r\n echo openstack aggregate remove host \"$l_aggregate_name_remove\" $l_hypervisor_hostname\r\n fi\r\n if ! openstack aggregate show \"$l_aggregate_name_add\" | grep -e '^| \\+hosts \\+|' | grep --quiet \"'$l_hypervisor_hostname'\"; then\r\n echo openstack aggregate add host \"$l_aggregate_name_add\" $l_hypervisor_hostname\r\n fi\r\ndone<\/code><\/pre>\n<\/li>\n- Modify Nova Scheduler via
\/etc\/nova\/nova.conf<\/code> to use the AggregateInstanceExtraSpecsFilter so that the virt_type<\/code> property we set above will be used to filter out hosts that have incompatible hypervisors:\n[DEFAULT]\r\nscheduler_default_filters=AggregateInstanceExtraSpecsFilter,RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter<\/code><\/pre>\n<\/li>\n- Restart Nova Scheduler:\n
systemctl restart openstack-nova-scheduler<\/code><\/pre>\n<\/li>\n<\/ol>\nProof of the Pudding is in the Tasting<\/h3>\n
We first integrated a simple Ubuntu SSHD Docker image:<\/p>\n
docker pull rastasheep\/ubuntu-sshd:14.04\r\ndocker save rastasheep\/ubuntu-sshd:14.04 > rastasheep_ubuntu-sshd:14.04\r\nglance image-create --architecture x86_64 --name 'rastasheep\/ubuntu-sshd:14.04' \\\r\n --visibility public --os-version 14.04 --disk-format raw --os-distro ubuntu \\\r\n --container-format docker --file .\/rastasheep_ubuntu-sshd\\:14.04 \\\r\n --progress<\/code><\/pre>\nVerify it’s on file:<\/p>\n
[root@lposhostx076 nova(lvosksclu120-rc-admin)]# glance image-show $(glance image-list | grep rastasheep | awk -F'|' '{print $2}')<\/strong>\r\n+------------------+--------------------------------------+\r\n| Property | Value |\r\n+------------------+--------------------------------------+\r\n| architecture | x86_64 |\r\n| checksum | d78a9ff8d12f9e91cd5387e40d36fa9f |\r\n| container_format | docker |\r\n| created_at | 2015-09-21T21:33:58Z |\r\n| disk_format | raw |\r\n| id | ba0ba9e0-630f-42b6-8f73-fca19f3c7ee6 |\r\n| min_disk | 0 |\r\n| min_ram | 0 |\r\n| name | rastasheep\/ubuntu-sshd:14.04 |\r\n| os_distro | ubuntu |\r\n| os_version | 14.04 |\r\n| owner | f1397ff9930c4c20bc0712eba2555e43 |\r\n| protected | False |\r\n| size | 264800256 |\r\n| status | active |\r\n| tags | [] |\r\n| updated_at | 2015-09-21T21:34:05Z |\r\n| virtual_size | None |\r\n| visibility | public |\r\n+------------------+--------------------------------------+<\/code><\/pre>\nNow boot an instance from the image. I won’t show the nova boot<\/code> commands because it uses lots of ID values; just use Horizon for this. When you are done, you can see the instance (I’ve named it dockertest<\/code> and I used a project named sab-sadmin-dev<\/code>. This also demonstrates a funny OpenStack one-liner:<\/p>\n[root@lposhostx076 nova(lvosksclu120-rc-admin)]# openstack server show $(openstack --os-project-id=\"$(openstack project list | grep sab-sadmin-dev | awk -F'|' '{print $2}' | sed -e 's# ##g')\" server list | grep dockertest | awk -F'|' '{print $2}')\r\n+--------------------------------------+---------------------------------------------------------------------+\r\n| Field | Value |\r\n+--------------------------------------+---------------------------------------------------------------------+\r\n| OS-DCF:diskConfig | AUTO |\r\n| OS-EXT-AZ:availability_zone | nova |\r\n| OS-EXT-SRV-ATTR:host | lposhostx076.hlsdev.local |\r\n| OS-EXT-SRV-ATTR:hypervisor_hostname | lposhostx076.hlsdev.local |\r\n| OS-EXT-SRV-ATTR:instance_name | instance-00000129 |\r\n| OS-EXT-STS:power_state | 1 |\r\n| OS-EXT-STS:task_state | None |\r\n| OS-EXT-STS:vm_state | active |\r\n| OS-SRV-USG:launched_at | 2015-09-23T13:57:09.000000 |\r\n| OS-SRV-USG:terminated_at | None |\r\n| accessIPv4 | |\r\n| accessIPv6 | |\r\n| addresses | sab-sadmin-dev-net=10.0.0.144, 172.20.142.21 |\r\n| config_drive | |\r\n| created | 2015-09-23T13:57:04Z |\r\n| flavor | docker.m1.small (c7d7df07-ff66-46e5-8cd9-a98136cfaa2a) |\r\n| hostId | 67cbd0497e9600dad5b8b630297e4bb8e2e805a87170a730cf7bf98f |\r\n| id | b1a5d738-a7c4-4009-a712-7514517b931d |\r\n| image | rastasheep\/ubuntu-sshd:14.04 (ba0ba9e0-630f-42b6-8f73-fca19f3c7ee6) |\r\n| key_name | new-admin |\r\n| name | dockertest |\r\n| os-extended-volumes:volumes_attached | [] |\r\n| progress | 0 |\r\n| project_id | 23a18e23522e4f55a08828da48e2b547 |\r\n| properties | |\r\n| security_groups | [{u'name': u'sab-sadmin-dev-icmp-ssh'}, {u'name': u'default'}] |\r\n| status | ACTIVE |\r\n| updated | 2015-09-23T13:57:09Z |\r\n| user_id | b0ebea8f7597b11f32b9cd71748c73b602d8ac7722b1e5bfe1fe42b32b7ead20 |\r\n+--------------------------------------+---------------------------------------------------------------------+<\/code><\/pre>\nYup, we are using the docker flavor defined above, and the Ubuntu SSHD image we imported to Glance.<\/p>\n
Here’s what it looks like on the Nova Compute hypervisor:<\/p>\n
[root@lposhostx076 nova(lvosksclu120-rc-admin)]# docker ps\r\nCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES\r\n6fe07a32a58f rastasheep\/ubuntu-sshd:14.04 \"\/usr\/sbin\/sshd -D\" About an hour ago Up About an hour nova-b1a5d738-a7c4-4009-a712-7514517b931d \r\n[root@lposhostx076 nova(lvosksclu120-rc-admin)]# docker inspect 6fe07a32a58f\r\n[\r\n{\r\n \"Id\": \"6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6\",\r\n \"Created\": \"2015-09-23T13:57:06.522408383Z\",\r\n \"Path\": \"\/usr\/sbin\/sshd\",\r\n \"Args\": [\r\n \"-D\"\r\n ],\r\n \"State\": {\r\n \"Running\": true,\r\n \"Paused\": false,\r\n \"Restarting\": false,\r\n \"OOMKilled\": false,\r\n \"Dead\": false,\r\n \"Pid\": 38562,\r\n \"ExitCode\": 0,\r\n \"Error\": \"\",\r\n \"StartedAt\": \"2015-09-23T13:57:06.861698662Z\",\r\n \"FinishedAt\": \"0001-01-01T00:00:00Z\"\r\n },\r\n \"Image\": \"2069c778a8155d55245e1b37e382841b5e78d296d2c5a2d5b4f9714e2859417b\",\r\n \"NetworkSettings\": {\r\n \"Bridge\": \"\",\r\n \"EndpointID\": \"\",\r\n \"Gateway\": \"\",\r\n \"GlobalIPv6Address\": \"\",\r\n \"GlobalIPv6PrefixLen\": 0,\r\n \"HairpinMode\": false,\r\n \"IPAddress\": \"\",\r\n \"IPPrefixLen\": 0,\r\n \"IPv6Gateway\": \"\",\r\n \"LinkLocalIPv6Address\": \"\",\r\n \"LinkLocalIPv6PrefixLen\": 0,\r\n \"MacAddress\": \"\",\r\n \"NetworkID\": \"\",\r\n \"PortMapping\": null,\r\n \"Ports\": null,\r\n \"SandboxKey\": \"\",\r\n \"SecondaryIPAddresses\": null,\r\n \"SecondaryIPv6Addresses\": null\r\n },\r\n \"ResolvConfPath\": \"\",\r\n \"HostnamePath\": \"\/home\/rootdir\/var\/lib\/docker\/containers\/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6\/hostname\",\r\n \"HostsPath\": \"\",\r\n \"LogPath\": \"\/home\/rootdir\/var\/lib\/docker\/containers\/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6\/6fe07a32a58f369f44225b119e29a1f78d9885e3f26a465d45e7b84e894c55f6-json.log\",\r\n \"Name\": \"\/nova-b1a5d738-a7c4-4009-a712-7514517b931d\",\r\n \"RestartCount\": 0,\r\n \"Driver\": \"devicemapper\",\r\n \"ExecDriver\": \"native-0.2\",\r\n \"MountLabel\": \"system_u:object_r:svirt_sandbox_file_t:s0:c138,c900\",\r\n \"ProcessLabel\": \"system_u:system_r:svirt_lxc_net_t:s0:c138,c900\",\r\n \"Volumes\": {},\r\n \"VolumesRW\": {},\r\n \"AppArmorProfile\": \"\",\r\n \"ExecIDs\": null,\r\n \"HostConfig\": {\r\n \"Binds\": null,\r\n \"ContainerIDFile\": \"\",\r\n \"LxcConf\": null,\r\n \"Memory\": 0,\r\n \"MemorySwap\": 0,\r\n \"CpuShares\": 0,\r\n \"CpuPeriod\": 0,\r\n \"CpusetCpus\": \"\",\r\n \"CpusetMems\": \"\",\r\n \"CpuQuota\": 0,\r\n \"BlkioWeight\": 0,\r\n \"OomKillDisable\": false,\r\n \"Privileged\": false,\r\n \"PortBindings\": null,\r\n \"Links\": null,\r\n \"PublishAllPorts\": false,\r\n \"Dns\": [\r\n \"192.168.1.2\"\r\n ],\r\n \"DnsSearch\": null,\r\n \"ExtraHosts\": null,\r\n \"VolumesFrom\": null,\r\n \"Devices\": null,\r\n \"NetworkMode\": \"bridge\",\r\n \"IpcMode\": \"\",\r\n \"PidMode\": \"\",\r\n \"UTSMode\": \"\",\r\n \"CapAdd\": null,\r\n \"CapDrop\": null,\r\n \"RestartPolicy\": {\r\n \"Name\": \"\",\r\n \"MaximumRetryCount\": 0\r\n },\r\n \"SecurityOpt\": null,\r\n \"ReadonlyRootfs\": false,\r\n \"Ulimits\": null,\r\n \"LogConfig\": {\r\n \"Type\": \"json-file\",\r\n \"Config\": {}\r\n },\r\n \"CgroupParent\": \"\"\r\n },\r\n \"Config\": {\r\n \"Hostname\": \"instance-00000129\",\r\n \"Domainname\": \"\",\r\n \"User\": \"\",\r\n \"AttachStdin\": false,\r\n \"AttachStdout\": true,\r\n \"AttachStderr\": true,\r\n \"PortSpecs\": null,\r\n \"ExposedPorts\": {\r\n \"22\/tcp\": {}\r\n },\r\n \"Tty\": false,\r\n \"OpenStdin\": false,\r\n \"StdinOnce\": false,\r\n \"Env\": [\r\n \"PATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin\"\r\n ],\r\n \"Cmd\": [\r\n \"\/usr\/sbin\/sshd\",\r\n \"-D\"\r\n ],\r\n \"Image\": \"rastasheep\/ubuntu-sshd:14.04\",\r\n \"Volumes\": null,\r\n \"VolumeDriver\": \"\",\r\n \"WorkingDir\": \"\",\r\n \"Entrypoint\": null,\r\n \"NetworkDisabled\": true,\r\n \"MacAddress\": \"\",\r\n \"OnBuild\": null,\r\n \"Labels\": {}\r\n }\r\n}\r\n]<\/code><\/pre>\nThat is all. Enjoy!<\/p>\n","protected":false},"excerpt":{"rendered":"
Found a number of references but here’s the recipe from zero to hero. References Useful writeup on CentOS 7: https:\/\/ask.openstack.org\/en\/question\/67682\/docker-setup-on-kilo\/ Discussion of Docker storage: http:\/\/www.projectatomic.io\/blog\/2015\/06\/notes-on-fedora-centos-and-docker-storage-drivers\/ Writeup on resize Docker storage: https:\/\/jpetazzo.github.io\/2014\/01\/29\/docker-device-mapper-resize\/ CentOS 7 bug with overlayfs on XFS filesystem (obviates …<\/p>\n