CoreOS Cluster on OpenStack Pike

Run a simple CoreOS cluster natively in OpenStack. Plus, learn a tremendous amount about the OpenStack CLI tools as opposed to Heat:https://docs.openstack.org/heat/latest/

References

• https://coreos.com/os/docs/latest/booting-on-openstack.html – Latest official documentation
• http://lollyrock.com/articles/coreos-openstack/ – Primary resource
• https://docs.openstack.org/magnum/latest/user/ – Magnum-specific settings (so this page can be used for OpenStack Magnum integration)
• https://cloudssky.com/en/blog/CoreOS-Cluster-on-OpenStack/ – Useful info including ports to open for CoreOS

Creation

Steps:

1. Set your binary names:
   OPENSTACK_BINARY=openstack
   GLANCE_BINARY=glance

2. Download / install to Glance:
   cd /tmp
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2
bunzip2 coreos_production_openstack_image.img.bz2

   \
   To upload to OpenStack:
   $GLANCE_BINARY image-create --name coreos_production_openstack_image.img \
--file ./coreos_production_openstack_image.img \
--architecture x86_64 --os-distro coreos \
--disk-format qcow2 --container-format bare \
--protected True --visibility public \
--progress

3. For this basic demo, we will leverage the public CoreOS Container Linux cluster discovery endpoint. This is simply used to bootstrap the CoreOS cluster.
   (Adding additional nodes to the cluster will not use the public etcd
   cluster discovery endpoint.) Determine the number of nodes in the
   cluster:
   read -p 'Enter # of nodes for the CoreOS cluster: ' OS_COREOS_CLUSTER_SIZE

   \
   Retrieve the “discovery token” from the CoreOS public endpoint (for
   cluster bootstrapping):
   OS_COREOS_ETCD_DISCOVERY_TOKEN=$(curl -w "\n" "https://discovery.etcd.io/new?size=$OS_COREOS_CLUSTER_SIZE")

4. We need one or more SSH public keys. Why? Because we will not be
   able to use Nova’s built-in SSH key management as we are manually
   specifying the cloud-config configuration file to use.
   Here are a couple of commands you can run to discover the first SSH
   key registered in your OpenStack project and information about that
   key:
   MY_FIRST_KEYPAIR=$($OPENSTACK_BINARY keypair list -f value -c Name | head -n 1 | sed 's/\r$//')

   \
   Next, show information about the keypair so you can get the public
   key information used to create the CoreOS cluster:
   $OPENSTACK_BINARY keypair show $MY_FIRST_KEYPAIR

   \
   Finally, load the public key into a variable. For example:
   OS_COREOS_PUBKEY=$(cat ~/.ssh/MY_FIRST_KEYPAIR.pub)

   \
   The above command will only work if your SSH public key follows that
   naming convention; YMMV.

5. Create the cloud-config.yaml file which we will run
   through the CoreOS Config
   Transpiler
   to pass to OpenStack Nova for the etcd cluster creation. Assumes all
   the above has been completed first.

   • First, we generate the “Container Linux Config” (this is a
     human-readable file):

     ```
     cat > /tmp/cloud-config.yaml << EOF
     # This config is meant to be consumed by the config transpiler, which will
     # generate the corresponding Ignition config. Do not pass this config directly
     # to instances of Container Linux.
     
     etcd:
       # All options get passed as command line flags to etcd.
       # Any information inside curly braces comes from the machine at boot time.
     
       # multi_region and multi_cloud deployments need to use {PUBLIC_IPV4}
       advertise_client_urls:       "http://{PRIVATE_IPV4}:2379"
       initial_advertise_peer_urls: "http://{PRIVATE_IPV4}:2380"
       # listen on both the official ports and the legacy ports
       # legacy ports can be omitted if your application doesn't depend on them
       listen_client_urls:          "http://0.0.0.0:2379"
       listen_peer_urls:            "http://{PRIVATE_IPV4}:2380"
       # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
       # specify the initial size of your cluster with ?size=X
       discovery:                   "$OS_COREOS_ETCD_DISCOVERY_TOKEN"
     passwd:
       users:
         - name: core
           ssh_authorized_keys:
             - $OS_COREOS_PUBKEY
     EOF
     ```
     </code></pre></li>
     <li>Next, generate the "CoreOS Ignition" file:
     
     <pre><code>```
     ct -in-file /tmp/cloud-config.yaml -out-file /tmp/cloud-config.ign -pretty -platform openstack-metadata
     ```
     

6. Security group setup on OpenStack – there are a few ports which
   should be opened. While these could be added to the
   default security group, we don’t like that do we??
   Here’s some code that will check for / create the security group:
   • Get the OpenStack project ID based on your currently defined
     project name:
     MY_PROJECT_ID=$($OPENSTACK_BINARY project show $OS_PROJECT_NAME -f json -c id | sed 's/\r$//' | jq -r '.id')

   • Get the default security group ID for the current
     project:
     MY_SECURITY_GROUP_DEFAULT_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"default\")|.ID")

   • Get the std-access security group (note that you
     may use a different security group for SSH access):
     MY_SECURITY_GROUP_STD_ACCESS_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"std-access\")|.ID")

   • Get the list of security groups for this project:
     MY_SECURITY_GROUP_IDS=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|.ID")

   • Do we have any that support the ports we want?
     MY_SECURITY_GROUP_ID=''
my_tmp="/tmp/$$.tmp"
for i in $MY_SECURITY_GROUP_IDS ; do
echo -n "Check security group '$i': "
$OPENSTACK_BINARY security group rule list $i -f value -c 'Port Range' > "$my_tmp" 2>/dev/null
grep -e '4001' "$my_tmp" 2>&1 >/dev/null && grep -e '7001' "$my_tmp" 2>&1 >/dev/null && MY_SECURITY_GROUP_ID="$i"
rm -f "$my_tmp"
[ x"$MY_SECURITY_GROUP_ID" != x ] && echo 'Found!' && break
echo ''
done
if [ x"$MY_SECURITY_GROUP_ID" = x ] ; then
echo -n ' Create new security group: '
MY_SECURITY_GROUP_ID=$($OPENSTACK_BINARY security group create coreos --description coreos --project $MY_PROJECT_ID -f value -c id | sed 's/\r$//')
echo "$MY_SECURITY_GROUP_ID"
echo -n ' Create port rules...'
$OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 4001 $MY_SECURITY_GROUP_ID
$OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 7001 $MY_SECURITY_GROUP_ID
fi

7. Let’s get the remaining information:

   • Let’s make sure we have the prefix for the cluster:
     read -p 'CoreOS cluster prefix (short, please!): ' MY_CLUSTER_PREFIX

   • Get the image (from the uploaded CoreOS image above):
     MY_IMAGE_ID=$($OPENSTACK_BINARY image list --public --property os_distro=coreos --status active --limit 1 -f json -c ID | sed 's/\r$//' | jq -r '.[]|.ID')

   • Get the flavor to use:
     MY_FLAVOR=''
while [ x"$MY_FLAVOR" = x ] ; do
read -p 'Type the flavor you want (SHOW to show them): ' MY_FLAVOR
[ x"$MY_FLAVOR" != xSHOW ] && break
echo 'OpenStack flavors:'
$OPENSTACK_BINARY flavor list
echo ''
MY_FLAVOR=''
done

   • Get the network to use (example below uses first internal
     network for your current project):
     MY_NETWORK_ID=$($OPENSTACK_BINARY network list --project $MY_PROJECT_ID --enable --internal -f value -c ID | sed 's/\r$//' | head -n 1)

   • And the subnet:
     MY_SUBNET_ID=$($OPENSTACK_BINARY subnet list --network $MY_NETWORK_ID -f value -c ID | sed 's/\r$//' | head -n 1)

8. Now we should have enough information to create the cluster:

   • First – let’s display all the environment variables we will be
     using:
     my_tmp="/tmp/$$.tmp"
set | grep -e '^\(MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_\)\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)=' > $my_tmp
cat "$my_tmp"
echo ''
found_items=$(wc -l $my_tmp | awk '{print $1}')
[ $found_items -ne 8 ] && echo 'You are missing some items - please double-check...' || echo 'You have all the items defined'
rm -f "$my_tmp"

     \
     Here is some typical output:
     MY_CLUSTER_PREFIX=coreos-k8s
MY_FIRST_KEYPAIR=demo-abruce-new-admin
MY_FLAVOR=m1.medium
MY_IMAGE_ID=db581d51-514f-41d6-9517-2af1e96202ff
MY_NETWORK_ID=2c737b0b-fa55-4208-a779-37cf9dfa18d1
MY_SECURITY_GROUP_DEFAULT_ID=0a30e90a-c5a9-4e63-906e-4ffa835984d4
MY_SECURITY_GROUP_ID=d1cf8fef-b57a-4ccc-9720-4012a5dcff27
OS_COREOS_CLUSTER_SIZE=3

   • All look good? Then let’s create the cluster:
     $OPENSTACK_BINARY server create \
--image $MY_IMAGE_ID --flavor $MY_FLAVOR \
--security-group $MY_SECURITY_GROUP_DEFAULT_ID --security-group $MY_SECURITY_GROUP_STD_ACCESS_ID --security-group $MY_SECURITY_GROUP_ID \
--key-name $MY_FIRST_KEYPAIR --user-data /tmp/cloud-config.ign --network $MY_NETWORK_ID \
--min $OS_COREOS_CLUSTER_SIZE --max $OS_COREOS_CLUSTER_SIZE \
--wait \
$MY_CLUSTER_PREFIX

   • Assign public IP (s) to the instances created; you can find them
     by running:
     $OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX"

     \
     Sample output looks like:
     +--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
| 37ecb1b9-978e-41fa-9e73-ed6a62f634af | coreos-k8s-3 | ACTIVE | abruce-workspace-net=10.0.0.8 | coreos_production_openstack_image.img | m1.medium |
| 2663b584-7b98-4fbb-bdfc-ceeee06cf8b1 | coreos-k8s-2 | ACTIVE | abruce-workspace-net=10.0.0.4 | coreos_production_openstack_image.img | m1.medium |
| 06c296e9-a5a9-48ac-9f7f-aadfbb343e74 | coreos-k8s-1 | ACTIVE | abruce-workspace-net=10.0.0.3 | coreos_production_openstack_image.img | m1.medium |
+--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+

9. Here’s an example of creating / applying a floating IP to the each
   server found:

   • Get the list of servers created:
     MY_EXT_NET_NAME=ext-net
MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//')

   • Process each one, adding a floating IP if necessary:
     for i in $MY_SERVER_IDS ; do
echo -n "Checking server '$i': "
THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1)
THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1)
[ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && continue
echo 'Create...'
$OPENSTACK_BINARY floating ip create --port $THE_SERVER_PORT_ID $MY_EXT_NET_NAME
done

Cleanup

Steps:

1. Make sure you have all your environment variables defined from
   above:
   set | grep -e '^\(MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_\)\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)='

2. Load in your servers:
   MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//')

3. Cleanup floating IPs:
   for i in $MY_SERVER_IDS ; do
echo -n "Checking server '$i': "
THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1)
THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1)
[ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && $OPENSTACK_BINARY floating ip delete $THE_SERVER_FLOATING_IP_ID && continue
echo '[n/a]'
done

4. Cleanup servers:
   for i in $MY_SERVER_IDS ; do
echo -n "Delete server '$i': "
$OPENSTACK_BINARY server delete --wait $i
done