CoreOS Cluster on OpenStack Pike

Run a simple CoreOS cluster natively in OpenStack. Plus, learn a tremendous amount about the OpenStack CLI tools as opposed to Heat:https://docs.openstack.org/heat/latest/

References

https://coreos.com/os/docs/latest/booting-on-openstack.html – Latest official documentation
http://lollyrock.com/articles/coreos-openstack/ – Primary resource
https://docs.openstack.org/magnum/latest/user/ – Magnum-specific settings (so this page can be used for OpenStack Magnum integration)
https://cloudssky.com/en/blog/CoreOS-Cluster-on-OpenStack/ – Useful info including ports to open for CoreOS

Creation

Steps:

Set your binary names:
OPENSTACK_BINARY=openstack
GLANCE_BINARY=glance
Download / install to Glance:
cd /tmp wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2 bunzip2 coreos_production_openstack_image.img.bz2

\
To upload to OpenStack:
$GLANCE_BINARY image-create --name coreos_production_openstack_image.img \ --file ./coreos_production_openstack_image.img \ --architecture x86_64 --os-distro coreos \ --disk-format qcow2 --container-format bare \ --protected True --visibility public \ --progress
For this basic demo, we will leverage the public CoreOS Container Linux cluster discovery endpoint. This is simply used to bootstrap the CoreOS cluster.
(Adding additional nodes to the cluster will not use the public etcd
cluster discovery endpoint.) Determine the number of nodes in the
cluster:
read -p 'Enter # of nodes for the CoreOS cluster: ' OS_COREOS_CLUSTER_SIZE

\
Retrieve the “discovery token” from the CoreOS public endpoint (for
cluster bootstrapping):
OS_COREOS_ETCD_DISCOVERY_TOKEN=$(curl -w "\n" "https://discovery.etcd.io/new?size=$OS_COREOS_CLUSTER_SIZE")
We need one or more SSH public keys. Why? Because we will not be
able to use Nova’s built-in SSH key management as we are manually
specifying the cloud-config configuration file to use.
Here are a couple of commands you can run to discover the first SSH
key registered in your OpenStack project and information about that
key:
MY_FIRST_KEYPAIR=$($OPENSTACK_BINARY keypair list -f value -c Name | head -n 1 | sed 's/\r$//')

\
Next, show information about the keypair so you can get the public
key information used to create the CoreOS cluster:
$OPENSTACK_BINARY keypair show $MY_FIRST_KEYPAIR

\
Finally, load the public key into a variable. For example:
OS_COREOS_PUBKEY=$(cat ~/.ssh/MY_FIRST_KEYPAIR.pub)

\
The above command will only work if your SSH public key follows that
naming convention; YMMV.

Create the cloud-config.yaml file which we will run
through the CoreOS Config
Transpiler
to pass to OpenStack Nova for the etcd cluster creation. Assumes all
the above has been completed first.

First, we generate the “Container Linux Config” (this is a
human-readable file):

```
cat > /tmp/cloud-config.yaml << EOF
# This config is meant to be consumed by the config transpiler, which will
# generate the corresponding Ignition config. Do not pass this config directly
# to instances of Container Linux.

etcd:
  # All options get passed as command line flags to etcd.
  # Any information inside curly braces comes from the machine at boot time.

  # multi_region and multi_cloud deployments need to use {PUBLIC_IPV4}
  advertise_client_urls:       "http://{PRIVATE_IPV4}:2379"
  initial_advertise_peer_urls: "http://{PRIVATE_IPV4}:2380"
  # listen on both the official ports and the legacy ports
  # legacy ports can be omitted if your application doesn't depend on them
  listen_client_urls:          "http://0.0.0.0:2379"
  listen_peer_urls:            "http://{PRIVATE_IPV4}:2380"
  # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
  # specify the initial size of your cluster with ?size=X
  discovery:                   "$OS_COREOS_ETCD_DISCOVERY_TOKEN"
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - $OS_COREOS_PUBKEY
EOF
```
</code></pre></li>
<li>Next, generate the "CoreOS Ignition" file:

<pre><code>```
ct -in-file /tmp/cloud-config.yaml -out-file /tmp/cloud-config.ign -pretty -platform openstack-metadata
```

Security group setup on OpenStack – there are a few ports which
should be opened. While these could be added to the
default security group, we don’t like that do we??
Here’s some code that will check for / create the security group:
- Get the OpenStack project ID based on your currently defined
  project name:
  MY_PROJECT_ID=$($OPENSTACK_BINARY project show $OS_PROJECT_NAME -f json -c id | sed 's/\r$//' | jq -r '.id')
- Get the default security group ID for the current
  project:
  MY_SECURITY_GROUP_DEFAULT_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"default\")|.ID")
- Get the std-access security group (note that you
  may use a different security group for SSH access):
  MY_SECURITY_GROUP_STD_ACCESS_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"std-access\")|.ID")
- Get the list of security groups for this project:
  MY_SECURITY_GROUP_IDS=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|.ID")
- Do we have any that support the ports we want?
  MY_SECURITY_GROUP_ID='' my_tmp="/tmp/$$.tmp" for i in $MY_SECURITY_GROUP_IDS ; do echo -n "Check security group '$i': " $OPENSTACK_BINARY security group rule list $i -f value -c 'Port Range' > "$my_tmp" 2>/dev/null grep -e '4001' "$my_tmp" 2>&1 >/dev/null && grep -e '7001' "$my_tmp" 2>&1 >/dev/null && MY_SECURITY_GROUP_ID="$i" rm -f "$my_tmp" [ x"$MY_SECURITY_GROUP_ID" != x ] && echo 'Found!' && break echo '' done if [ x"$MY_SECURITY_GROUP_ID" = x ] ; then echo -n ' Create new security group: ' MY_SECURITY_GROUP_ID=$($OPENSTACK_BINARY security group create coreos --description coreos --project $MY_PROJECT_ID -f value -c id | sed 's/\r$//') echo "$MY_SECURITY_GROUP_ID" echo -n ' Create port rules...' $OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 4001 $MY_SECURITY_GROUP_ID $OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 7001 $MY_SECURITY_GROUP_ID fi
Let’s get the remaining information:
- Let’s make sure we have the prefix for the cluster:
  read -p 'CoreOS cluster prefix (short, please!): ' MY_CLUSTER_PREFIX
- Get the image (from the uploaded CoreOS image above):
  MY_IMAGE_ID=$($OPENSTACK_BINARY image list --public --property os_distro=coreos --status active --limit 1 -f json -c ID | sed 's/\r$//' | jq -r '.[]|.ID')
- Get the flavor to use:
  MY_FLAVOR='' while [ x"$MY_FLAVOR" = x ] ; do read -p 'Type the flavor you want (SHOW to show them): ' MY_FLAVOR [ x"$MY_FLAVOR" != xSHOW ] && break echo 'OpenStack flavors:' $OPENSTACK_BINARY flavor list echo '' MY_FLAVOR='' done
- Get the network to use (example below uses first internal
  network for your current project):
  MY_NETWORK_ID=$($OPENSTACK_BINARY network list --project $MY_PROJECT_ID --enable --internal -f value -c ID | sed 's/\r$//' | head -n 1)
- And the subnet:
  MY_SUBNET_ID=$($OPENSTACK_BINARY subnet list --network $MY_NETWORK_ID -f value -c ID | sed 's/\r$//' | head -n 1)
Now we should have enough information to create the cluster:
- First – let’s display all the environment variables we will be
  using:
  my_tmp="/tmp/$$.tmp" set | grep -e '^$MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_$\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)=' > $my_tmp cat "$my_tmp" echo '' found_items=$(wc -l $my_tmp | awk '{print $1}') [ $found_items -ne 8 ] && echo 'You are missing some items - please double-check...' || echo 'You have all the items defined' rm -f "$my_tmp"
  
  \
  Here is some typical output:
  MY_CLUSTER_PREFIX=coreos-k8s MY_FIRST_KEYPAIR=demo-abruce-new-admin MY_FLAVOR=m1.medium MY_IMAGE_ID=db581d51-514f-41d6-9517-2af1e96202ff MY_NETWORK_ID=2c737b0b-fa55-4208-a779-37cf9dfa18d1 MY_SECURITY_GROUP_DEFAULT_ID=0a30e90a-c5a9-4e63-906e-4ffa835984d4 MY_SECURITY_GROUP_ID=d1cf8fef-b57a-4ccc-9720-4012a5dcff27 OS_COREOS_CLUSTER_SIZE=3
- All look good? Then let’s create the cluster:
  $OPENSTACK_BINARY server create \ --image $MY_IMAGE_ID --flavor $MY_FLAVOR \ --security-group $MY_SECURITY_GROUP_DEFAULT_ID --security-group $MY_SECURITY_GROUP_STD_ACCESS_ID --security-group $MY_SECURITY_GROUP_ID \ --key-name $MY_FIRST_KEYPAIR --user-data /tmp/cloud-config.ign --network $MY_NETWORK_ID \ --min $OS_COREOS_CLUSTER_SIZE --max $OS_COREOS_CLUSTER_SIZE \ --wait \ $MY_CLUSTER_PREFIX
- Assign public IP (s) to the instances created; you can find them
  by running:
  $OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX"
  
  \
  Sample output looks like:
  +--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+ | 37ecb1b9-978e-41fa-9e73-ed6a62f634af | coreos-k8s-3 | ACTIVE | abruce-workspace-net=10.0.0.8 | coreos_production_openstack_image.img | m1.medium | | 2663b584-7b98-4fbb-bdfc-ceeee06cf8b1 | coreos-k8s-2 | ACTIVE | abruce-workspace-net=10.0.0.4 | coreos_production_openstack_image.img | m1.medium | | 06c296e9-a5a9-48ac-9f7f-aadfbb343e74 | coreos-k8s-1 | ACTIVE | abruce-workspace-net=10.0.0.3 | coreos_production_openstack_image.img | m1.medium | +--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
Here’s an example of creating / applying a floating IP to the each
server found:
- Get the list of servers created:
  MY_EXT_NET_NAME=ext-net MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//')
- Process each one, adding a floating IP if necessary:
  for i in $MY_SERVER_IDS ; do echo -n "Checking server '$i': " THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1) THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1) [ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && continue echo 'Create...' $OPENSTACK_BINARY floating ip create --port $THE_SERVER_PORT_ID $MY_EXT_NET_NAME done

Cleanup

Steps:

Make sure you have all your environment variables defined from
above:
set | grep -e '^$MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_$\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)='
Load in your servers:
MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//')
Cleanup floating IPs:
for i in $MY_SERVER_IDS ; do echo -n "Checking server '$i': " THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1) THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1) [ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && $OPENSTACK_BINARY floating ip delete $THE_SERVER_FLOATING_IP_ID && continue echo '[n/a]' done
Cleanup servers:
for i in $MY_SERVER_IDS ; do echo -n "Delete server '$i': " $OPENSTACK_BINARY server delete --wait $i done

CoreOS Cluster on OpenStack Pike

References

Creation

Cleanup

Leave a Reply Cancel reply