CoreOS Cluster on OpenStack Pike

Run a simple CoreOS cluster natively in OpenStack. Plus, learn a tremendous amount about the OpenStack CLI tools as opposed to Heat:https://docs.openstack.org/heat/latest/

References

  • https://coreos.com/os/docs/latest/booting-on-openstack.html – Latest official documentation
  • http://lollyrock.com/articles/coreos-openstack/ – Primary resource
  • https://docs.openstack.org/magnum/latest/user/ – Magnum-specific settings (so this page can be used for OpenStack Magnum integration)
  • https://cloudssky.com/en/blog/CoreOS-Cluster-on-OpenStack/ – Useful info including ports to open for CoreOS

Creation

Steps:

  1. Set your binary names:
    OPENSTACK_BINARY=openstack
    GLANCE_BINARY=glance

  2. Download / install to Glance:
    cd /tmp
    wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2
    bunzip2 coreos_production_openstack_image.img.bz2

    \
    To upload to OpenStack:
    $GLANCE_BINARY image-create --name coreos_production_openstack_image.img \
    --file ./coreos_production_openstack_image.img \
    --architecture x86_64 --os-distro coreos \
    --disk-format qcow2 --container-format bare \
    --protected True --visibility public \
    --progress

  3. For this basic demo, we will leverage the public CoreOS Container Linux cluster discovery endpoint. This is simply used to bootstrap the CoreOS cluster.
    (Adding additional nodes to the cluster will not use the public etcd
    cluster discovery endpoint.) Determine the number of nodes in the
    cluster:
    read -p 'Enter # of nodes for the CoreOS cluster: ' OS_COREOS_CLUSTER_SIZE

    \
    Retrieve the “discovery token” from the CoreOS public endpoint (for
    cluster bootstrapping):
    OS_COREOS_ETCD_DISCOVERY_TOKEN=$(curl -w "\n" "https://discovery.etcd.io/new?size=$OS_COREOS_CLUSTER_SIZE")

  4. We need one or more SSH public keys. Why? Because we will not be
    able to use Nova’s built-in SSH key management as we are manually
    specifying the cloud-config configuration file to use.
    Here are a couple of commands you can run to discover the first SSH
    key registered in your OpenStack project and information about that
    key:
    MY_FIRST_KEYPAIR=$($OPENSTACK_BINARY keypair list -f value -c Name | head -n 1 | sed 's/\r$//')

    \
    Next, show information about the keypair so you can get the public
    key information used to create the CoreOS cluster:
    $OPENSTACK_BINARY keypair show $MY_FIRST_KEYPAIR

    \
    Finally, load the public key into a variable. For example:
    OS_COREOS_PUBKEY=$(cat ~/.ssh/MY_FIRST_KEYPAIR.pub)

    \
    The above command will only work if your SSH public key follows that
    naming convention; YMMV.

  5. Create the cloud-config.yaml file which we will run
    through the CoreOS Config
    Transpiler

    to pass to OpenStack Nova for the etcd cluster creation. Assumes all
    the above has been completed first.

    • First, we generate the “Container Linux Config” (this is a
      human-readable file):

      ```
      cat > /tmp/cloud-config.yaml << EOF
      # This config is meant to be consumed by the config transpiler, which will
      # generate the corresponding Ignition config. Do not pass this config directly
      # to instances of Container Linux.
      
      etcd:
        # All options get passed as command line flags to etcd.
        # Any information inside curly braces comes from the machine at boot time.
      
        # multi_region and multi_cloud deployments need to use {PUBLIC_IPV4}
        advertise_client_urls:       "http://{PRIVATE_IPV4}:2379"
        initial_advertise_peer_urls: "http://{PRIVATE_IPV4}:2380"
        # listen on both the official ports and the legacy ports
        # legacy ports can be omitted if your application doesn't depend on them
        listen_client_urls:          "http://0.0.0.0:2379"
        listen_peer_urls:            "http://{PRIVATE_IPV4}:2380"
        # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3
        # specify the initial size of your cluster with ?size=X
        discovery:                   "$OS_COREOS_ETCD_DISCOVERY_TOKEN"
      passwd:
        users:
          - name: core
            ssh_authorized_keys:
              - $OS_COREOS_PUBKEY
      EOF
      ```
      </code></pre></li>
      <li>Next, generate the "CoreOS Ignition" file:
      
      <pre><code>```
      ct -in-file /tmp/cloud-config.yaml -out-file /tmp/cloud-config.ign -pretty -platform openstack-metadata
      ```
      
  6. Security group setup on OpenStack – there are a few ports which
    should be opened. While these could be added to the
    default security group, we don’t like that do we??
    Here’s some code that will check for / create the security group:

    • Get the OpenStack project ID based on your currently defined
      project name:
      MY_PROJECT_ID=$($OPENSTACK_BINARY project show $OS_PROJECT_NAME -f json -c id | sed 's/\r$//' | jq -r '.id')

    • Get the default security group ID for the current
      project:
      MY_SECURITY_GROUP_DEFAULT_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"default\")|.ID")

    • Get the std-access security group (note that you
      may use a different security group for SSH access):
      MY_SECURITY_GROUP_STD_ACCESS_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"std-access\")|.ID")

    • Get the list of security groups for this project:
      MY_SECURITY_GROUP_IDS=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|.ID")

    • Do we have any that support the ports we want?
      MY_SECURITY_GROUP_ID=''
      my_tmp="/tmp/$$.tmp"
      for i in $MY_SECURITY_GROUP_IDS ; do
      echo -n "Check security group '$i': "
      $OPENSTACK_BINARY security group rule list $i -f value -c 'Port Range' > "$my_tmp" 2>/dev/null
      grep -e '4001' "$my_tmp" 2>&1 >/dev/null && grep -e '7001' "$my_tmp" 2>&1 >/dev/null && MY_SECURITY_GROUP_ID="$i"
      rm -f "$my_tmp"
      [ x"$MY_SECURITY_GROUP_ID" != x ] && echo 'Found!' && break
      echo ''
      done
      if [ x"$MY_SECURITY_GROUP_ID" = x ] ; then
      echo -n ' Create new security group: '
      MY_SECURITY_GROUP_ID=$($OPENSTACK_BINARY security group create coreos --description coreos --project $MY_PROJECT_ID -f value -c id | sed 's/\r$//')
      echo "$MY_SECURITY_GROUP_ID"
      echo -n ' Create port rules...'
      $OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 4001 $MY_SECURITY_GROUP_ID
      $OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 7001 $MY_SECURITY_GROUP_ID
      fi

  7. Let’s get the remaining information:

    • Let’s make sure we have the prefix for the cluster:
      read -p 'CoreOS cluster prefix (short, please!): ' MY_CLUSTER_PREFIX

    • Get the image (from the uploaded CoreOS image above):
      MY_IMAGE_ID=$($OPENSTACK_BINARY image list --public --property os_distro=coreos --status active --limit 1 -f json -c ID | sed 's/\r$//' | jq -r '.[]|.ID')

    • Get the flavor to use:
      MY_FLAVOR=''
      while [ x"$MY_FLAVOR" = x ] ; do
      read -p 'Type the flavor you want (SHOW to show them): ' MY_FLAVOR
      [ x"$MY_FLAVOR" != xSHOW ] && break
      echo 'OpenStack flavors:'
      $OPENSTACK_BINARY flavor list
      echo ''
      MY_FLAVOR=''
      done

    • Get the network to use (example below uses first internal
      network for your current project):
      MY_NETWORK_ID=$($OPENSTACK_BINARY network list --project $MY_PROJECT_ID --enable --internal -f value -c ID | sed 's/\r$//' | head -n 1)

    • And the subnet:
      MY_SUBNET_ID=$($OPENSTACK_BINARY subnet list --network $MY_NETWORK_ID -f value -c ID | sed 's/\r$//' | head -n 1)

  8. Now we should have enough information to create the cluster:

    • First – let’s display all the environment variables we will be
      using:
      my_tmp="/tmp/$$.tmp"
      set | grep -e '^\(MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_\)\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)=' > $my_tmp
      cat "$my_tmp"
      echo ''
      found_items=$(wc -l $my_tmp | awk '{print $1}')
      [ $found_items -ne 8 ] && echo 'You are missing some items - please double-check...' || echo 'You have all the items defined'
      rm -f "$my_tmp"

      \
      Here is some typical output:
      MY_CLUSTER_PREFIX=coreos-k8s
      MY_FIRST_KEYPAIR=demo-abruce-new-admin
      MY_FLAVOR=m1.medium
      MY_IMAGE_ID=db581d51-514f-41d6-9517-2af1e96202ff
      MY_NETWORK_ID=2c737b0b-fa55-4208-a779-37cf9dfa18d1
      MY_SECURITY_GROUP_DEFAULT_ID=0a30e90a-c5a9-4e63-906e-4ffa835984d4
      MY_SECURITY_GROUP_ID=d1cf8fef-b57a-4ccc-9720-4012a5dcff27
      OS_COREOS_CLUSTER_SIZE=3

    • All look good? Then let’s create the cluster:
      $OPENSTACK_BINARY server create \
      --image $MY_IMAGE_ID --flavor $MY_FLAVOR \
      --security-group $MY_SECURITY_GROUP_DEFAULT_ID --security-group $MY_SECURITY_GROUP_STD_ACCESS_ID --security-group $MY_SECURITY_GROUP_ID \
      --key-name $MY_FIRST_KEYPAIR --user-data /tmp/cloud-config.ign --network $MY_NETWORK_ID \
      --min $OS_COREOS_CLUSTER_SIZE --max $OS_COREOS_CLUSTER_SIZE \
      --wait \
      $MY_CLUSTER_PREFIX

    • Assign public IP (s) to the instances created; you can find them
      by running:
      $OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX"

      \
      Sample output looks like:
      +--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
      | ID | Name | Status | Networks | Image | Flavor |
      +--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
      | 37ecb1b9-978e-41fa-9e73-ed6a62f634af | coreos-k8s-3 | ACTIVE | abruce-workspace-net=10.0.0.8 | coreos_production_openstack_image.img | m1.medium |
      | 2663b584-7b98-4fbb-bdfc-ceeee06cf8b1 | coreos-k8s-2 | ACTIVE | abruce-workspace-net=10.0.0.4 | coreos_production_openstack_image.img | m1.medium |
      | 06c296e9-a5a9-48ac-9f7f-aadfbb343e74 | coreos-k8s-1 | ACTIVE | abruce-workspace-net=10.0.0.3 | coreos_production_openstack_image.img | m1.medium |
      +--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+

  9. Here’s an example of creating / applying a floating IP to the each
    server found:

    • Get the list of servers created:
      MY_EXT_NET_NAME=ext-net
      MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//')

    • Process each one, adding a floating IP if necessary:
      for i in $MY_SERVER_IDS ; do
      echo -n "Checking server '$i': "
      THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1)
      THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1)
      [ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && continue
      echo 'Create...'
      $OPENSTACK_BINARY floating ip create --port $THE_SERVER_PORT_ID $MY_EXT_NET_NAME
      done

Cleanup

Steps:

  1. Make sure you have all your environment variables defined from
    above:
    set | grep -e '^\(MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_\)\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)='

  2. Load in your servers:
    MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//')

  3. Cleanup floating IPs:
    for i in $MY_SERVER_IDS ; do
    echo -n "Checking server '$i': "
    THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1)
    THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1)
    [ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && $OPENSTACK_BINARY floating ip delete $THE_SERVER_FLOATING_IP_ID && continue
    echo '[n/a]'
    done

  4. Cleanup servers:
    for i in $MY_SERVER_IDS ; do
    echo -n "Delete server '$i': "
    $OPENSTACK_BINARY server delete --wait $i
    done

Team-oriented systems mentor with deep knowledge of numerous software methodologies, technologies, languages, and operating systems. Excited about turning emerging technology into working production-ready systems. Focused on moving software teams to a higher level of world-class application development. Specialties:Software analysis and development...Product management through the entire lifecycle...Discrete product integration specialist!

Tagged with: ,

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Human Verification: In order to verify that you are a human and not a spam bot, please enter the answer into the following box below based on the instructions contained in the graphic.