CoreOS Cluster on OpenStack Pike
FollowRun a simple CoreOS cluster natively in OpenStack. Plus, learn a tremendous amount about the OpenStack CLI tools as opposed to Heat:https://docs.openstack.org/heat/latest/
References
- https://coreos.com/os/docs/latest/booting-on-openstack.html – Latest official documentation
- http://lollyrock.com/articles/coreos-openstack/ – Primary resource
- https://docs.openstack.org/magnum/latest/user/ – Magnum-specific settings (so this page can be used for OpenStack Magnum integration)
- https://cloudssky.com/en/blog/CoreOS-Cluster-on-OpenStack/ – Useful info including ports to open for CoreOS
Creation
Steps:
- Set your binary names:
OPENSTACK_BINARY=openstack
GLANCE_BINARY=glance -
Download / install to Glance:
cd /tmp
wget http://beta.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2
bunzip2 coreos_production_openstack_image.img.bz2\
To upload to OpenStack:
$GLANCE_BINARY image-create --name coreos_production_openstack_image.img \
--file ./coreos_production_openstack_image.img \
--architecture x86_64 --os-distro coreos \
--disk-format qcow2 --container-format bare \
--protected True --visibility public \
--progress -
For this basic demo, we will leverage the public CoreOS Container Linux cluster discovery endpoint. This is simply used to bootstrap the CoreOS cluster.
(Adding additional nodes to the cluster will not use the public etcd
cluster discovery endpoint.) Determine the number of nodes in the
cluster:
read -p 'Enter # of nodes for the CoreOS cluster: ' OS_COREOS_CLUSTER_SIZE\
Retrieve the “discovery token” from the CoreOS public endpoint (for
cluster bootstrapping):
OS_COREOS_ETCD_DISCOVERY_TOKEN=$(curl -w "\n" "https://discovery.etcd.io/new?size=$OS_COREOS_CLUSTER_SIZE") -
We need one or more SSH public keys. Why? Because we will not be
able to use Nova’s built-in SSH key management as we are manually
specifying thecloud-configconfiguration file to use.
Here are a couple of commands you can run to discover the first SSH
key registered in your OpenStack project and information about that
key:
MY_FIRST_KEYPAIR=$($OPENSTACK_BINARY keypair list -f value -c Name | head -n 1 | sed 's/\r$//')\
Next, show information about the keypair so you can get the public
key information used to create the CoreOS cluster:
$OPENSTACK_BINARY keypair show $MY_FIRST_KEYPAIR\
Finally, load the public key into a variable. For example:
OS_COREOS_PUBKEY=$(cat ~/.ssh/MY_FIRST_KEYPAIR.pub)\
The above command will only work if your SSH public key follows that
naming convention; YMMV. -
Create the
cloud-config.yamlfile which we will run
through the CoreOS Config
Transpiler
to pass to OpenStack Nova for the etcd cluster creation. Assumes all
the above has been completed first.- First, we generate the “Container Linux Config” (this is a
human-readable file):``` cat > /tmp/cloud-config.yaml << EOF # This config is meant to be consumed by the config transpiler, which will # generate the corresponding Ignition config. Do not pass this config directly # to instances of Container Linux. etcd: # All options get passed as command line flags to etcd. # Any information inside curly braces comes from the machine at boot time. # multi_region and multi_cloud deployments need to use {PUBLIC_IPV4} advertise_client_urls: "http://{PRIVATE_IPV4}:2379" initial_advertise_peer_urls: "http://{PRIVATE_IPV4}:2380" # listen on both the official ports and the legacy ports # legacy ports can be omitted if your application doesn't depend on them listen_client_urls: "http://0.0.0.0:2379" listen_peer_urls: "http://{PRIVATE_IPV4}:2380" # generate a new token for each unique cluster from https://discovery.etcd.io/new?size=3 # specify the initial size of your cluster with ?size=X discovery: "$OS_COREOS_ETCD_DISCOVERY_TOKEN" passwd: users: - name: core ssh_authorized_keys: - $OS_COREOS_PUBKEY EOF ``` </code></pre></li> <li>Next, generate the "CoreOS Ignition" file: <pre><code>``` ct -in-file /tmp/cloud-config.yaml -out-file /tmp/cloud-config.ign -pretty -platform openstack-metadata ```
- First, we generate the “Container Linux Config” (this is a
- Security group setup on OpenStack – there are a few ports which
should be opened. While these could be added to the
defaultsecurity group, we don’t like that do we??
Here’s some code that will check for / create the security group:- Get the OpenStack project ID based on your currently defined
project name:
MY_PROJECT_ID=$($OPENSTACK_BINARY project show $OS_PROJECT_NAME -f json -c id | sed 's/\r$//' | jq -r '.id') -
Get the
defaultsecurity group ID for the current
project:
MY_SECURITY_GROUP_DEFAULT_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"default\")|.ID") -
Get the
std-accesssecurity group (note that you
may use a different security group for SSH access):
MY_SECURITY_GROUP_STD_ACCESS_ID=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|select(.Name==\"std-access\")|.ID") -
Get the list of security groups for this project:
MY_SECURITY_GROUP_IDS=$($OPENSTACK_BINARY security group list --project $MY_PROJECT_ID -f json -c ID -c Name | sed 's/\r$//' | jq -r ".[]|.ID") -
Do we have any that support the ports we want?
MY_SECURITY_GROUP_ID=''
my_tmp="/tmp/$$.tmp"
for i in $MY_SECURITY_GROUP_IDS ; do
echo -n "Check security group '$i': "
$OPENSTACK_BINARY security group rule list $i -f value -c 'Port Range' > "$my_tmp" 2>/dev/null
grep -e '4001' "$my_tmp" 2>&1 >/dev/null && grep -e '7001' "$my_tmp" 2>&1 >/dev/null && MY_SECURITY_GROUP_ID="$i"
rm -f "$my_tmp"
[ x"$MY_SECURITY_GROUP_ID" != x ] && echo 'Found!' && break
echo ''
done
if [ x"$MY_SECURITY_GROUP_ID" = x ] ; then
echo -n ' Create new security group: '
MY_SECURITY_GROUP_ID=$($OPENSTACK_BINARY security group create coreos --description coreos --project $MY_PROJECT_ID -f value -c id | sed 's/\r$//')
echo "$MY_SECURITY_GROUP_ID"
echo -n ' Create port rules...'
$OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 4001 $MY_SECURITY_GROUP_ID
$OPENSTACK_BINARY security group rule create --project $MY_PROJECT_ID --dst-port 7001 $MY_SECURITY_GROUP_ID
fi
- Get the OpenStack project ID based on your currently defined
-
Let’s get the remaining information:
- Let’s make sure we have the prefix for the cluster:
read -p 'CoreOS cluster prefix (short, please!): ' MY_CLUSTER_PREFIX -
Get the image (from the uploaded CoreOS image above):
MY_IMAGE_ID=$($OPENSTACK_BINARY image list --public --property os_distro=coreos --status active --limit 1 -f json -c ID | sed 's/\r$//' | jq -r '.[]|.ID') -
Get the flavor to use:
MY_FLAVOR=''
while [ x"$MY_FLAVOR" = x ] ; do
read -p 'Type the flavor you want (SHOW to show them): ' MY_FLAVOR
[ x"$MY_FLAVOR" != xSHOW ] && break
echo 'OpenStack flavors:'
$OPENSTACK_BINARY flavor list
echo ''
MY_FLAVOR=''
done -
Get the network to use (example below uses first internal
network for your current project):
MY_NETWORK_ID=$($OPENSTACK_BINARY network list --project $MY_PROJECT_ID --enable --internal -f value -c ID | sed 's/\r$//' | head -n 1) -
And the subnet:
MY_SUBNET_ID=$($OPENSTACK_BINARY subnet list --network $MY_NETWORK_ID -f value -c ID | sed 's/\r$//' | head -n 1)
- Let’s make sure we have the prefix for the cluster:
-
Now we should have enough information to create the cluster:
- First – let’s display all the environment variables we will be
using:
my_tmp="/tmp/$$.tmp"
set | grep -e '^\(MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_\)\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)=' > $my_tmp
cat "$my_tmp"
echo ''
found_items=$(wc -l $my_tmp | awk '{print $1}')
[ $found_items -ne 8 ] && echo 'You are missing some items - please double-check...' || echo 'You have all the items defined'
rm -f "$my_tmp"\
Here is some typical output:
MY_CLUSTER_PREFIX=coreos-k8s
MY_FIRST_KEYPAIR=demo-abruce-new-admin
MY_FLAVOR=m1.medium
MY_IMAGE_ID=db581d51-514f-41d6-9517-2af1e96202ff
MY_NETWORK_ID=2c737b0b-fa55-4208-a779-37cf9dfa18d1
MY_SECURITY_GROUP_DEFAULT_ID=0a30e90a-c5a9-4e63-906e-4ffa835984d4
MY_SECURITY_GROUP_ID=d1cf8fef-b57a-4ccc-9720-4012a5dcff27
OS_COREOS_CLUSTER_SIZE=3 -
All look good? Then let’s create the cluster:
$OPENSTACK_BINARY server create \
--image $MY_IMAGE_ID --flavor $MY_FLAVOR \
--security-group $MY_SECURITY_GROUP_DEFAULT_ID --security-group $MY_SECURITY_GROUP_STD_ACCESS_ID --security-group $MY_SECURITY_GROUP_ID \
--key-name $MY_FIRST_KEYPAIR --user-data /tmp/cloud-config.ign --network $MY_NETWORK_ID \
--min $OS_COREOS_CLUSTER_SIZE --max $OS_COREOS_CLUSTER_SIZE \
--wait \
$MY_CLUSTER_PREFIX -
Assign public IP (s) to the instances created; you can find them
by running:
$OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX"\
Sample output looks like:
+--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
| 37ecb1b9-978e-41fa-9e73-ed6a62f634af | coreos-k8s-3 | ACTIVE | abruce-workspace-net=10.0.0.8 | coreos_production_openstack_image.img | m1.medium |
| 2663b584-7b98-4fbb-bdfc-ceeee06cf8b1 | coreos-k8s-2 | ACTIVE | abruce-workspace-net=10.0.0.4 | coreos_production_openstack_image.img | m1.medium |
| 06c296e9-a5a9-48ac-9f7f-aadfbb343e74 | coreos-k8s-1 | ACTIVE | abruce-workspace-net=10.0.0.3 | coreos_production_openstack_image.img | m1.medium |
+--------------------------------------+--------------+--------+-------------------------------+---------------------------------------+-----------+
- First – let’s display all the environment variables we will be
-
Here’s an example of creating / applying a floating IP to the each
server found:- Get the list of servers created:
MY_EXT_NET_NAME=ext-net
MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//') -
Process each one, adding a floating IP if necessary:
for i in $MY_SERVER_IDS ; do
echo -n "Checking server '$i': "
THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1)
THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1)
[ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && continue
echo 'Create...'
$OPENSTACK_BINARY floating ip create --port $THE_SERVER_PORT_ID $MY_EXT_NET_NAME
done
- Get the list of servers created:
Cleanup
Steps:
-
Make sure you have all your environment variables defined from
above:
set | grep -e '^\(MY_\(CLUSTER_PREFIX\|IMAGE_ID\|FLAVOR\|SECURITY_GROUP_\(DEFAULT_\)\?ID\|FIRST_KEYPAIR\|NETWORK_ID\)\|OS_COREOS_CLUSTER_SIZE\)=' -
Load in your servers:
MY_SERVER_IDS=$($OPENSTACK_BINARY server list --name "$MY_CLUSTER_PREFIX" -f value -c ID | sed 's/\r$//') -
Cleanup floating IPs:
for i in $MY_SERVER_IDS ; do
echo -n "Checking server '$i': "
THE_SERVER_PORT_ID=$($OPENSTACK_BINARY port list --server $i -f value -c ID | sed 's/\r$//' | head -n 1)
THE_SERVER_FLOATING_IP_ID=$($OPENSTACK_BINARY floating ip list --port $THE_SERVER_PORT_ID -f value -c ID | sed 's/\r$//' | head -n 1)
[ x"$THE_SERVER_FLOATING_IP_ID" != x ] && echo $THE_SERVER_FLOATING_IP_ID && $OPENSTACK_BINARY floating ip delete $THE_SERVER_FLOATING_IP_ID && continue
echo '[n/a]'
done -
Cleanup servers:
for i in $MY_SERVER_IDS ; do
echo -n "Delete server '$i': "
$OPENSTACK_BINARY server delete --wait $i
done
Team-oriented systems mentor with deep knowledge of numerous software methodologies, technologies, languages, and operating systems. Excited about turning emerging technology into working production-ready systems. Focused on moving software teams to a higher level of world-class application development. Specialties:Software analysis and development...Product management through the entire lifecycle...Discrete product integration specialist!
Leave a Reply