Deployment using Kubernetes Helm

One of the challenges I face in my development setup is that I want to quickly and often create and deploy my robotics stack. I often want to change and redeploy my entire stack from scratch, because I want to iterate quickly and also reduce my costs as much as possible. My Jenkins jobs have helped a great deal here, and automation is definitely key. However I have recently started experimenting with Kubernetes Helm which is a package manager for Kubernetes which has made this even easier for me.

Kubernetes Helm

Helm is a package manager that allows you to define a package with all its dependent deployment objects for Kubernetes. With helm and this package you can then ask a cluster to install the entire package in one go instead of passing individual deployment commands. This means for me that instead of asking Kubernetes to install each of my several micro-services to be installed I simply ask it to install the entire package/release in one atomic action which also includes all of the dependent services like databases and message brokers I use.

Installing Helm

In this blog I want to give a small taste on how nice Helm is. So how do we get started? Well in order to get started with Helm you should first follow the installation instructions at this page: https://github.com/kubernetes/helm/blob/master/docs/install.md

In case you are using OSX (like me) its relatively simple if you are using homebrew, simply run the following cask:

brew cask install helm

Once helm is installed it should also be installed in your cluster. In my case I will be testing against a minikube installation as described in my previous post: https://renzedevries.wordpress.com/2016/11/07/using-kubernetes-minikube-for-local-test-deployments/

On the command line I have a kubernetes command line client (kubectl) with my configuration pointing towards my minikube cluster. The only thing I have to do is the following to install Helm in my cluster:

helm init

This will install a container named tiller in my cluster, this container will understand how to deploy the Helm packages (charts) into my cluster. This is in essence the main endpoint the helm client will use to interrogate the cluster for package deployments and package changes.

Creating the package

Next we need to start creating something which is called a Chart, this is the unit of packaging in Helm. For this post I will reduce the set of services I have used in previous posts and only deploy the core services Cassandra, MQTT and ActiveMQ. The first thing to define is the *Chart.yaml** which is the package manifest:

Chart.yaml
The manifest looks pretty simple, most important is the version number, the rest is mainly metadata for indexing:

name: robotics
version: 0.1
description: Robotic automation stack
keywords:
- robotics
- application
maintainers:
- name: Renze de Vries
engine: gotpl

The second I am going to define is the deployment objects I want to deploy. For this we create a ‘Charts’ subdirectory which contains these dependent services. In this case I am going to deploy MQTT, ActiveMQ and Cassandra which are required for my project. For each of these services I create a templates folder which contains the Kubernetes Deployment.yaml descriptor and Kubernetes service descriptor file and have their own Charts.yaml file as well.

When you have this all ready it look as following:
screen-shot-2016-11-16-at-20-42-12

I am not going to write out all the files in this blog, if you want to have a look at the full source have a look at the github repository here that contains the full Helm chart structure describe in this post: https://github.com/renarj/helm-repo

Packaging a release

Now that the Chart source files have been created the last thing to do is to create the actual package. For this we have to do nothing else than simply run the following command:

helm package .

This will create a file called robotics-0.1.tgz that we can use further to deploy our release. In a future blog post I will talk a bit about Helm repositories and how you can distribute these packages, but for now we keep them on the local file system.

Installing a release

Once we have defined the packages the only thing thats remaining is to simply install a release into the cluster. This will install all the services that are packaged in the Chart.

In order to install the package we have created above we just have to run the following command:

helm install robotics-0.1.tgz
NAME: washing-tuatar
LAST DEPLOYED: Sun Nov  6 20:42:30 2016
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME          CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
amq       10.0.0.59   <nodes>   61616/TCP   1s
mqtt      10.0.0.131   <nodes>   1883/TCP   1s
cassandra-svc   10.0.0.119   <nodes>   9042/TCP,9160/TCP   1s

==> extensions/Deployment
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
mqtt      1         1         1            0           1s
amq       1         1         1         0         1s
cassandra   1         1         1         0         1s

We can ask Helm which packages are installed in the cluster by simply asking a list of installed packages as following:

helm list
NAME          	REVISION	UPDATED                 	STATUS  	CHART       
washing-tuatar	1       	Sun Nov  6 20:42:30 2016	DEPLOYED	robotics-0.1

Please note that the name for the installation is a random generated name, in case you want a well known name you can install using the ‘-name’ switch and specify the name yourself.

In order to delete all the deployed objects I can simply ask Helm to uninstall the release as following:

helm delete washing-tuatar

Conclusion

I have found that Helm has a big potential, it allows me to very quickly define a full software solution composed out of many individual deployments. In a future blog post I will talk a bit more about the templating capabilities of Helm and the packaging and distributing of your packages. In the end I hope this blog shows everyone that with Helm you can make all of your Kubernetes work even easier than it already is today🙂

Using Kubernetes Minikube for Local test deployments

One of the many challenges I face with my Robotics Cloud development is the need to test locally and constantly re-create the stack from scratch. Now I have a lot of automation to deploy against AWS using Jenkins as seen in previous posts. However setting up a local development environment is the thing I do the most and that is costing a lot of time because the tooling always was painful to use.

Now in the last few months there have been a lot of innovations happening in the Kubernetes field. In particular in this blog post I want to talk about using Minikube.

Minikube

One of the big pains was always to setup a Kubernetes cluster on your local machine. Before there were some solutions, the simplest one was to use vagrant or the kube-up script that would create some vm’s in virtualbox. However my experience was that they were error prone and did not always complete succesfully. For local machine development setups there is now a new solution called minikube. In essence using minikube you can create a single machine kubernetes test cluster to get you quickly up and running.

The simplest way to get started is to install minikube first using the latest release instructions, in my case for OSX on the 0.12.2 release I install it using this command:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.12.2/minikube-darwin-amd64 &amp;&amp; chmod +x minikube &amp;&amp; sudo mv minikube /usr/local/bin/

Please visit this page for the latest release of minikube: https://github.com/kubernetes/minikube/releases

In essence the above command downloads the minikube binary and moves it to the local usr bin directory so its available on the path. After this we can start creating the minikube machine, in my case I will use virtualbox as the provider which is automatically detected if its installed. In my case all i have to do is the following:
minikube start --memory=8196

The above will start a single node kubernetes cluster which acts both as master and worker in virtualbox with 8GB of memory. Also it will ensure my local kubernetes (kubectl) client configuration is set to point to the cluster master. This will take a few minutes to get up and running but the cluster should be available after this and you can check if its ready by doing this:

kubectl get nodes
NAME       STATUS    AGE
minikube   Ready     1h

The minikube setup has created a virtual box setup that exposes all its services via the virtualbox ip. The minikube binary provides a shortcut to get that ip using below command:

minikube ip
192.168.99.100

This ip can be used to directly access all services that are exposed on the kubernetes cluster.

Now the cluster is available you can start deploying to your hearts content, but you might want to use the kubernetes dashboard for this which is handy for the overview. In order to quickly get to the dashboard you can run this minikube command:

minikube dashboard

If you want to stop the cluster you can simply type the following command:

minikube stop

The next time you start the cluster it will resume the state it was in previously. So all previously running containers will also be started once the cluster comes back up which is quite handy in case of development.

Conclusion

I hope this post helps people who are struggling setting up their own Kubernetes cluster and getting them quickly started. I am sure there is a lot more to come from the Kubernetes folks, its really getting easier and easier🙂

Versioning and deploying Docker containers using Kubernetes and Jenkins

In the last few posts the main comment that keeps coming back is ‘you should not use latest’. I totally agree on that and in this blog I will finally do something about it🙂 I mainly used the setup for development purpose, however in production I need something more reliable, meaning versioned deployments.

To do this there are of course two parts, first I have to actually create a build job that can version the containers and second I will need a build job that rolls out the update deployed container.

Versioning and Releasing

Using the below Jenkins pipeline I have a build job that uses a Jenkins input parameter that defines the release version. In order to use this create a Jenkins pipeline job that has one input parameter called ‘RELEASE_VERSION’. The default value I keep to ‘dev-latest’ for development purposes, but when releasing we obviously need to use a sensible number.

node {
    stage 'build'
    build 'home projects/command-svc/master'

    stage 'test-deploy'
    sh "\$(aws ecr get-login)"
    sh "docker tag home-core/command-svc:latest aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:dev-latest"
    sh "docker push aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:dev-latest"    

    stage 'qa'
    build 'home projects/command-svc-tests/master'

    stage 'Publish containers'
    sh "docker tag home-core/command-svc:latest aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:'$RELEASE_VERSION'"
    sh "docker push aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:'$RELEASE_VERSION'"
}

Now when I trigger the build Job, Jenkins will ask me to input the version for releasing that container. So let’s in this article use the version ‘0.0.1’. When running the build job, Jenkins will go through a few stages.
1. Building the container
2. Pushing a dev-latest version
3. Running tests which deploy a container against the test cluster
4. Release the container using a fixed version.

In the latest stage I release the container using the input parameter specified on the job triggering. The build job does not actually deploy to production, that is for the time being still a manual action.

Deploying to Kubernetes

For deploying to production I have done an initial deployment of the service using below deployment descriptor. I started out deploying version ‘0.0.1’ which I have built above.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: command-svc
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: command-svc
    spec:
      containers:
      - name: command-svc
        image: aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:0.0.1
        ports:
        - containerPort: 8080
        env:
        - name: amq_host
          value: amq
        - name: SPRING_PROFILES_ACTIVE
          value: production

Doing a rolling update

This works fine for an initial deployment, however if i want to do an upgrade of my production containers I need something more. In production in essence I just want to upgrade the container image version. This is a relatively simple operation with Kubernetes. Let’s assume i have released a newer version of the container with version ‘0.0.2’.

In order to update the container in Kubernetes I can simply do a rolling update by changing the image of the Deployment object in Kubernetes as following:

kubectl set image deployment/command-svc command-svc=aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:0.0.2

I am currently not yet integrating this into my build pipeline as I want a production upgrade to be a conscious decision still. But once all the quality gates are in place, there should be no reason to not automate the above step as well. More on this in some future blog posts.

Having fun with Robots and Model trains

Last few blog posts have all been about heavy docker and Kubernetes stuff. I thought it was time for something more light and today I want to blog about my hobby robotics project again. Before I had robots to tinker with actually I used to play around a lot with trying to automate a model train setup. So recently I had the idea why can’t I combine this and let one of the robots have some fun by playing with a model train🙂

In this blog post I will use MQTT and my Robot Cloud SDK I have developed to hook up our Nao Robot to MQTT together with the model train.

Needed materials

In order to build a automated train layout I needed a model train setup, I have a already existing H0 based Roco/Fleischmann based model train setup. All the trains on this setup are digitised using decoders that are based on DCC. If you do not know what this means, you can read a about digital train systems here: http://www.dccwiki.com/DCC_Tutorial_(Basic_System)

Hooking up the train

The train system I have is controlled using an Ecos controller which has a well defined TCP network protocol I can use for controlling it. I have written a small library that hooks the controller to my IoT/robot cloud that I have described in previous blogposts. The commands for moving the train are sent to MQTT which are then translated to a TCP command the controller can understand.

I will have a MQTT broker available somewhere in the Cloud (AWS/Kubernetes) where also my robots can connect to so this will be the glue connecting the robot and trains.

I don’t really want to bother people to much with the technicals of the train and code behind it, but if you are interested in the code I have put it on Github: https://github.com/renarj/rc-train

Hooking up the robots

Hooking up the robots is actually quite simple, I have done this before and am using the same setup before. The details of this are all available in this blog post: https://renzedevries.wordpress.com/2016/06/10/robot-interaction-with-a-nao-robot-and-a-raspberry-pi-robot/

In this case I will be using our Nao Robot and hook this up to the MQTT bridge. The framework have developed contains a standard message protocol on top of MQTT. This means the messages are always defined the same way and all parties adhering to this can give states and commands to each other. In this case both the train and robot use the same message protocol via MQTT, hence why we can hook them up.

In order to make this a bit more entertaining I want to run a small scenario:
1. Nao walks a bit towards the train and the controller
2. Sits down and says something
3. Starts the train
4. Reacts when the train is running

I always like to put a bit of code in a post, so this code is used to create this scenario:

    private static void runScenario(Robot robot) {
        //Step1: Walk to the train controller (1.2 meters)
        robot.getMotionEngine().prepareWalk();
        robot.getMotionEngine().walk(WalkDirection.FORWARD, 1.2f);

        //Step2: Let's say something and sit down
        robot.getCapability(SpeechEngine.class).say("Oh is that a model train, let me sit and play with it", "english");
        robot.getMotionEngine().goToPosture("Sit");
        
        //Step3: Let's start the train
        sleepUninterruptibly(1, TimeUnit.SECONDS);
        startTrain(robot.getRemoteDriver(), "1005", "forward");
        
        //Step4: Nao is having lots of fun
        sleepUninterruptibly(5, TimeUnit.SECONDS);
        robot.getCapability(SpeechEngine.class).say("I am having so much fun playing with the train", "english");
    }

    private static void startTrain(RemoteDriver remoteDriver, String trainId, String direction) {
        remoteDriver.publish(BasicCommandBuilder.create("ecos")
                .item("train").label("control")
                .property("trainId", trainId).build());
        remoteDriver.publish(BasicCommandBuilder.create("ecos")
                .item("train").label("light")
                .property("trainId", trainId)
                .property("state", "on").build());
        remoteDriver.publish(BasicCommandBuilder.create("ecos")
                .item("train").label("direction")
                .property("trainId", trainId)
                .property("direction", direction).build());
        remoteDriver.publish(BasicCommandBuilder.create("ecos")
                .item("train").label("speed")
                .property("trainId", trainId)
                .property("speed", "127").build());
    }

What happens here is that in the Robot SDK there is a bit of code that can translate Java objects into MQTT messages. Those MQTT messages are then received by the train controller from the MQTT bridge which translates this again into TCP messages.

For people that are interested in also this piece of code on how I create the scenario’s around the Nao robot it’s also available on github: https://github.com/renarj/robo-pep

End result

So how does this end result look like, well video’s say more than a thousand words (actually ±750 for this post🙂 )

This is just to show that you can have a bit of fun integrating very different devices. Using protocols like MQTT could really empower robot and other appliances to be tightly integrated very easily. The glue that I am adding is to make sure there is a standard message on top of MQTT for the different appliances and hooking them up to MQTT. Stay tuned for some more posts about my Robotics and hobby projects.

Deploying Docker containers to Kubernetes with Jenkins

After all my recent posts about deploying a Kubernetes cluster to AWS the one step I still wanted to talk about is how you can deploy the Docker containers to a Kubernetes cluster using a bit of automation. I will try to explain here how you can relatively simply do this by using Jenkins pipelines and some groovy scripting🙂

Pre-requisites
* Working Kubernetes cluster (see here: https://renzedevries.wordpress.com/2016/07/18/deploying-kubernetes-to-aws-using-jenkins/)
* Jenkins slave/master setup
* Kubectl tool installed and configured on the Jenkins master/slave and desktop
* Publicly accessible Docker images (AWS ECR for example see: https://renzedevries.wordpress.com/2016/07/20/publishing-a-docker-image-to-amazon-ecr-using-jenkins/)

What are we deploying
In order to deploy containers against kubernetes there are two things that are needed. First I need to deploy the services that will ensure that we have ingress traffic via AWS ELB’s and this also ensures we have an internal DNS lookup capability for service to service communication. Second I need to deploy the actual containers using Kubernetes Deployments.

In this post I will focus on mainly one service which is called ‘command-service’. If you want to read a bit more about the services that I deploy you can find that here: https://renzedevries.wordpress.com/2016/06/10/robot-interaction-with-a-nao-robot-and-a-raspberry-pi-robot/

Creating the services

The first task I do is to create the actual kubernetes service for the command-service. The service descriptors are relatively simple in my case, the command-service needs to be publicly load balanced so I want kubernetes to create an AWS ELB for me. I will deploy this service by first checking out my git repository where I contain the service descriptors using Kubernetes yaml files. I will then use a Jenkins pipeline with some groovy scripting to deploy it.

The service descriptor for the public loadbalanced command-svc looks like this. This is a load balancer that is backed by all pods that have a label ‘app’ with value ‘command-svc’ and then attached to the AWS ELB backing this service.

apiVersion: v1
kind: Service
metadata:
  name: command-svc
  labels:
    app: command-svc
spec:
  type: LoadBalancer
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
    name: command-svc
  selector:
    app: command-svc

In order to actually create this services I use the below Jenkins pipeline code. In this code I use the apply command because the services are not very likely to change and this way it works both in clean and already existing environments. Because I constantly create new environments and sometimes update existing ones, I want all my scripts to be runnable multiple times regardless of current cluster/deployment state.

import groovy.json.*

node {
    stage 'prepare'
    deleteDir()

    git credentialsId: 'bb420c66-8efb-43e5-b5f6-583b5448e984', url: 'git@bitbucket.org:oberasoftware/haas-build.git'
    sh "wget http://localhost:8080/job/kube-deploy/lastSuccessfulBuild/artifact/*zip*/archive.zip"
    sh "unzip archive.zip"
    sh "mv archive/* ."

    stage "deploy services"
    sh "kubectl apply -f command-svc.yml --kubeconfig=kubeconfig"
    waitForServices()
}

Waiting for creation
One of the challenges I faced tho is that I have a number of containers that I want to deploy that depend on these service definitions. However it takes a bit of time to deploy these services and for the ELB’s to be fully created. So I have created a bit of small waiting code in Groovy that checks if the services are up and running. This is being called using the ‘waitForServices()’ method in the pipeline, you can see the code for this below:

def waitForServices() {
  sh "kubectl get svc -o json > services.json --kubeconfig=kubeconfig"

  while(!toServiceMap(readFile('services.json')).containsKey('command-svc')) {
        sleep(10)
        echo "Services are not yet ready, waiting 10 seconds"
        sh "kubectl get svc -o json > services.json --kubeconfig=kubeconfig"
  }
  echo "Services are ready, continuing"
}

@com.cloudbees.groovy.cps.NonCPS
Map toServiceMap(servicesJson) {
  def json = new JsonSlurper().parseText(servicesJson)

  def serviceMap = [:]
  json.items.each { i ->
    def serviceName = i.metadata.name
    def ingress = i.status.loadBalancer.ingress
    if(ingress != null) {
      def serviceUrl = ingress[0].hostname
      serviceMap.put(serviceName, serviceUrl)
    }
  }

  return serviceMap
}

This should not complete until at least all the services are ready for usage, in this case my command-svc with its ELB backing.

Creating the containers

The next step is actually the most important, deploying the actual container. In this example I will be using the deployments objects that are there since Kubernetes 1.2.x.

Let’s take a look again at the command-svc container that I want to deploy. I use again the yaml file syntax for describing the deployment object:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: command-svc
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: command-svc
    spec:
      containers:
      - name: command-svc
        image: account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:latest
        ports:
        - containerPort: 8080
        env:
        - name: amq_host
          value: amq
        - name: SPRING_PROFILES_ACTIVE
          value: production

Let’s put all that together for the rest of my deployments for the other containers. In this case I have one additional container that I deploy the edge-service. Using Jenkins pipelines this looks relatively simple:

    stage "deploy"
    sh "kubectl apply -f kubernetes/command-deployment.yml --kubeconfig=kubeconfig"

I currently do not have any active health checking at the end of the deployment, i am still planning on it. For now I just check that the pods and deployments are properly deployed, you can also do this by simply running these commands:
kubectl get deployments

This will yield something like below:

NAME          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
command-svc   1         1         1            1           1m

If you check the running pods kubectl get po you can see the deployment has scheduled a single pod:

NAME                          READY     STATUS    RESTARTS   AGE
command-svc-533647621-e85yo   1/1       Running   0          2m

Conclusion

I hope in this article I have taken away a bit of the difficulty on how to deploy your containers against Kubernetes. It can be done relatively simple, of course its not production grade but it shows on a very basic level how with any basic scripting (groovy) you can accomplish this task by just using Jenkins.

Upgrades
In this particular article I have not zoomed into the act of upgrading a cluster or the containers running on them. I will discuss this in a future blog post where I will zoom in on the particulars of doing rolling-updates on your containers and eventually will address the upgrade of the cluster itself on AWS.

Publishing a Docker image to Amazon ECR using Jenkins

I wanted to do a quick post, because some recent posts have lead to some questions about how do I actually make a docker container available on AWS. Luckily Amazon has a solution for this and its called Amazon ECR (EC2 Container Registry).

How to push a container

Let me share a few quick steps on how you can push your Docker container to the Amazon ECR repository.

Step1: Creating a repository
The first step is to create a repository where your Docker container can be pushed to. A single repository can contain multiple versions of a docker container with a maximum of 2k versions. For different docker containers you would create individual repositories.

In order to create a repository for let’s say our test-svc docker container let’s just run this command using the AWS CLI:

aws ecr create-repository --repository-name test-svc

Please note the returned repositoryUri we will need it in the next steps.

Step2: Logging in to ECR
In order to be able to push containers via Docker, you need to login to the AWS ECR repository. You can do this by running this AWS CLI:

aws ecr get-login

This will give an output something like this:

docker login -u AWS -p password -e none https://aws_account_id.dkr.ecr.eu-west-1.amazonaws.com

You need to take that output and run it in the console to do the actual login so that you can push your container.

Step3: Pushing the container
Now that we are authenticated we can start pushing the docker container, let’s make sure that we tag the container we want to push first:

docker tag test-svc:latest aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/test-svc:latest

And after this we push the container as following:

docker push aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/test-svc:latest

Note: Please make sure to replace aws_account_id with your actual AWS account id. This repository URL with the account ID is also returned when your repository was created in Step 1.

Automating in a Jenkins job

For people that have read my other posts, I tend to automate everything via Jenkins this also includes docker container publishing to Amazon ECR. This can be quite simply done by creating a small Jenkins job using this Jenkinsfile, I ask for input to confirm publish is needed, after that input it gets published to AWS ECR:

node {
    stage 'build-test-svc'
    //this triggers the Jenkins job that builds the container
    //build 'test-svc'

    stage 'Publish containers'
    shouldPublish = input message: 'Publish Containers?', parameters: [[$class: 'ChoiceParameterDefinition', choices: 'yes\nno', description: '', name: 'Deploy']]
    if(shouldPublish == "yes") {
     echo "Publishing docker containers"
     sh "\$(aws ecr get-login)"

     sh "docker tag test-svc:latest aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/state-svc:latest"
     sh "docker push aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/test-svc:latest"
    }
}

Note: There is also a plugin in Jenkins that can publish to ECR, however up until this moment that does not support the eu-west region in AWS ECR yet and gives a login issue.

Conclusion

Hope the above helps for people that want to publish their Docker containers to AWS ECR🙂 If you have any questions do not hesitate to reach out to me via the different channels.

Deploying Kubernetes to AWS using Jenkins

Based on some of my previous posts I am quite busy creating a complete Continuous Integration pipeline using Docker and Jenkins-pipelines. For those who have read my previous blogposts I am a big fan of Kubernetes for Docker container orchestration and what I ideally want to achieve is have a full CI pipeline where even the kubernetes cluster gets deployed.

In this blog post I will detail how you can setup Jenkins to be able to deploy a Kubernetes cluster. I wanted to use cloudformation scripts as that makes most sense for a structural deployment method. It seems actually the easiest way to do this is using the kube-aws tool from core-os. The kube-aws tool is provided by core-os and can generate a cloudformation script that we can use for deploying a core-os based kubernetes cluster on AWS.

Preparing Jenkins

I am still using a Docker container to host my Jenkins installation (https://renzedevries.wordpress.com/2016/06/30/building-containers-with-docker-in-docker-and-jenkins/). If you just want to know how to use the kube-aws tool from Jenkins please skip this section🙂.

In order for me to use the kube-aws tool I need to do modify my docker Jenkins container, I need to do three things for that:

  1. Install the aws command line client
    Installing the aws-cli is actually relatively simple, I just need to make sure python & python pip are install them I can simply install the python package. These two packages are simply available from the apt-get repository, so regardless if you are running Jenkins in Docker or on a regular Ubuntu box you can install aws-cli as following:

    RUN apt-get install -qqy python
    RUN apt-get install -qqy python-pip
    RUN pip install awscli
    

  2. Install the kube-aws tool
    Next we need to install the kube-aws tool from core-os, this is a bit more tricky as there is nothing available in the default package repositories. So instead i simply download a specific release from the core-os site using wget, unpack it and then move the tool binary to /usr/local/bin

    RUN wget https://github.com/coreos/coreos-kubernetes/releases/download/v0.7.1/kube-aws-linux-amd64.tar.gz
    RUN tar zxvf kube-aws-linux-amd64.tar.gz
    RUN mv linux-amd64/kube-aws /usr/local/bin
    

3. Provide AWS keys
Because I do not want to prebake the AWS identity keys into the Jenkins image I will instead use the ability to inject these as environment variables during Docker container startup. Because I start Jenkins using a docker-compose start sequence, I can simply modify my compose file to inject the AWS identity keys, this looks as following:

jenkins:
  container_name: jenkins
  image: myjenkins:latest
  ports:
    - "8080:8080"
  volumes:
    - /Users/renarj/dev/docker/volumes/jenkins:/var/jenkins_home
    - /var/run:/var/run:rw
  environment:
    - AWS_ACCESS_KEY_ID=MY_AWS_ACCESS_KEY
    - AWS_SECRET_ACCESS_KEY=MY_AWS_ACCESS_KEY_SECRET
    - AWS_DEFAULT_REGION=eu-west-1

Putting it all together
Jenkins Dockerfile used to install all the tooling, looks as following:

from jenkinsci/jenkins:latest

USER root
RUN apt-get update -qq
RUN apt-get install -qqy apt-transport-https ca-certificates
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
RUN echo deb https://apt.dockerproject.org/repo debian-jessie main > /etc/apt/sources.list.d/docker.list
RUN apt-get update -qq
RUN apt-get install -qqy docker-engine
RUN usermod -a -G staff jenkins
RUN apt-get install -qqy python
RUN apt-get install -qqy python-pip
RUN pip install awscli
RUN wget https://github.com/coreos/coreos-kubernetes/releases/download/v0.7.1/kube-aws-linux-amd64.tar.gz
RUN tar zxvf kube-aws-linux-amd64.tar.gz
RUN mv linux-amd64/kube-aws /usr/local/bin

USER jenkins

Generating the CloudFormation script

The kube-aws tooling has relatively simple input parameters, and the tool is relatively straightforward. What the tool does is to generate a cloudformation script you can use to deploy a Kubernetes stack on AWS.

The tool has in essence three phases
1. Initialise the cluster settings
2. Render the CloudFormation templates
3. Validate and Deploy the cluster

To start we need to initialise the tool, we do this by specifying a number of things. We need to tell the name of the cluster, we need to specify what the external DNS name will be (for the kube console for example) and you need to specify the name of the key-pair to use for the nodes that will be created.

kube-aws init --cluster-name=robo-cluster \
   --external-dns-name=cluster.mydomain.com \
   --region=eu-west-1 \
   --availability-zone=eu-west-1c \
   --key-name=kubelet-key \
   --kms-key-arn='arn:aws:kms:eu-west-1:11111111:key/dfssdfsdfsfsdfsfsdfdsfsdfsdfs'

You also will need a KMS encryption key from AWS (see here how: http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html), the ARN of that encryption you need to specify in the above call to the kube-aws tool.

Rendering CloudFormation templates
The next step is actually very simple, we need to render the CloudFormation templates, we do this by simply executing the below command:

kube-aws render

This renders in the local directory a number of files, where most importantly you have the cluster.yaml which contains all the configuration settings used in the CloudFormation template. The cloudformation template is available as well under the filename stack-template.json

Setting nodeType and number of nodes
Before we deploy we need to set some typical settings in the generated cluster.yaml settings file. The main settings I want to change are the AWS instance type and the number of kubernetes worker nodes. Because I want to automate the deployment, I have chosen to do a simple in line replacement of values using sed

With the below code I change the workerInstanceType to a Environment variable called ‘INSTANCE_TYPE’ and I change the workerCount property (the amount of kube nodes) to an environment variable ‘$WORKER_COUNT’.

sed -i '''s/#workerCount: 1/workerCount: '''$WORKER_COUNT'''/''' cluster.yaml
sed -i '''s/#workerInstanceType: m3.medium/workerInstanceType: '''$INSTANCE_TYPE'''/''' cluster.yaml

The reason I use environment variables is because this way I can use Jenkins job input parameters later on to specify them. This way when deploying a cluster I can specify the values on triggering of the job.

Last step to do is to validate the changes we have made are correct, we do this as following

kube-aws validate

Deploying the Kubernetes cluster to AWS

The next and final step is to actually run the CloudFormation scripts, the kube-aws tool has a handy command for this, running this will execute the CloudFormation template. Simply run this:

kube-aws up

You are of course free to run the actual template yourself manually via the aws-cli or using the web console. I do like this handy shortcut built in the kube-aws tool for automation purposes. All the files for running this manually are available after the render and validate step described above. The output you need to use are the cluster.yaml,stack-template.json,userdata/* folder and the credentials folder.

Kubeconfig
After the kube-aws command completes it will have written a valid kubeconfig file to the local working directory. This kubeconfig can be used to access the kubernetes cluster. The cluster will take roughly an additional 5-10 minutes to be available but once it is up you can execute the regular kubernetes commands to validate your cluster is up and running:

kubectl --kubeconfig=kubeconfig get nodes

Creating a Jenkins Job

I want to put all of the above together in a single Jenkins job. For this I have created a Jenkins pipelines file where we put the entire process in as put in smaller pieces above. On top of this I introduced a multi-stage Jenkins pipeline:
1. Initialise the tool
2. Change the cluster configuration based on input parameters
3. Archive the generated cloudformation scripts
4. Deploy the cluster
5. Destroy the cluster

Input parameters
In the Jenkins pipeline I have defined multiple input parameters. These allow me to customise the worker count and instance types as described before. In order to use them you can do this two ways, you can hard code this in a freestyle job, but if you want to use the new style build pipelines in Jenkins you can use this:

   WORKER_COUNT = input message: 'Number of Nodes', parameters: [[$class: 'StringParameterDefinition', defaultValue: '4', description: '', name: 'WORKER_COUNT']]
   INSTANCE_TYPE = input message: 'Number of Nodes', parameters: [[$class: 'StringParameterDefinition', defaultValue: 't2.micro', description: '', name: 'INSTANCE_TYPE']]

For step 4 and 5 I use an additional Input step in Jenkins to check if the cluster really needs to be deployed. Also an additional fifth step is introduced to potentially allow destroying of the cluster.

The full pipeline looks like this:

node {
   stage 'Kube-aws init'
   deleteDir()

   sh "kube-aws init --cluster-name=robo-cluster \
   --external-dns-name=kube.robot.renarj.nl \
   --region=eu-west-1 \
   --availability-zone=eu-west-1c \
   --key-name=kube-key \
   --kms-key-arn='arn:aws:kms:eu-west-1:11111111:key/dfssdfsdfsfsdfsfsdfdsfsdfsdfs'"

   stage "Kube-aws render"

   WORKER_COUNT = input message: 'Number of Nodes', parameters: [[$class: 'StringParameterDefinition', defaultValue: '4', description: '', name: 'WORKER_COUNT']]
   INSTANCE_TYPE = input message: 'Number of Nodes', parameters: [[$class: 'StringParameterDefinition', defaultValue: 't2.micro', description: '', name: 'INSTANCE_TYPE']]

   sh "kube-aws render"
   sh "sed -i '''s/#workerCount: 1/workerCount: '''$WORKER_COUNT'''/''' cluster.yaml"
   sh "sed -i '''s/#workerInstanceType: m3.medium/workerInstanceType: '''$INSTANCE_TYPE'''/''' cluster.yaml"
   sh "kube-aws validate"

   stage "Archive CFN"
   step([$class: 'ArtifactArchiver', artifacts: 'cluster.yaml,stack-template.json,credentials/*,userdata/*', fingerprint: true])

   stage "Deploy Cluster"
   shouldDeploy = input message: 'Deploy Cluster?', parameters: [[$class: 'ChoiceParameterDefinition', choices: 'yes\nno', description: '', name: 'Deploy']]
   if(shouldDeploy == "yes") {
    echo "deploying Kubernetes cluster"
    sh "kube-aws up"
    step([$class: 'ArtifactArchiver', artifacts: 'kubeconfig', fingerprint: true])
   }

   stage "Destroy cluster"
   shouldDestroy = input message: 'Destroy the cluster?', parameters: [[$class: 'BooleanParameterDefinition', defaultValue: false, description: '', name: 'Destroy the cluster']]
   if(shouldDestroy) {
    sh "kube-aws destroy"
   }
}

In Jenkins you can create a new job of type ‘Pipeline’ and then simply copy this above Jenkinsfile script in the job. But you can also check this file into sourcecontrol and let Jenkins scan your repositories for the presence of these files using the Pipeline multibranch capability.

The pipeline stage-view will roughly look as following after the above:
Screen Shot 2016-07-18 at 08.30.57

Conclusion

I hope based on the above people one way of automating the deployment of a Kubernetes cluster. Please bear in mind that this is not ready for a production setup and will require more work. I hope the above demonstrates the possibilities of deploying and automating a Kubernetes cluster in a simple way. However it is still lacking significant steps like doing rolling upgrades of existing clusters, deploying parallel clusters, cluster expansion etc. etc.

But for now for my use-case, it helps me to quickly and easily deploy a Kubernetes cluster for my project experiments.