Autoscaling Containers on Kubernetes on AWS

One of the challenges I faced recently was the ability to autoscale my containers on my Kubernetes cluster. I realised I had not written yet about this concept and thought I would share how this can be done and what the pitfalls there were for me.

If you combine this concept with my previous post about autoscaling your kube cluster (https://renzedevries.wordpress.com/2017/01/10/autoscaling-your-kubernetes-cluster-on-aws/) you can create a very nice and balanced scalable deployment at lower costs.

Preparing your cluster

In my case I have used Kubernetes KOPS to create my cluster in AWS. However by default this does not install some of the add-ons we need for autoscaling our workloads like Heapster.

Heapster monitors and analyses the resource usage in our cluster. These metrics it monitors are very important to build scaling rules, it allows us for example to scale based on a cpu percentage. Heapster records these metrics and offers an API to Kubernetes so it can act based on this data.

In order to deploy heapster I used the following command:

kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/monitoring-standalone/v1.3.0.yaml

Please note that in your own kubernetes setup you might already have heapster or want to run a different version.

Optional dashboard
I also find it handy to run the Kubernetes dashboard, which you can deploy as following under KOPS:

kubectl create -f https://raw.githubusercontent.com/kubernetes/kops/master/addons/kubernetes-dashboard/v1.5.0.yaml

Deploying Workload

In order to get started I will deploy a simple workload, in this case its the command service for my robotics framework (see previous posts). This is a simple HTTP REST endpoint that takes in JSON data and passes this along to a message queue.

This is the descriptor of the deployment object for Kubernetes:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: command-svc
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: command-svc
    spec:
      containers:
      - name: command-svc
        image: ecr.com/command-svc:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 10
        env:
        - name: amq_host
          value: amq
        - name: SPRING_PROFILES_ACTIVE
          value: production

Readyness and Liveness
I have added a liveness and readyness probe to the container, this allows Kubernetes to detect when a container is ready and if its still alive. This is important in autoscaling as otherwise you might get pods already enabled in your loadbalanced service that are not actually ready to accept work. This is because Kubernetes by default can only detect if a pod has started, not if the process in the pod is ready for accepting workloads.

These probes test if a certain condition is true and only then the pod will get added to the load balanced service. In my case I have a probe to check if the port 8080 of my rest service is available. I am using a simple TCP probe as the HTTP probe that is also offered gave strange errors and the TCP probe works just as well for my purpose.

Deploying
Now we are ready to deploy the workload and we deploy this as following:

kubectl create -f command-deployment.yaml


## Enabling Autoscaling
The next step is to enable autoscaling rules on our workload, as mentioned above we have deployed heapster which can monitor resource usage. In this case I have set some resource constraints for the pods to indicate how much CPU its allowed to consume. For the command-svc per pod we have a limit of 500m, which translates to roughly 0.5 CPU core. This means if we create a rule to scale at 80 cpu usage this is based on this limit, so it will scale 80% usage of the 0.5 CPU limit.

We can create a rule that says there is always minimum of 1 pod and a maximum of 3 and we scale-up once the cpu usage exceeds 80% of the pod limit.

kubectl autoscale deployment command-svc --cpu-percent=80 --min=1 --max=3

We can ask for information on the autoscaling with the following command and monitor the scaling changes:

kubectl get hpa -w

Creating a load

I have deployed the command-svc pod and want to simulate a load using a simple tool. For this I have simple resorted to Apache JMeter, its not a perfect tool but it works well and most important its free. I have created a simple thread group with 40 users doing 100k requests against the command-svc from my desktop.

This is the result when monitoring the autoscaler:

command-svc   Deployment/command-svc   1% / 80%   1         3         1          4m
command-svc   Deployment/command-svc   39% / 80%   1         3         1         6m
command-svc   Deployment/command-svc   130% / 80%   1         3         1         7m
command-svc   Deployment/command-svc   130% / 80%   1         3         1         7m
command-svc   Deployment/command-svc   130% / 80%   1         3         2         7m
command-svc   Deployment/command-svc   199% / 80%   1         3         2         8m
command-svc   Deployment/command-svc   183% / 80%   1         3         2         9m
command-svc   Deployment/command-svc   153% / 80%   1         3         2         10m
command-svc   Deployment/command-svc   76% / 80%   1         3         2         11m
command-svc   Deployment/command-svc   64% / 80%   1         3         2         12m
command-svc   Deployment/command-svc   67% / 80%   1         3         2         13m
command-svc   Deployment/command-svc   91% / 80%   1         3         2         14m
command-svc   Deployment/command-svc   91% / 80%   1         3         2         14m
command-svc   Deployment/command-svc   91% / 80%   1         3         3         14m
command-svc   Deployment/command-svc   130% / 80%   1         3         3         15m
command-svc   Deployment/command-svc   133% / 80%   1         3         3         16m
command-svc   Deployment/command-svc   130% / 80%   1         3         3         17m
command-svc   Deployment/command-svc   126% / 80%   1         3         3         18m
command-svc   Deployment/command-svc   118% / 80%   1         3         3         19m
command-svc   Deployment/command-svc   137% / 80%   1         3         3         20m
command-svc   Deployment/command-svc   82% / 80%   1         3         3         21m
command-svc   Deployment/command-svc   0% / 80%   1         3         3         22m
command-svc   Deployment/command-svc   0% / 80%   1         3         3         22m
command-svc   Deployment/command-svc   0% / 80%   1         3         1         22m

You can also see that it neatly scales down at the end once the load goes away again.

Pitfalls

I have noticed a few things about the autoscaling that are important to take into account:
1. The CPU percentage is based on the resource limits you define in your pods, if you don't define them it won't work as expected
2. Make sure to have readyness and liveness probes in your container else your pods might not be ready but already get hit with external requests
3. I could only have probes that use TCP for some reason in AWS, unsure why this is the case, HTTP probes failed for me with timeout exceptions.

Conclusion

I hope this post helps people get the ultimate autoscaling setup of both your workloads and also your cluster. This is a very powerfull and dynamic setup on AWS in combination with the cluster autoscaler as described in my previous post: https://renzedevries.wordpress.com/2017/01/10/autoscaling-your-kubernetes-cluster-on-aws/

Advertisements

Deploying a Highly Available Kubernetes cluster to AWS using KOPS

In my previous posts I have talked a lot about deploying a kubernetes cluster. For most part I have used kube-aws from CoreOS which has served me quite well. In the last few months however a lot has happened in the Kubernetes space and a new tool has started becoming very interest called Kops which is a subproject of the kubernetes project.

Both the kube-aws tool and Kops tool started getting support for HA deployments (a requirement for production workloads) and cluster upgrades. One of the major advantages of the Kops tool is its ability to manage and maintain multiple clusters because its stores the ‘cluster state’ in an s3 bucket.

In this post I will do a walkthrough on how to deploy a Highly available cluster using Kops. I will base this on the tutorial page here: https://github.com/kubernetes/kops/blob/master/docs/aws.md The main difference I will describe deploying a HA deployment instead of a regular deployment.

Installing the toolset

Pretty much most of this is covered in the tutorial here(https://github.com/kubernetes/kops/blob/master/docs/aws.md). I am using a mac and will use ‘brew install’ to get the needed command line tools installed. You need both the AWS command line client(aws-cli), Kubernetes client (kubectl) and Kops client installed.

Install the clients:
brew install awscli
brew install kubernetes-cli
brew install kops

Once these tools are installed please make sure to configure aws-cli, you can see here how: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials

Creating the cluster

First we need to set an environment variable to the s3 bucket where we are going to keep the Kops state of all the deployed clusters. We can do this by simply setting an environment variable as following:

export KOPS_STATE_STORE=s3://my-kops-bucket-that-is-a-secret

This bucket is needed because the Kops tool maintains the cluster state in this s3 bucket. This means we can get an overview of all deployed clusters using the Kops tool and it will query the filestructure on the s3 bucket.

Once the s3 bucket is created and variable is set we can go ahead creating the cluster as following:

kops create cluster --name=dev.robot.mydomain.com --master-zones=eu-west-1a,eu-west-1b,eu-west-1c --zones=eu-west-1a,eu-west-1b,eu-west-1c --node-size=t2.micro --node-count=5

The parameters are relatively self explanatory, it is however important that the name includes the fully qualified domain name. Kops will try to register the subdomain in the route53 hosted zone. The most important parameters that makes the setup HA is to specify multiple availability zones, for each zone it will deploy one master node and it will spread the worker nodes across the specified zones as well.

The above command has created a cluster configuration that is now stored in the s3 bucket, however the actual cluster is not yet launched. You can further edit the cluster configuration as following:

kops edit cluster dev.robot.mydomain.com

Launching the cluster
After you have finished editing we can Launch the cluster:

kops update cluster dev.robot.mydomain.com --yes

This will take a bit to complete, but after a while you should see roughly the following list of running EC2 instances, where we can see the nodes running in different availability zones:

master-eu-west-1c.masters.dev.robot.mydomain.com	eu-west-1c	m3.medium	running
master-eu-west-1b.masters.dev.robot.mydomain.com	eu-west-1b	m3.medium	running
master-eu-west-1a.masters.dev.robot.mydomain.com	eu-west-1a	m3.medium	running
nodes.dev.robot.mydomain.com	eu-west-1a	t2.micro	running
nodes.dev.robot.mydomain.com	eu-west-1b	t2.micro	running
nodes.dev.robot.mydomain.com	eu-west-1b	t2.micro	running
nodes.dev.robot.mydomain.com	eu-west-1c	t2.micro	running
nodes.dev.robot.mydomain.com	eu-west-1c	t2.micro	running

Kops will also have sorted out your kubectl configuration so that we can ask for all available nodes as below:

kubectl get nodes
NAME                                           STATUS         AGE
ip-172-20-112-108.eu-west-1.compute.internal   Ready,master   8m
ip-172-20-114-138.eu-west-1.compute.internal   Ready          6m
ip-172-20-126-52.eu-west-1.compute.internal    Ready          7m
ip-172-20-56-106.eu-west-1.compute.internal    Ready,master   7m
ip-172-20-58-2.eu-west-1.compute.internal      Ready          7m
ip-172-20-69-113.eu-west-1.compute.internal    Ready          7m
ip-172-20-75-48.eu-west-1.compute.internal     Ready,master   8m
ip-172-20-86-155.eu-west-1.compute.internal    Ready          7m

Chaos monkey

I have tried actually killing some of the Master nodes and seeing if I could still schedule a load. The problem I faced here was that the cluster could still operate and the containers remained available, but I could not schedule new workloads. This was due to the ‘etcd’ cluster being deployed as part of the master nodes and suddenly the minimum number of nodes for the etcd cluster was no longer present. Most likely moving etcd out of the master nodes would increase the reliability further.

The good news is that once the master nodes recovered from the unexpected termination the cluster resumed regular operation.

Conclusion

I hope that above shows that it is now relatively easy to setup a HA Kubernetes cluster. In practice its quite handy to have HA cluster, next step is to move out etcd to make the solution even more resilient.

AutoScaling your Kubernetes cluster on AWS

One of the challenges I have faced in the last few months is the autoscaling of my Kubernetes cluster. This is perfectly working on Google Cloud, however as my cluster is deployed on AWS I have no such fortune. However since recently the autoscaling support for AWS has been made possible due to this little contribution that was made: https://github.com/kubernetes/contrib/tree/master/cluster-autoscaler/cloudprovider/aws

In this post I want to describe how you can autoscale your kubernetes cluster based on the container workload. This is a very important principle because normal autoscaling by AWS can not do this on metrics available to the cluster, it can only do it on for example memory or cpu utilisation. What we want is that in case a container cannot be scheduled on a cluster because there are not enough resources the cluster gets scaled out.

Defining resource usage

One of the very important things to do when you are defining your deployments against Kubernetes is to define the resource usage of your containers. In each deployment object it is therefore imperative that you specify the resource allocation that is expected. Kubernetes will then use this to allocate this to a node that has enough capacity. In this exercise we will deploy two containers that both at their limit require 1.5GB of memory, this looks as following in fragment of one of the deployment descriptors:

    spec:
      containers:
      - name: amq
        image: rmohr/activemq:latest
        resources:
          limits:
            memory: "1500Mi"
            cpu: "500m"

Setting up scaling

So given this we will start out with a cluster of one node of type m3.medium which has 3.75GB of memory, we do this on purpose with a limited initial cluster to test out our autoscaling.

If you execute kubectl get nodes we see the following response:

NAME                                       STATUS    AGE
ip-10-0-0-236.eu-west-1.compute.internal   Ready     10m

In order to apply autoscaling we need to deploy a specific deployment object and container that checks the Kubernetes Cluster for unscheduled workloads and if needed will trigger an AWS autoscale group. This deployment object looks as following:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: cluster-autoscaler
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      containers:
        - image: gcr.io/google_containers/cluster-autoscaler:v0.4.0
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --nodes=MIN_SCALE:MAX_SCALE:ASG_NAME
          env:
            - name: AWS_REGION
              value: us-east-1
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs/ca-certificates.crt
              readOnly: true
          imagePullPolicy: "Always"
      volumes:
        - name: ssl-certs
          hostPath:
            path: "/etc/ssl/certs/ca-certificates.crt"

Note: The image we are using is a google supplied image with the autoscaler script on there, you can check here for the latest version: https://console.cloud.google.com/kubernetes/images/tags/cluster-autoscaler?location=GLOBAL&project=google-containers&pli=1

Settings
In the above deployment object ensure to replace the MIN_SCALE and MAX_SCALE settings for the autoscaling and ensure the right autoscaling group (ASG_NAME) is set. Please note that the minimum and maximum scaling rule need to be allowed in the AWS scaling group as the scaling process cannot modify the auto scaling group rule itself.

AWS Policy
In AWS we need to ensure there is an IAM policy in place that allows all resources to query the auto scaling groups and modify the desired capacity of the group. I have used the below role definition, which is very wide:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
    ]
}

Make sure to attach this to the role that is related to the worked nodes, in my core-os kube-aws generated cluster this is something like ‘testcluster-IAMRoleController-GELKOS5QWHRU’ where testcluster is my clustername.

Deploying the autoscaler
Now let’s deploy the autoscaler just like any other deployment object against the Kubernetes cluster

kubectl create -f autoscaler.yaml

Let’s check next that the autoscaler is working:

➜  test kubectl get po
NAME                                  READY     STATUS    RESTARTS   AGE
cluster-autoscaler-2111878067-nhr01   1/1       Running   0          2m

We can also check the logs using kubectl logs cluster-autoscaler-2111878067-nhr01

No unschedulable pods
Scale down status: unneededOnly=true lastScaleUpTime=2017-01-06 08:44:01.400735149 +0000 UTC lastScaleDownFailedTrail=2017-01-06 08:44:01.400735354 +0000 UTC schedulablePodsPresent=false
Calculating unneded nodes
Node ip-10-0-0-135.eu-west-1.compute.internal - utilization 0.780000
Node ip-10-0-0-135.eu-west-1.compute.internal is not suitable for removal - utilization to big (0.780000)
Node ip-10-0-0-236.eu-west-1.compute.internal - utilization 0.954000
Node ip-10-0-0-236.eu-west-1.compute.internal is not suitable for removal - utilization to big (0.954000)

We can see the autoscaler checks regulary the workload on the nodes and check if they can be scaled down and check if additional worker nodes are needed.

Let’s try it out

Now that we have deployed our autoscaling container let’s start to schedule our workload against AWS. In this case we will deploy two objects, being ActiveMQ and Cassandra where both require a 1.5GB memory footprint. The combined deployment plus the system containers will cause the scheduler of Kubernetes to determine there is no capacity available, and in this case Cassandra cannot be scheduled as can be seen in below snippet from kubectl describe po/cassandra-2599509460-g3jzt:

FailedScheduling	pod (cassandra-2599509460-g3jzt) failed to fit in any node

When we check in the logs of the autoscaler we can see the below:

Estimated 1 nodes needed in testcluster-AutoScaleWorker-19KN6Y4AR18Z
Scale-up: setting group testcluster-AutoScaleWorker-19KN6Y4AR18Z size to 2
Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"cassandra-2599509460-mt275", UID:"af53ac7b-d3ec-11e6-bd28-0add02d2d0c1", APIVersion:"v1", ResourceVersion:"2224", FieldPath:""}): type: 'Normal' reason: 'TriggeredScaleUp' pod triggered scale-up, group: testcluster-AutoScaleWorker-19KN6Y4AR18Z, sizes (current/new): 1/2

It is scheduling an additional worker by increasing the desired capacity of our auto scaling group in AWS. After a small wait we can see the additional node has been made available:

NAME                                       STATUS    AGE
ip-10-0-0-236.eu-west-1.compute.internal   Ready     27m
ip-10-0-0-68.eu-west-1.compute.internal    Ready     11s

And a short while after the node came up we also see that the pod with Cassandra has become active:

NAME                                  READY     STATUS    RESTARTS   AGE
amq-4187240469-hlkhh                  1/1       Running   0          20m
cassandra-2599509460-mt275            1/1       Running   0          8m
cluster-autoscaler-2111878067-nhr01   1/1       Running   0          11m

Conclusion

We have been able to autoscale our AWS deployed Kubernetes cluster which is extremely useful. I can use this in production to quickly scale out and down my cluster. But perhaps even more important for my case in development i can use it to during idle moments run a minimum size cluster and during workloads it scales back up to full capacity, saving me quite some money.

Remote Controlling Nao robot using a Raspberry Pi Robot

Today I want to take some time to write about the next step I am currently taking to have both my self-build Raspberry PI robot and the Nao robot interact with each other on a useful basis. You might have already seen some posts before like https://renzedevries.wordpress.com/2016/06/10/robot-interaction-with-a-nao-robot-and-a-raspberry-pi-robot/ about robot interaction or perhaps the model train one https://renzedevries.wordpress.com/2016/09/13/having-fun-with-robots-and-model-trains/. However both these posts did not really demonstrate a practical use-case.

Recently I presented about this topic at the Devoxx conference in Antwerp where I attempt to demonstrate how to control one robot from another using Kubernetes, Helm and Minikube combined with some IoT glue 🙂 The scenario I demonstrated was to create a Robotic Arm from my Raspberry PI robot that I use to remote control a Nao robot.

Robot arm
In order to have some form of remote control I have created a Robot Arm which i can use as a sort of joystick. I have created the robot from the same parts as described in this post (https://renzedevries.wordpress.com/2016/03/31/build-a-raspberry-pi-robot-part-2/). The robot arm is controller via a Raspberry PI that has a bit of Java software to connect it to MQTT to send servo position changes and to receive commands from MQTT to execute motions on the robot arm.

The robot looks like this:
67d466ea-019e-4216-a1e0-4d577bf7038e

Nao Robot
For the Nao robot I have written a customer Java controller that connects to the remote API of Nao. This controller software does nothing else but allowing remote control of the Nao robot by listening to commands coming from MQTT.

Connecting the Robots

Like before in previous setups I will be using my custom Robot Cloud deployment setup for this experiment. I will be deploying a number of micro-services to a Kubernetes cluster that is running on AWS. The most important public services are the MQTT message bus which is where the robots are sending status (sensors/servo’s) towards and received commands from (animations, walk commands etc.). For more detail on the actual services and their deployment you can check here https://renzedevries.wordpress.com/2016/06/10/robot-interaction-with-a-nao-robot-and-a-raspberry-pi-robot/

The most important part of bridging the gap between the robots is to have a specific container that receives updates from the servo’s on the robot arm. Based on events from those servo’s (move the joystick forward) I want to trigger the Nao robot to start walking. The full code with a lot more detail is available in this git repository: https://github.com/renarj/robo-bridge

Conclusion

It’s quite a complex setup, but the conclusion is that by using my Kubernetes deployed Robot Cloud stack I can use the robot Arm to control the Nao robot. If you want to see a bit more with a live demo you can check out my Devoxx presentation here:

One thing I could not demo at Devoxx was the interaction with a real Nao Robot, I have made a recording how that would look and also put this on youtube here:

Deployment using Kubernetes Helm

One of the challenges I face in my development setup is that I want to quickly and often create and deploy my robotics stack. I often want to change and redeploy my entire stack from scratch, because I want to iterate quickly and also reduce my costs as much as possible. My Jenkins jobs have helped a great deal here, and automation is definitely key. However I have recently started experimenting with Kubernetes Helm which is a package manager for Kubernetes which has made this even easier for me.

Kubernetes Helm

Helm is a package manager that allows you to define a package with all its dependent deployment objects for Kubernetes. With helm and this package you can then ask a cluster to install the entire package in one go instead of passing individual deployment commands. This means for me that instead of asking Kubernetes to install each of my several micro-services to be installed I simply ask it to install the entire package/release in one atomic action which also includes all of the dependent services like databases and message brokers I use.

Installing Helm

In this blog I want to give a small taste on how nice Helm is. So how do we get started? Well in order to get started with Helm you should first follow the installation instructions at this page: https://github.com/kubernetes/helm/blob/master/docs/install.md

In case you are using OSX (like me) its relatively simple if you are using homebrew, simply run the following cask:

brew cask install helm

Once helm is installed it should also be installed in your cluster. In my case I will be testing against a minikube installation as described in my previous post: https://renzedevries.wordpress.com/2016/11/07/using-kubernetes-minikube-for-local-test-deployments/

On the command line I have a kubernetes command line client (kubectl) with my configuration pointing towards my minikube cluster. The only thing I have to do is the following to install Helm in my cluster:

helm init

This will install a container named tiller in my cluster, this container will understand how to deploy the Helm packages (charts) into my cluster. This is in essence the main endpoint the helm client will use to interrogate the cluster for package deployments and package changes.

Creating the package

Next we need to start creating something which is called a Chart, this is the unit of packaging in Helm. For this post I will reduce the set of services I have used in previous posts and only deploy the core services Cassandra, MQTT and ActiveMQ. The first thing to define is the *Chart.yaml** which is the package manifest:

Chart.yaml
The manifest looks pretty simple, most important is the version number, the rest is mainly metadata for indexing:

name: robotics
version: 0.1
description: Robotic automation stack
keywords:
- robotics
- application
maintainers:
- name: Renze de Vries
engine: gotpl

The second I am going to define is the deployment objects I want to deploy. For this we create a ‘Charts’ subdirectory which contains these dependent services. In this case I am going to deploy MQTT, ActiveMQ and Cassandra which are required for my project. For each of these services I create a templates folder which contains the Kubernetes Deployment.yaml descriptor and Kubernetes service descriptor file and have their own Charts.yaml file as well.

When you have this all ready it look as following:
screen-shot-2016-11-16-at-20-42-12

I am not going to write out all the files in this blog, if you want to have a look at the full source have a look at the github repository here that contains the full Helm chart structure describe in this post: https://github.com/renarj/helm-repo

Packaging a release

Now that the Chart source files have been created the last thing to do is to create the actual package. For this we have to do nothing else than simply run the following command:

helm package .

This will create a file called robotics-0.1.tgz that we can use further to deploy our release. In a future blog post I will talk a bit about Helm repositories and how you can distribute these packages, but for now we keep them on the local file system.

Installing a release

Once we have defined the packages the only thing thats remaining is to simply install a release into the cluster. This will install all the services that are packaged in the Chart.

In order to install the package we have created above we just have to run the following command:

helm install robotics-0.1.tgz
NAME: washing-tuatar
LAST DEPLOYED: Sun Nov  6 20:42:30 2016
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME          CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
amq       10.0.0.59   <nodes>   61616/TCP   1s
mqtt      10.0.0.131   <nodes>   1883/TCP   1s
cassandra-svc   10.0.0.119   <nodes>   9042/TCP,9160/TCP   1s

==> extensions/Deployment
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
mqtt      1         1         1            0           1s
amq       1         1         1         0         1s
cassandra   1         1         1         0         1s

We can ask Helm which packages are installed in the cluster by simply asking a list of installed packages as following:

helm list
NAME          	REVISION	UPDATED                 	STATUS  	CHART       
washing-tuatar	1       	Sun Nov  6 20:42:30 2016	DEPLOYED	robotics-0.1

Please note that the name for the installation is a random generated name, in case you want a well known name you can install using the ‘-name’ switch and specify the name yourself.

In order to delete all the deployed objects I can simply ask Helm to uninstall the release as following:

helm delete washing-tuatar

Conclusion

I have found that Helm has a big potential, it allows me to very quickly define a full software solution composed out of many individual deployments. In a future blog post I will talk a bit more about the templating capabilities of Helm and the packaging and distributing of your packages. In the end I hope this blog shows everyone that with Helm you can make all of your Kubernetes work even easier than it already is today 🙂

Using Kubernetes Minikube for Local test deployments

One of the many challenges I face with my Robotics Cloud development is the need to test locally and constantly re-create the stack from scratch. Now I have a lot of automation to deploy against AWS using Jenkins as seen in previous posts. However setting up a local development environment is the thing I do the most and that is costing a lot of time because the tooling always was painful to use.

Now in the last few months there have been a lot of innovations happening in the Kubernetes field. In particular in this blog post I want to talk about using Minikube.

Minikube

One of the big pains was always to setup a Kubernetes cluster on your local machine. Before there were some solutions, the simplest one was to use vagrant or the kube-up script that would create some vm’s in virtualbox. However my experience was that they were error prone and did not always complete succesfully. For local machine development setups there is now a new solution called minikube. In essence using minikube you can create a single machine kubernetes test cluster to get you quickly up and running.

The simplest way to get started is to install minikube first using the latest release instructions, in my case for OSX on the 0.12.2 release I install it using this command:

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.12.2/minikube-darwin-amd64 &amp;&amp; chmod +x minikube &amp;&amp; sudo mv minikube /usr/local/bin/

Please visit this page for the latest release of minikube: https://github.com/kubernetes/minikube/releases

In essence the above command downloads the minikube binary and moves it to the local usr bin directory so its available on the path. After this we can start creating the minikube machine, in my case I will use virtualbox as the provider which is automatically detected if its installed. In my case all i have to do is the following:
minikube start --memory=8196

The above will start a single node kubernetes cluster which acts both as master and worker in virtualbox with 8GB of memory. Also it will ensure my local kubernetes (kubectl) client configuration is set to point to the cluster master. This will take a few minutes to get up and running but the cluster should be available after this and you can check if its ready by doing this:

kubectl get nodes
NAME       STATUS    AGE
minikube   Ready     1h

The minikube setup has created a virtual box setup that exposes all its services via the virtualbox ip. The minikube binary provides a shortcut to get that ip using below command:

minikube ip
192.168.99.100

This ip can be used to directly access all services that are exposed on the kubernetes cluster.

Now the cluster is available you can start deploying to your hearts content, but you might want to use the kubernetes dashboard for this which is handy for the overview. In order to quickly get to the dashboard you can run this minikube command:

minikube dashboard

If you want to stop the cluster you can simply type the following command:

minikube stop

The next time you start the cluster it will resume the state it was in previously. So all previously running containers will also be started once the cluster comes back up which is quite handy in case of development.

Conclusion

I hope this post helps people who are struggling setting up their own Kubernetes cluster and getting them quickly started. I am sure there is a lot more to come from the Kubernetes folks, its really getting easier and easier 🙂

Versioning and deploying Docker containers using Kubernetes and Jenkins

In the last few posts the main comment that keeps coming back is ‘you should not use latest’. I totally agree on that and in this blog I will finally do something about it 🙂 I mainly used the setup for development purpose, however in production I need something more reliable, meaning versioned deployments.

To do this there are of course two parts, first I have to actually create a build job that can version the containers and second I will need a build job that rolls out the update deployed container.

Versioning and Releasing

Using the below Jenkins pipeline I have a build job that uses a Jenkins input parameter that defines the release version. In order to use this create a Jenkins pipeline job that has one input parameter called ‘RELEASE_VERSION’. The default value I keep to ‘dev-latest’ for development purposes, but when releasing we obviously need to use a sensible number.

node {
    stage 'build'
    build 'home projects/command-svc/master'

    stage 'test-deploy'
    sh "\$(aws ecr get-login)"
    sh "docker tag home-core/command-svc:latest aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:dev-latest"
    sh "docker push aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:dev-latest"    

    stage 'qa'
    build 'home projects/command-svc-tests/master'

    stage 'Publish containers'
    sh "docker tag home-core/command-svc:latest aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:'$RELEASE_VERSION'"
    sh "docker push aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:'$RELEASE_VERSION'"
}

Now when I trigger the build Job, Jenkins will ask me to input the version for releasing that container. So let’s in this article use the version ‘0.0.1’. When running the build job, Jenkins will go through a few stages.
1. Building the container
2. Pushing a dev-latest version
3. Running tests which deploy a container against the test cluster
4. Release the container using a fixed version.

In the latest stage I release the container using the input parameter specified on the job triggering. The build job does not actually deploy to production, that is for the time being still a manual action.

Deploying to Kubernetes

For deploying to production I have done an initial deployment of the service using below deployment descriptor. I started out deploying version ‘0.0.1’ which I have built above.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: command-svc
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: command-svc
    spec:
      containers:
      - name: command-svc
        image: aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:0.0.1
        ports:
        - containerPort: 8080
        env:
        - name: amq_host
          value: amq
        - name: SPRING_PROFILES_ACTIVE
          value: production

Doing a rolling update

This works fine for an initial deployment, however if i want to do an upgrade of my production containers I need something more. In production in essence I just want to upgrade the container image version. This is a relatively simple operation with Kubernetes. Let’s assume i have released a newer version of the container with version ‘0.0.2’.

In order to update the container in Kubernetes I can simply do a rolling update by changing the image of the Deployment object in Kubernetes as following:

kubectl set image deployment/command-svc command-svc=aws_account_id.dkr.ecr.eu-west-1.amazonaws.com/command-svc:0.0.2

I am currently not yet integrating this into my build pipeline as I want a production upgrade to be a conscious decision still. But once all the quality gates are in place, there should be no reason to not automate the above step as well. More on this in some future blog posts.