It has been a long while since I posted but recently I have been messing around with crypto and decided I needed a way to quickly gain insights in to several statisics around Crypto currencies. In specific I wanted to have different Pairs (BTC vs EUR and BTC vs ETH) in a single page. Each of these pages can be easily found on the interwebs, but I could not find a single dashboard.
Hence I was wondering if I could not run a Grafana dashboard and publish some of the output from these exchanges to this? Seems easy enough so let’s get to it and hopefully you can also get this up and running quickly!!
When I do these projects I always love to challenge myself a bit and use some technologies I am not familiar with yet and learn more about them, hence I picked Kotlin as my main language. I first drafted a high level concept Architecture on what I wanted to achieve:
Rationales for the components:
1. The reason I am going with Kafka as this later allows me to hook in additional consumers with different responsibilities and keep a clean separation of concerns.
2. For this dashboard I am using Kotlin for building the Kraken scraper and the consumer that pushes the data into InfluxDB. As mentioned I love a new challenge, have read a lot about Kotlin but never used in a production use-case, so this is a good as moment as ever to try this.
3. For storing the data for the dashboard I decided to use an old favorite of mine InfluxDB which is perfectly suited as a Time-Series DB to display the crypto values over time and allow me to do some queries on this for the dashboard.
Building the services
All of the services where written in Kotlin, this is actually where I spend most of the time as this was my first serious attempt writing something in Kotlin that needs to be more production level and can actually be deployed. I have used Maven for building and used the spotify-maven plugin for creating dockerized processes, nothing to fancy.
Framework wise I used Fuel as the HTTP client framework that helps me scrape the Kraken API and Clikt for some command line parsing that allows me to parameterise all the different configuration settings. This allows me to use Kubernetes to set all the appropriate configuration values and inject them either using the startup arguments or environment variables.
The code of this project is freely available here: https://github.com/renarj/crypto-dashboard
Deploying to AWS Kubernetes
I am using a Kubernetes cluster on AWS that is deployed using KOPS you can read about this in my previous blog post (https://renzedevries.wordpress.com/2017/02/09/deploying-a-highly-available-kubernetes-cluster-to-aws-using-kops/)
The reason I am using Kops instead of the service in AWS for Kubernetes is mainly convenience, Kops gets the cluster up and running quickly for me and I can tweak all the settings easily. The AWS K8 offering looks like more tinkering is needed, hence I decided to skip it for now.
The deployment is quite straightforward, I create a number of services for the internal components (influxdb, zookeeper and kafka). For Kafka I needed to use a bit of custom configuration specially in the listeners area to get this working. I am not an expert in this domain, and given my setup only uses a single broker with a single partition I did not find it so relevant to spend a lot time on this as scaling this out is probably not needed for this simple use-case.
In case you want to deploy this yourself, please clone this Github Repo: https://github.com/renarj/crypto-dashboard
For deploying Kafka, Zookeeper and InfluxDB (supporting services) use the following commands (in this order preferably):
kubectl apply -f k8-deployment/zookeeper-service.yaml kubectl apply -f k8-deployment/zookeeper-deployment.yaml kubectl apply -f k8-deployment/kafka-service.yaml kubectl apply -f k8-deployment/kafka-public-service.yaml kubectl apply -f k8-deployment/kafka-deployment.yaml kubectl apply -f k8-deployment/influxdb-config.yaml kubectl apply -f k8-deployment/influxdb-service.yaml kubectl apply -f k8-deployment/influxdb-deployment.yaml kubectl apply -f k8-deployment/grafana-service.yaml kubectl apply -f k8-deployment/grafana-deployment.yaml
This above should deploy all the supporting services for our Crypto dashboard, but now we need to deploy the scraper and consumer to influxdb processes. Let me put the code here inline so it is visible what configuration options there are:
The Kraken scraper has this deployment descriptor:
apiVersion: apps/v1 kind: Deployment metadata: labels: app: crypto-producer-kraken name: crypto-producer-kraken spec: replicas: 1 selector: matchLabels: app: crypto-producer-kraken template: metadata: labels: app: crypto-producer-kraken spec: containers: - env: - name: KAFKA_PRODUCER_TOPIC value: "crypto-topic" - name: KAFKA_HOST value: "kafka-public-svc" - name: KAFKA_PORT value: "30092" image: renarj/kraken-producer:1.0.004 imagePullPolicy: IfNotPresent name: crypto-producer-kraken
The consumer has the following:
apiVersion: apps/v1 kind: Deployment metadata: labels: app: crypto-subscriber name: crypto-subscriber spec: replicas: 1 selector: matchLabels: app: crypto-subscriber template: metadata: labels: app: crypto-subscriber spec: containers: - env: - name: KAFKA_CONSUMER_TOPIC value: "crypto-topic" - name: KAFKA_HOST value: "kafka-public-svc" - name: KAFKA_PORT value: "30092" - name: KAFKA_GROUP value: "crypto-tickers" - name: INFLUXDB_HOST value: "influxdb-service" - name: INFLUXDB_PORT value: "8086" - name: INFLUXDB_NAME value: "cryptoseries" image: renarj/crypto-subscriber:1.0.003 imagePullPolicy: IfNotPresent name: crypto-subscriber
Most important configuration is to configure the Kafka connectivity and which topic to produce/consume from and for the consumer the location of the InfluxDB service. In this case they connect to the internal public service on port 30092, which is a NodePort service. This can work only in this setup with kafka due to only have a single Broker in this cluster and no need to scale-out.
All the deployment descriptors and the services needed can be found in this repository and have been tested both on a Minikube setup and an AWS K8 Kops based Cluster:
Note of warning: All of the deployment descriptors are available but not hardened in anyway, recommend to run these in a shielded Kubernetes Cluster
I really hope people can use this to fine-tune and run their own crypto dashboard, all the Code and deployment descriptors are available: https://github.com/renarj/crypto-dashboard/