This is a guest post by Lili Cosic, a Software Engineer at Kinvolk where she works on various Linux technologies, currently on the Habitat Operator. She has a passion for distributed systems and Kubernetes.

It was originally published at https://kinvolk.io/blog/2017/10/running-kubernetes-on-travis-ci-with-minikube/.

Running Kubernetes on Travis CI with minikube

It is not easily possible to run Kubernetes on Travis CI, as most methods of setting up a cluster need to create resources on AWS, or another cloud provider. And setting up VMs is also not possible as Travis CI doesn’t allow nested virtualization. This post explains how to use minikube without additional resources, with a few simple steps.

As Kinvolk is currently working with Chef on a project to integrate Habitat with Kubernetes(Habitat Operator), we needed a way to run the end-to-end tests on every pull request. Locally we use minikube, a tool to set up a local one-node Kubernetes cluster for development; or when we need a multi- node cluster, kube-spawn. But for automated CI tests we only currently require a single node setup. So we decided to use minikube to be able to easily catch any failed tests and debug and reproduce those locally.

Typically minikube requires a virtual machine to set up Kubernetes. One day, a colleague shared this tweet in our Slack.

TIL minikube has an undocumented driver --vm-driver=none
Doesn't use a VM, just containers. Very nice

— Justin Garrison (@rothgar) August 1, 2017

It seems that minikube has a not-so-well-documented way of running Kubernetes with no need for virtualization as it sets up localkube, a single binary for kubernetes that is executed in a Docker container and Travis CI already has Docker support. There is a warning against running this locally, but since we only use it on Travis CI, in an ephemeral environment, we concluded that this is an acceptable use case.

So this is what our setup looks like. Following is the example .travis.yml file:

sudo: required

env:
- CHANGE_MINIKUBE_NONE_USER=true

before_script:
- curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/v1.7.0/bin/linux/amd64/kubectl && \
    chmod +x kubectl && sudo mv kubectl /usr/local/bin/
- curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && \
    chmod +x minikube && sudo mv minikube /usr/local/bin/
- sudo minikube start --vm-driver=none --kubernetes-version=v1.7.0
- minikube update-context
- JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}'; \
    until kubectl get nodes -o jsonpath="$JSONPATH" 2>&1 | grep -q "Ready=True"; do sleep 1; done

Here’s how it works.

First, it installs kubectl, which is a requirement of minikube. The need for sudo: required comes from minikube’s starting processes, which requires to be root. Having set the enviorment variable CHANGE_MINIKUBE_NONE_USER, minikube will automatically move config files to the appropriate place as well as adjust the permissions respectively. When using the none driver, the kubectl config and credentials generated will be owned by root and will appear in the root user’s home directory. The none driver then does the heavy lifting of setting up localkube on the host. Then the kubeconfig is updated with minikube update-context. And lastly we wait for Kubernetes to be up and ready.

Learning more

This work is already being used in the Habitat Operator. For a simple live example setup have a look at https://github.com/LiliC/travis-minikube. If you have any questions feel free to ping me on twitter @LiliCosic.