Bare-metal Kubernetes

A few years ago, I attended my first Linux conference, DevConf 2014. Many of the speakers talked about containers and how wonderful they were, and my interest was piqued, but I’ve never really had an opportunity to use them.

As the sysadmin for a school, there just isn’t much need for the scalability provided for by containers. Our internal web site runs on a single VM, and the short downtimes required for system updates and upgrades are not a problem, especially if I plan them for the weekends. On the flip side, having something that we can use to spin up web services quickly isn’t a bad idea, so, over the last few months, I’ve been experimenting with Kubernetes.

My main goal was to get something running that was easy to set up and Just Works™. Well, my experience setting up Kubernetes was anything but easy, but, now that it’s up, it does seem to just work. My main problem was that I wanted to use my three oVirt nodes (running CentOS 7) as both Kubernetes nodes and masters, which meant the tutorials took a bit of finessing to get working.

I mostly followed this guide for the initial setup, and then this guide, but I did run into a few problems that I’d like to document. The first was that my containers were inaccessible on their cluster IP range, the primary symptom being that the kube-dashboard service couldn’t connect to the kubernetes service. It turned out that I, rather stupidly, forgot to start kube-proxy, which does all the iptables magic to direct traffic to the correct destination.

The second problem I ran into was that I couldn’t get pretty graphs in kube-dashboard because the heapster service wouldn’t start because I hadn’t set up the cluster DNS service, kube-dns. To be fair, the instructions for doing so are pretty unclear. In the end, I downloaded skydns-rc.yaml.sed and skydns-svc.yaml.sed, and replaced $DNS_DOMAIN and $DNS_SERVER_IP with the values I wanted to use.

The final problem I ran into is that I’m using our school’s local Certificate Authority for all the certificates we use, and I’ve had to keep on adding new subject alternative names to the server cert and then regenerate it. At the moment, it’s got the following:
# Where I replaced $DNS_DOMAIN with “local”
IP Address:
IP Address:
# Where our cluster IP range is

I suspect I could now get rid of some of those hostnames/addresses, and I’m not even sure if this method is best practice, but at least it’s all working.

So I’m at the point now where I need to see if I can setup our MikroTik router as a load balancer and then see if I can get our web based marking system, LESSON, moved over to a container with multiple replicas. Hurrah for redundancy!

Broken Wagon Wheel by Kevin Casper is in the public domain / used under a CC0 license


5 thoughts on “Bare-metal Kubernetes

  1. Pingback: Bare-metal #Kubernetes… | Dr. Roy Schestowitz (罗伊)

  2. Pingback: Links 1/5/2017: Linux 4.11, Linux Mint 18.2 Plans | Techrights

    1. jdieter Post author

      Yes, though I haven’t done much with it or Kubernetes since then. I was running into trouble with the CRS models. They had configuration options that supposedly allowed them to be configured as a load balancer, but I never managed to get them working. We then swapped in an RB3011, and it did work correctly.

      1. Lars Solberg

        I am thinking about using k8s on openstack instaed, so I can use LoadBalancer. But I found a way to do this using mikrotik as well. Kinda.. At least the “HA” part.
        Since NodePort are routed to the service no matter which node you are hitting, you can have a little scheduled script on the mikrotik to change the to-address based on which one is up. I think something like this should do the trick, at least for my part:

        :global changeK8SdefaultIP do={
        /ip firewall nat
        :foreach entry in=[find comment=”k8s-autoip”] do={
        :local currentip [get $entry to-addresses]
        :if ($currentip!=$1) do={
        set $entry to-addresses=$1

        :if ([/ping count=1]=1) do={
        } else={
        :if ([/ping count=1]=1) do={
        } else={
        :if ([/ping count=1]=1) do={

        Then I can use the “k8s-autoip” comment on the dstnat rules I need this on.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s