Skip to content

Install Kubernetes Cluster Behind Tailscale VPN

TL;DR

In this post we will install a multi-master Kubernetes cluster behind Tailscale VPN. This scenario can be useful when:

  • Your Kubernetes nodes are not in the same subnet.
  • You are building a home-lab system, and the nodes are behind two or more NAT-ted network, or even behind CGNAT.
  • Your nodes are running in separate data centers, and don't want to publish API ports on the public internet.
  • You want to access your cluster only from private VPN network.
  • You want extra security by encrypted connection between nodes.
  • Or the mixture of above scenarios.

Why Tailscale VPN?

You can use any other VPN solution like Wireguard, OpenVPN, IPSec, etc. But nowadays I think Tailscale is the easiest way to bring up a VPN network. With a free registration you get 100 device, subnet routers, exit nodes, (Magic)DNS, and so many useful features.

For more information check the following links:

But as I mentioned you can use any other VPN solution, personally I'm using Wireguard in my home-lab system.

Warning

Tailscale assigns IP address from 100.64.0.0/10 range! IP Address Assignment If you are planning to use Kube-OVN networking don't forget to change the CIDR, because Kube-OVN is also use this subnet!

Infrastructure

As I mentioned we will deploy a multi-master Kubernetes cluster:

  • 3 master|worker nodes, without worker nodes. Later additional worker nodes can be added to the cluster, but for the simplicity we won't deploy extra worker nodes.
  • We need an additional TCP load balancer for the API requests. I prefer HAProxy for this purpose, because it is easy to set up and lightweight.
    • For this lab I will deploy only one Load Balancer, but if you need HA solution, at least two Load Balancers are needed. This can be achieved by using Keppalived. Or you can use external load balancer like F5. But this demo is not about HA Load balancers, so it is just enough to have only one LB.
Hostname Role IP Address VPN IP Address
kube02-m1 Control Plane Node 1 172.16.1.77 Later
kube02-m2 Control Plane Node 2 172.16.1.78 Later
kube02-m3 Control Plane Node 3 172.16.1.79 Later
kube02-haproxy HAProxy Load Balancer 172.16.1.80 Later
ansible Ansible Host 172.16.0.252 ---

Note

You don't need the additional Ansible host, if you preparing the OS manually.

Note

You can use one of the kubernetes node for HAProxy, but in this case you need to configure either the HAProxy listen port or --apiserver-bind-port (kubadm init).

The nodes in this test environment are connected each other on the same subnet.

Hardware

These nodes are completely identical both on hardware and OS level, running on Proxmox Virtualization platform with KVM.

  • 2 CPU cores
  • 2GB memory
  • 32B system disk
  • Debian OS
  • Debian GNU/Linux 11 (bullseye)
  • Kernel: 5.10.0-21-amd64

Preparing The OS

In this post I'm using Ansible to prepare the Debian OSes for Kubernetes installation. I'm highly recommend to use some kind of automatization tool(s) or scirpt(s) to maintain your infrastructure, especially if you planning to have a bunch of nodes, not just a home-lab. And if something goes wrong you can start it over in a minute.

Ansible

Just a quick overview about my Ansible configuration and variables.

ansible.cfg

Click Here For Raw Source
1
2
3
[defaults]
inventory = hosts
host_key_checking = False

myvars.yml

Click Here For Raw Source
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
yq_url: https://github.com/mikefarah/yq/releases/download/v4.30.8/yq_linux_amd64
kube_version: 1.26.4-00

common_packages:
  - apt-transport-https
  - sudo
  - vim
  - vnstat
  - xz-utils
  - tcpdump
  - screen
  - rsync
  - sshpass
  - git
  - mc
  - wget
  - curl
  - jq
  - htop
  - net-tools
  - dnsutils
  - ca-certificates
  - telnet
  - htop
  - iotop
  - curl
  - gnupg
  - lsb-release
  - wireguard
  - wireguard-tools
  - whois
  - samba-common
  - smbclient
  - cifs-utils
  - nfs-common
  - open-iscsi
  - gnupg-agent
  - software-properties-common

docker_packages:
  - docker-ce
  - docker-ce-cli
  - containerd.io
  - docker-compose-plugin

Details:

  • yq_url: Yq binary URL. This version of yq will be installed on the hosts.
  • kube_version: Here you can define which version of Kubernetes you want to install. (kubelet, kubeadm and kubectl)
  • common_packages: These packages will be installed on the hosts. "Common packages" because usually I install these packages on my VMs, regardless of deploying Docker or Kubernetes.
  • docker_packages: Packages for installing Docker/Containerd engine.

hosts

Click Here For Raw Source
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
[pve-kube02]
172.16.1.77
172.16.1.78
172.16.1.79
172.16.1.80

[pve-kube02:vars]
ansible_connection=ssh
ansible_user=root
ansible_password=rootpassword

Important

Ansible host must access the VMs over ssh. Before you run any of the playbooks please enable root login. For example: sed -i -e 's/^#\(PermitRootLogin \).*/\1 yes/' /etc/ssh/sshd_config and restart sshd daemon. It is highly recommended to use dedicated ansible user (with sudo right) and ssh key authentication! And don't forget to accept ssh key by login to the remotes systems before run the playbooks. If you are using other user than root, you may want to use become: 'yes' option it the plays.

Update

I usually start with updating the OS to the latest version, unless the application to be installed has strict requirements.

playbook-upgrade-debian.yaml

Click Here For Raw Source
1
2
3
4
5
6
7
8
- hosts: pve-kube02
  name: Install
  tasks:
    - name: Run Included Task - Upgrade Ddebian
      ansible.builtin.import_tasks:
        file: task_allow_release_info_change.yaml
    - name: Reboot the machine 
      ansible.builtin.reboot:

task_allow_release_info_change.yaml

Click Here For Raw Source
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
- name: Allow release info change
  lineinfile:
    path: /etc/apt/apt.conf.d/99releaseinfochange
    state: present
    create: yes
    line: Acquire::AllowReleaseInfoChange::Suite "true";

- name: Update apt cache
  ansible.builtin.apt:
    update_cache: yes
    upgrade: full

Run this playbook:

ansible-playbook playbook-upgrade-debian.yaml

Install Common Packages

playbook-install-common-packages.yaml

Click Here For Raw Source
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
- name: Gather Facts
  hosts: 127.0.0.1
  connection: local
  tasks:
    - include_vars: myvars.yml
    - name: Download Yq
      ansible.builtin.get_url:
        url: "{{ yq_url }}"
        dest: /tmp/yq
        mode: '755'
    - name: Calculate MD5
      ansible.builtin.stat:
        path: /tmp/yq
        checksum_algorithm: md5
      register: yq_md5
    - name: Delete /tmp/yq
      ansible.builtin.file:
        path: /tmp/yq
        state: absent

- hosts: pve-kube02
  name: Install
  become: 'yes'
  tasks:
    - include_vars: myvars.yml
    - name: Run the equivalent of "apt-get update" as a separate step
      ansible.builtin.apt:
        update_cache: yes
    - name: Set Fact For YQ md5
      set_fact:
        yq_checksum: "{{ hostvars['127.0.0.1']['yq_md5'].stat.checksum }}"
    - name: debug
      debug:
        msg: "MD5 hash : {{ yq_checksum }}"
    - name: Ensure a list of packages installed
      ansible.builtin.apt:
        name: "{{ common_packages }}"
        state: present
    - name: All done!
      debug:
        msg: Packages have been successfully installed
    - name: Calculate Already Existing jq hash
      ansible.builtin.stat:
        path: /usr/bin/yq
        checksum_algorithm: md5
      register: exist_yq_md5
    - name: Print Existing yq md5 hash
      debug:
        msg: "MD5 hash of existing : {{ exist_yq_md5.stat.checksum }}"
      when: exist_yq_md5.stat.exists == true
    - name: Remove Old Version Of YQ
      ansible.builtin.file:
        path: /usr/bin/yq
        state: absent
      when: exist_yq_md5.stat.exists == false or exist_yq_md5.stat.checksum != yq_checksum
    - name: Download Yq
      ansible.builtin.get_url:
        url: "{{ yq_url }}"
        dest: /usr/bin/yq
        mode: '755'
      when: exist_yq_md5.stat.exists == false or exist_yq_md5.stat.checksum != yq_checksum
    - name: Fix Vimrc
      ansible.builtin.replace:
        path: /etc/vim/vimrc
        regexp: '^"\s?(let g:skip_defaults_vim.*)'
        replace: '\1'
    - name: Fix Vimrc 2
      ansible.builtin.replace:
        path: /etc/vim/vimrc
        regexp: '^"\s?(set compatible.*)'
        replace: '\1'
    - name: Fix Vimrc 3
      ansible.builtin.replace:
        path: /etc/vim/vimrc
        regexp: '^"\s?(set background).*'
        replace: '\1=dark'
    - name: Fix Vimrc 4
      ansible.builtin.replace:
        path: /etc/vim/vimrc
        regexp: '^"\s?(syntax on).*'
        replace: '\1'
    - name: Fix Vimrc 4
      ansible.builtin.replace:
        path: /etc/vim/vimrc
        regexp: '^"\s?(set mouse).*'
        replace: '\1=c'
    - name: Allow 'sudo' group to have passwordless sudo
      lineinfile:
        dest: /etc/sudoers
        state: present
        regexp: '^%sudo'
        line: '%sudo   ALL=(ALL:ALL) NOPASSWD:ALL'
        validate: visudo -cf %s

Run this playbook:

ansible-playbook playbook-install-common-packages.yaml

Install Container Engine

playbook-install-docker.yaml

Click Here For Raw Source
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
- hosts: pve-kube02
  become: 'yes'
  tasks:
    - include_vars: myvars.yml
    - name: determine codeversion
      command: "lsb_release -cs"
      register: release_output
    - set_fact:
        codename: "{{ release_output.stdout }}"

    - name: Run the equivalent of "apt-get update" as a separate step
      ansible.builtin.apt:
        update_cache: yes
    - name: add Docker GPG key
      apt_key:
        url: https://download.docker.com/linux/debian/gpg
        state: present
    - name: add docker repository to apt
      apt_repository:
        repo: deb https://download.docker.com/linux/debian "{{ codename }}" stable
        state: present
    - name: add tailscale gpg key
      apt_key:
        url: https://pkgs.tailscale.com/stable/debian/bullseye.noarmor.gpg
        state: present
    - name: add tailscale repository to apt
      apt_repository:
        repo: deb https://pkgs.tailscale.com/stable/debian "{{ codename }}" main
        state: present
    - name: install docker
      ansible.builtin.apt:
        name: "{{ docker_packages }}"
        state: present
        update_cache: yes

    - name: install tailscale
      apt:
        name: tailscale
        state: latest

    - name: check if docker is started properly
      service:
        name: docker
        state: started
        enabled: yes

Run this playbook:

ansible-playbook playbook-install-docker.yaml

Install Kubernetes Tools

playbook-install-kubernetes.yaml

Click Here For Raw Source
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
- hosts: pve-kube02
  become: 'yes'
  vars:
    kubepackages:
      - kubelet={{ kube_version }}
      - kubeadm={{ kube_version }}
      - kubectl={{ kube_version }}

  tasks:
    - include_vars: myvars.yml
    - name: Register architecture (dpkg_output)
      command: "dpkg --print-architecture"
      register: dpkg_output
    - set_fact:
        arch: "{{ dpkg_output.stdout }}"

    - name: Register lsb_release
      command: "lsb_release -cs"
      register: release_output
    - set_fact:
        codename: "{{ release_output.stdout }}"


    - name: Add Kubernetes gpg to keyring
      apt_key:
        url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
        state: present

    - name: add kubernetes repository to apt
      apt_repository:
        repo: deb https://apt.kubernetes.io/ kubernetes-xenial main

    - name: Disable SWAP since kubernetes can't work with swap enabled (1/2)
      shell: |
        swapoff -a


    - name: Disable SWAP in fstab since kubernetes can't work with swap enabled (2/2)
      replace:
        path: /etc/fstab
        regexp: '^([^#].*?\sswap\s+sw\s+.*)$'
        replace: '# \1'

    - name: Enable overlay & br_netfilter module
      ansible.builtin.copy:
        content: |
          overlay
          br_netfilter
        dest: /etc/modules-load.d/k8s.conf

    - name: Running modprobe
      shell: |
        modprobe overlay
        modprobe br_netfilter


    - name: Set up sysctl /etc/sysctl.d/k8s.conf
      ansible.builtin.copy:
        content: |
          net.bridge.bridge-nf-call-iptables  = 1
          net.bridge.bridge-nf-call-ip6tables = 1
          net.ipv4.ip_forward                 = 1
        dest: /etc/sysctl.d/k8s.conf

    - name: Add the overlay module
      community.general.modprobe:
        name: overlay
        state: present
    - name: Add the br_netfilter module
      community.general.modprobe:
        name: br_netfilter
        state: present
    - name: sysctl
      ansible.builtin.shell: "sysctl --system"

    - name: Generate default containerd config
      ansible.builtin.shell: "containerd config default > /etc/containerd/config.toml"

    - name: Change /etc/containerd/config.toml file SystemdCgroup  to true
      ansible.builtin.replace:
        path: /etc/containerd/config.toml
        after: 'plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options'
        before: 'plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]'
        regexp: 'SystemdCgroup.*'
        replace: 'SystemdCgroup = true'
      diff: yes


    - name: Run the equivalent of "apt-get update" as a separate step
      ansible.builtin.apt:
        update_cache: yes

    - name: Install Kubernetes Packages
      ansible.builtin.apt:
        name: "{{ kubepackages }}"
        state: present


    - name: Prevent kubelet from being upgraded
      ansible.builtin.dpkg_selections:
        name: kubelet
        selection: hold

    - name: Prevent kubeadm from being upgraded
      ansible.builtin.dpkg_selections:
        name: kubeadm
        selection: hold

    - name: Prevent kubectl from being upgraded
      ansible.builtin.dpkg_selections:
        name: kubectl
        selection: hold

    - name: Prevent containerd.io from being upgraded
      ansible.builtin.dpkg_selections:
        name: containerd.io
        selection: hold


    - name: FIX CRICTRL error
      ansible.builtin.copy:
        content: |
          runtime-endpoint: unix:///run/containerd/containerd.sock
          image-endpoint: unix:///run/containerd/containerd.sock
          timeout: 2
          debug: false
          pull-image-on-create: false
        dest: /etc/crictl.yaml


    - name: Restart service cron on centos, in all cases, also issue daemon-reload to pick up config changes
      ansible.builtin.systemd:
        state: restarted
        daemon_reload: true
        name: containerd

    - name: Install docker-compose from official github repo
      get_url:
        url : https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-linux-x86_64
        dest: /usr/local/bin/docker-compose
        mode: 'u+x,g+x'

Run this playbook:

ansible-playbook playbook-install-kubernetes.yaml

Now we have 3 identical nodes which are waiting for us to install & configure Tailscale VPN and Kubernetes cluster.

Before we proceed, I would like to advise you some really useful links and tips. These are helpful especially if you are not familiar with Ansible and don't want to bother with that:

If you follow these links you should be able to install everything without Ansible.

Tailscale VPN

I assume that Tailscale is successfully installed on every node. Before you begin please register a free account: Tailscale

Info

I recommend you to enable "MagicDNS" on the Tailscale web interface.

Run the following command on all 4 nodes (kube0-m[1,2,3] and kube02-haproxy):

tailscale up --accept-dns=true

Check if all your nodes have VPN IP address:

root@kube02-m3:~# tailscale status | grep kube02
100.124.70.97   kube02-m3            jvincze84@   linux   -
100.121.89.125  kube02-haproxy       jvincze84@   linux   -
100.122.123.2   kube02-m1            jvincze84@   linux   -
100.103.128.9   kube02-m2            jvincze84@   linux   -

Try ping:

root@kube02-m3:~# ping kube02-m1
PING kube02-m1.tailnet-a5cd.ts.net (100.122.123.2) 56(84) bytes of data.
64 bytes from kube02-m1.tailnet-a5cd.ts.net (100.122.123.2): icmp_seq=1 ttl=64 time=1.33 ms
64 bytes from kube02-m1.tailnet-a5cd.ts.net (100.122.123.2): icmp_seq=2 ttl=64 time=0.976 ms
^C
--- kube02-m1.tailnet-a5cd.ts.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.976/1.151/1.327/0.175 ms

root@kube02-m3:~# ping kube02-m2
PING kube02-m2.tailnet-a5cd.ts.net (100.103.128.9) 56(84) bytes of data.
64 bytes from kube02-m2.tailnet-a5cd.ts.net (100.103.128.9): icmp_seq=1 ttl=64 time=1.49 ms
64 bytes from kube02-m2.tailnet-a5cd.ts.net (100.103.128.9): icmp_seq=2 ttl=64 time=1.06 ms
^C
--- kube02-m2.tailnet-a5cd.ts.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 1.055/1.272/1.490/0.217 ms

root@kube02-m3:~# ping kube02-haproxy
PING kube02-haproxy.tailnet-a5cd.ts.net (100.121.89.125) 56(84) bytes of data.
64 bytes from kube02-haproxy.tailnet-a5cd.ts.net (100.121.89.125): icmp_seq=1 ttl=64 time=1.27 ms
^C
--- kube02-haproxy.tailnet-a5cd.ts.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.269/1.269/1.269/0.000 ms

It seems that everything is fine.

Now check tailscale status command again:

root@kube02-m3:~# tailscale status | grep kube02
100.124.70.97   kube02-m3            jvincze84@   linux   -
100.121.89.125  kube02-haproxy       jvincze84@   linux   active; direct 172.16.1.80:41641, tx 468 rx 348
100.122.123.2   kube02-m1            jvincze84@   linux   active; direct 172.16.1.77:41641, tx 724 rx 604
100.103.128.9   kube02-m2            jvincze84@   linux   active; direct 172.16.1.78:41641, tx 596 rx 476

After you connect to a host, the status command will show extra information: active; direct 172.16.1.80:41641, tx 468 rx 348

Updated IP Address Table

Hostname Role IP Address VPN IP Address
kube02-m1 Control Plane Node 1 172.16.1.77 100.122.123.2
kube02-m2 Control Plane Node 2 172.16.1.78 100.103.128.9
kube02-m3 Control Plane Node 3 172.16.1.79 100.124.70.97
kube02-haproxy HAProxy Load Balancer 172.16.1.80 100.121.89.125
ansible Ansible Host 172.16.0.252 ---

(Optional) Disable Direct Access

This step is optional. I want to simulate the situation when the nodes are not sitting in the same subnet, and can talk to each other only over the Tailscale VPN. This way maybe easier to understand what we doing with the VPN.

I don't want to make it complicated, so simply disable the communication between node with iptables.

kube02-m1

iptables -I INPUT -s 172.16.1.78 -j DROP
iptables -I INPUT -s 172.16.1.79 -j DROP

kube02-m2

iptables -I INPUT -s 172.16.1.77 -j DROP
iptables -I INPUT -s 172.16.1.79 -j DROP

kube02-m3

iptables -I INPUT -s 172.16.1.77 -j DROP
iptables -I INPUT -s 172.16.1.78 -j DROP

Note

These rules are not permanent. So, if you restart the machine you should apply them again.

Check the Tailscale Connection

Don't forget the ping hosts before the tailscale status command.

root@kube02-m1:~# tailscale status | grep kube02-m
100.122.123.2   kube02-m1            jvincze84@   linux   -
100.103.128.9   kube02-m2            jvincze84@   linux   active; relay "fra", tx 308 rx 220
100.124.70.97   kube02-m3            jvincze84@   linux   active; relay "waw", tx 308 rx 220

Now you can see that the hosts are connected to each other via relay servers (active; relay "fra", tx 308 rx 22) provided by Tailscale.

To see the available relays, run the tailscale netcheck command.

root@kube02-m1:~# tailscale netcheck

Report:
        * UDP: true
        * IPv4: yes, 176.*.*.107:58839
        * IPv6: no, but OS has support
        * MappingVariesByDestIP: false
        * HairPinning: false
        * PortMapping:
        * Nearest DERP: Frankfurt
        * DERP latency:
                - fra: 18.4ms  (Frankfurt)
                - waw: 19.6ms  (Warsaw)
                - ams: 25.1ms  (Amsterdam)
                - par: 32ms    (Paris)
                - mad: 44.7ms  (Madrid)
                - lhr: 59.5ms  (London)
                - nyc: 119.4ms (New York City)
                - tor: 119.5ms (Toronto)
                - ord: 119.9ms (Chicago)
                - dbi: 130.8ms (Dubai)
                - mia: 132.8ms (Miami)
                - den: 140.8ms (Denver)
                - dfw: 145.8ms (Dallas)
                - sfo: 166.8ms (San Francisco)
                - lax: 170.1ms (Los Angeles)
                - sin: 170.5ms (Singapore)
                - sea: 177.8ms (Seattle)
                - blr: 193.9ms (Bangalore)
                - jnb: 198.7ms (Johannesburg)
                - hkg: 209ms   (Hong Kong)
                - sao: 214.5ms (São Paulo)
                - hnl: 215.8ms (Honolulu)
                - syd:         (Sydney)
                - tok:         (Tokyo)
root@kube02-m1:~#

This is one of my favorite feature of Tailscale. You don't have to have stable static public IP address to use VPN service. But keep in mind, that connection over relay server can be significantly slower than direct connection.

Init The Cluster

Prepare Kubelet

Before you do anything, prepare the kubelet to use Tailscale VPN IP address as node IP address.

Run this command on all Kubernetes nodes:

echo "KUBELET_EXTRA_ARGS=--node-ip=$(tailscale ip --4)" | tee -a /etc/default/kubelet
Example Commands: 
root@kube02-m1:~# echo "KUBELET_EXTRA_ARGS=--node-ip=$(tailscale ip --4)" | tee -a /etc/default/kubelet
KUBELET_EXTRA_ARGS=--node-ip=100.122.123.2
root@kube02-m1:~#

root@kube02-m2:~# echo "KUBELET_EXTRA_ARGS=--node-ip=$(tailscale ip --4)" | tee -a /etc/default/kubelet
KUBELET_EXTRA_ARGS=--node-ip=100.103.128.9
root@kube02-m2:~#

root@kube02-m3:~# echo "KUBELET_EXTRA_ARGS=--node-ip=$(tailscale ip --4)" | tee -a /etc/default/kubelet
KUBELET_EXTRA_ARGS=--node-ip=100.124.70.97
root@kube02-m3:~#

Prepare The Load Balancer

I won't want to waste a lot time for this task, since this is only a lab env with just one function: demonstrate the installation. HAProxy is a really good example about how to configure an external Load Balancer for kubernetes control plane nodes.

Check if MagicDNS is working fine

root@kube02-haproxy:~# nslookup kube02-m1
Server:         100.100.100.100
Address:        100.100.100.100#53

Name:   kube02-m1.tailnet-a5cd.ts.net
Address: 100.122.123.2

root@kube02-haproxy:~# nslookup kube02-m2
Server:         100.100.100.100
Address:        100.100.100.100#53

Name:   kube02-m2.tailnet-a5cd.ts.net
Address: 100.103.128.9

root@kube02-haproxy:~# nslookup kube02-m3
Server:         100.100.100.100
Address:        100.100.100.100#53

Name:   kube02-m3.tailnet-a5cd.ts.net
Address: 100.124.70.97

/etc/haproxy.conf config file:

frontend kubeapi
  log global
  bind *:6443
  mode tcp
  option tcplog
  default_backend kubecontroleplain

backend kubecontroleplain
  option httpchk GET /healthz
  http-check expect status 200
  mode tcp
  log global
  balance roundrobin
  #option tcp-check
  option ssl-hello-chk
  server kube02-m1 kube02-m1.tailnet-a5cd.ts.net:6443 check
  server kube02-m2 kube02-m2.tailnet-a5cd.ts.net:6443 check
  server kube02-m3 kube02-m3.tailnet-a5cd.ts.net:6443 check


frontend stats
    mode http
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

Warning

As I know HAProxy resolve DNS only once at startup. So use DNS name in server section with caution. If the IP address has changed, do not forget to restart HAProxy.

Run HAProxy:

docker run --name haproxy -d -p 6443:6443 -p 8404:8404 -v /etc/haproxy.conf:/usr/local/etc/haproxy/haproxy.cfg haproxy

Info

*.tailnet-a5cd.ts.net is my MagicDNS name.

Tip

If you are looking for a solution without external Load Balancer you may want to take a look at Kube-Vip

Quote

kube-vip provides Kubernetes clusters with a virtual IP and load balancer for both the control plane (for building a highly-available cluster) and Kubernetes Services of type LoadBalancer without relying on any external hardware or software.

Init The First Control Plane Node

The command:

kubeadm init --cri-socket /var/run/containerd/containerd.sock \
--control-plane-endpoint kube02-haproxy.tailnet-a5cd.ts.net \
--apiserver-advertise-address $(tailscale ip --4) \
--pod-network-cidr 10.25.0.0/16 \
--service-cidr 10.26.0.0/16 \
--upload-certs

Important

If you don't have separate HAProxy node, and you are using one kubernetes node, you should consider changing the --apiserver-bind-port port or the listen port of the HAProxy.

Important

pod-network-cidr and service-cidr is required by flannel CNI.

Important

Do not forget the --upload-certs option, otherwise additional control plane nodes won't be able to join the cluster without extra steps.

Command Output 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
root@kube02-m1:# kubeadm init --cri-socket /var/run/containerd/containerd.sock \
--control-plane-endpoint kube02-haproxy.tailnet-a5cd.ts.net \
--apiserver-advertise-address $(tailscale ip --4) \
--pod-network-cidr 10.25.0.0/16 \
--service-cidr 10.26.0.0/16 \
--upload-certs
W0421 16:13:16.891232   25655 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/containerd/containerd.sock". Please update your configuration!
I0421 16:13:17.241235   25655 version.go:256] remote version is much newer: v1.27.1; falling back to: stable-1.26
[init] Using Kubernetes version: v1.26.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kube02-haproxy.tailnet-a5cd.ts.net kube02-m1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.26.0.1 100.122.123.2]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kube02-m1 localhost] and IPs [100.122.123.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kube02-m1 localhost] and IPs [100.122.123.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 101.038704 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
2f2caa21e13d7f4bece27faa2515d024c8b4e93e08d8d21612113a7ebacff5ea
[mark-control-plane] Marking the node kube02-m1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node kube02-m1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 1q32dn.swfpr7qj89hl2g4j
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join kube02-haproxy.tailnet-a5cd.ts.net:6443 --token 1q32dn.swfpr7qj89hl2g4j \
        --discovery-token-ca-cert-hash sha256:11c669ee4e4e27b997ae5431133dd2cd7c6a2050ddd16b38bee8bee544bbe680 \
        --control-plane --certificate-key 2f2caa21e13d7f4bece27faa2515d024c8b4e93e08d8d21612113a7ebacff5ea

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join kube02-haproxy.tailnet-a5cd.ts.net:6443 --token 1q32dn.swfpr7qj89hl2g4j \
        --discovery-token-ca-cert-hash sha256:11c669ee4e4e27b997ae5431133dd2cd7c6a2050ddd16b38bee8bee544bbe680

Run these commands:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

And finally check the nodes:

kubectl get nodes -o wide
NAME        STATUS     ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
kube02-m1   NotReady   control-plane   2m45s   v1.26.4   100.122.123.2   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   containerd://1.6.20

Init Additional Control Plane Nodes

Command:

kubeadm join kube02-haproxy.tailnet-a5cd.ts.net:6443 --token 1q32dn.swfpr7qj89hl2g4j \
--apiserver-advertise-address $(tailscale ip --4) \
--cri-socket /var/run/containerd/containerd.sock \
--discovery-token-ca-cert-hash sha256:11c669ee4e4e27b997ae5431133dd2cd7c6a2050ddd16b38bee8bee544bbe680 \
--control-plane --certificate-key 2f2caa21e13d7f4bece27faa2515d024c8b4e93e08d8d21612113a7ebacff5ea

Important

Important that the nodes must use their own VPN address as apiserver-advertise-address

Command Output 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
root@kube02-m2:# kubeadm join kube02-haproxy.tailnet-a5cd.ts.net:6443 --token 1q32dn.swfpr7qj89hl2g4j --apiserver-advertise-address $(tailscale ip --4) --cri-socket /var/run/containerd/containerd.sock --discovery-token-ca-cert-hash sha256:11c669ee4e4e27b997ae5431133dd2cd7c6a2050ddd16b38bee8bee544bbe680 --control-plane --certificate-key 2f2caa21e13d7f4bece27faa2515d024c8b4e93e08d8d21612113a7ebacff5ea
W0421 16:23:11.602945   26931 initconfiguration.go:119] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/containerd/containerd.sock". Please update your configuration!
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[download-certs] Saving the certificates to the folder: "/etc/kubernetes/pki"
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kube02-haproxy.tailnet-a5cd.ts.net kube02-m2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.26.0.1 100.103.128.9]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kube02-m2 localhost] and IPs [100.103.128.9 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kube02-m2 localhost] and IPs [100.103.128.9 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s

The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[mark-control-plane] Marking the node kube02-m2 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node kube02-m2 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

Finally check the nodes:

root@kube02-m1:~# kubectl get nodes -o wide
NAME        STATUS     ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
kube02-m1   NotReady   control-plane   10m   v1.26.4   100.122.123.2   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   containerd://1.6.20
kube02-m2   NotReady   control-plane   98s   v1.26.4   100.103.128.9   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   containerd://1.6.20
kube02-m3   NotReady   control-plane   16s   v1.26.4   100.124.70.97   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   containerd://1.6.20

(optional) Post Init Steps

These steps are useful when you won't have any worker nodes, just control-planes.

Mark Nodes As Worker

kubectl label node kube02-m1 node-role.kubernetes.io/worker=
kubectl label node kube02-m2 node-role.kubernetes.io/worker=
kubectl label node kube02-m3 node-role.kubernetes.io/worker=

Untaint The Nodes

kubectl taint nodes kube02-m1 node-role.kubernetes.io/control-plane=:NoSchedule-
kubectl taint nodes kube02-m2 node-role.kubernetes.io/control-plane=:NoSchedule-
kubectl taint nodes kube02-m3 node-role.kubernetes.io/control-plane=:NoSchedule-

Our nodes are in NotReady state, because no network plugin is installed.

Install Network Plugin

In this post I will show two network plugin: flannel and weave.

Weave

Download The Manifest

wget https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

You should consider to change iptables mode:

IPTABLES_BACKEND - set to nft to use nftables backend for iptables (default is iptables)

In my case I use nft, so I have to add IPTABLES_BACKEND environment variable and set to nft

1
2
3
4
5
6
7
          containers:
            - name: weave
              command:
                - /home/weave/launch.sh
              env:
                - name: IPTABLES_BACKEND
                  value: nft

Apply the manifest

kubectl  apply -f weave-daemonset-k8s.yaml

Check the status of the Weave PODs:

kubectl -n kube-system get pods -l name=weave-net -o wide

You should see something like this:

NAME              READY   STATUS    RESTARTS      AGE   IP              NODE        NOMINATED NODE   READINESS GATES
weave-net-drz44   2/2     Running   1 (59s ago)   68s   100.103.128.9   kube02-m2   <none>           <none>
weave-net-p72nl   2/2     Running   1 (59s ago)   68s   100.124.70.97   kube02-m3   <none>           <none>
weave-net-zzj9p   2/2     Running   1 (59s ago)   68s   100.122.123.2   kube02-m1   <none>           <none>

If everything went good, you should see that the coredns pods are running:

root@kube02-m1:~# kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP          NODE        NOMINATED NODE   READINESS GATES
coredns-787d4945fb-2d5zm   1/1     Running   0          21m   10.40.0.0   kube02-m1   <none>           <none>
coredns-787d4945fb-gmtbr   1/1     Running   0          21m   10.40.0.1   kube02-m1   <none>           <none>

Flannel

Download the manifest:

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Modify the manifest to use the tailscale0 interface (iface=tailscale0) and our network CIDR:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
--- kube-flannel-orig.yml       2023-03-22 15:48:52.000000000 +0100
+++ kube-flannel.yml    2023-04-25 08:27:05.183037370 +0200
@@ -81,21 +81,21 @@
         {
           "type": "portmap",
           "capabilities": {
             "portMappings": true
           }
         }
       ]
     }
   net-conf.json: |
     {
-      "Network": "10.244.0.0/16",
+      "Network": "10.25.0.0/16",
       "Backend": {
         "Type": "vxlan"
       }
     }
 kind: ConfigMap
 metadata:
   labels:
     app: flannel
     k8s-app: flannel
     tier: node
@@ -129,20 +129,21 @@
             nodeSelectorTerms:
             - matchExpressions:
               - key: kubernetes.io/os
                 operator: In
                 values:
                 - linux
       containers:
       - args:
         - --ip-masq
         - --kube-subnet-mgr
+        - --iface=tailscale0
         command:
         - /opt/bin/flanneld
         env:
         - name: POD_NAME
           valueFrom:
             fieldRef:
               fieldPath: metadata.name
         - name: POD_NAMESPACE
           valueFrom:
             fieldRef:
root@kube02-m1:~#

Apply the manifest:

kubectl apply -f kube-flannel.yml
Example flannel-ds logs kubectl -n kube-flannel logs kube-flannel-ds-h2rp2 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
I0425 06:34:24.144883       1 main.go:211] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[tailscale0] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true useMultiClusterCidr:false}
W0425 06:34:24.146218       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0425 06:34:24.207098       1 kube.go:144] Waiting 10m0s for node controller to sync
I0425 06:34:24.207158       1 kube.go:485] Starting kube subnet manager
I0425 06:34:24.220083       1 kube.go:506] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.25.0.0/24]
I0425 06:34:24.220116       1 kube.go:506] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.25.1.0/24]
I0425 06:34:24.220245       1 kube.go:506] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.25.2.0/24]
I0425 06:34:25.207246       1 kube.go:151] Node controller sync successful
I0425 06:34:25.207266       1 main.go:231] Created subnet manager: Kubernetes Subnet Manager - kube02-m1
I0425 06:34:25.207272       1 main.go:234] Installing signal handlers
I0425 06:34:25.207991       1 main.go:542] Found network config - Backend type: vxlan
I0425 06:34:25.208718       1 match.go:259] Using interface with name tailscale0 and address 100.122.123.2
I0425 06:34:25.208762       1 match.go:281] Defaulting external address to interface address (100.122.123.2)
I0425 06:34:25.208917       1 vxlan.go:140] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I0425 06:34:25.210776       1 main.go:356] Setting up masking rules
I0425 06:34:25.252429       1 main.go:407] Changing default FORWARD chain policy to ACCEPT
I0425 06:34:25.253690       1 iptables.go:290] generated 7 rules
I0425 06:34:25.256468       1 main.go:435] Wrote subnet file to /run/flannel/subnet.env
I0425 06:34:25.256489       1 main.go:439] Running backend.
I0425 06:34:25.256646       1 iptables.go:290] generated 3 rules
I0425 06:34:25.257126       1 vxlan_network.go:64] watching for new subnet leases
I0425 06:34:25.258108       1 watch.go:51] Batch elem [0] is { subnet.Event{Type:0, Lease:subnet.Lease{EnableIPv4:true, EnableIPv6:false, Subnet:ip.IP4Net{IP:0xa190100, PrefixLen:0x18}, IPv6Subnet:ip.IP6Net{IP:(*ip.IP6)(nil), PrefixLen:0x0}, Attrs:subnet.LeaseAttrs{PublicIP:0x64678009, PublicIPv6:(*ip.IP6)(nil), BackendType:"vxlan", BackendData:json.RawMessage{0x7b, 0x22, 0x56, 0x4e, 0x49, 0x22, 0x3a, 0x31, 0x2c, 0x22, 0x56, 0x74, 0x65, 0x70, 0x4d, 0x41, 0x43, 0x22, 0x3a, 0x22, 0x35, 0x32, 0x3a, 0x39, 0x65, 0x3a, 0x32, 0x62, 0x3a, 0x63, 0x33, 0x3a, 0x38, 0x34, 0x3a, 0x63, 0x37, 0x22, 0x7d}, BackendV6Data:json.RawMessage(nil)}, Expiration:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Asof:0}} }
I0425 06:34:25.258185       1 watch.go:51] Batch elem [0] is { subnet.Event{Type:0, Lease:subnet.Lease{EnableIPv4:true, EnableIPv6:false, Subnet:ip.IP4Net{IP:0xa190200, PrefixLen:0x18}, IPv6Subnet:ip.IP6Net{IP:(*ip.IP6)(nil), PrefixLen:0x0}, Attrs:subnet.LeaseAttrs{PublicIP:0x647c4661, PublicIPv6:(*ip.IP6)(nil), BackendType:"vxlan", BackendData:json.RawMessage{0x7b, 0x22, 0x56, 0x4e, 0x49, 0x22, 0x3a, 0x31, 0x2c, 0x22, 0x56, 0x74, 0x65, 0x70, 0x4d, 0x41, 0x43, 0x22, 0x3a, 0x22, 0x33, 0x61, 0x3a, 0x38, 0x31, 0x3a, 0x36, 0x30, 0x3a, 0x63, 0x64, 0x3a, 0x66, 0x63, 0x3a, 0x62, 0x63, 0x22, 0x7d}, BackendV6Data:json.RawMessage(nil)}, Expiration:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Asof:0}} }
I0425 06:34:25.290612       1 iptables.go:283] bootstrap done
I0425 06:34:25.311445       1 iptables.go:283] bootstrap done
I0425 06:34:25.322841       1 main.go:460] Waiting for all goroutines to exit

Warning

Choose only one CNI plugin, do not install both flannel and weave. If you want to replace weave you should remove it:

  • kubectl delete -f weave-daemonset-k8s.yaml
  • rm /etc/cni/net.d/10-weave.conflist
  • Additionally rebooting the nodes may be necessary.
  • After you applied the kube manifest run kubectl delete pods --all -A command. This will recreate all pods in the entire cluster!!!

I really recommend you to choose CNI network plugin carefully before you getting use your cluster, and avoid changing CNI in the future.

Switch Between iptables-legacy And iptables-nft

If you want or need to change iptables to or from iptables-legacy please check this link: https://wiki.debian.org/iptables

Example, how to change to legacy:

apt-get install arptables
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy

(bonus) - Persistent Storage

Almost all Kubernetes Cluster have some kind of PersistentVolume solution for storing data. Now we will deploy Longhorn

It is just a simple command:

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.4.1/deploy/longhorn.yaml

Info

If you want to use RWX volumes NFSv4 client must be installed on all Kubernetes nodes. LINK

Create PVC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc-02
spec:
  storageClassName: longhorn #
  accessModes:
    - ReadWriteMany # 
  resources:
    requests:
      storage: 1Gi
EOF

Create Pod To Consume The Storage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: hello-pvc
  namespace: default
spec:
  volumes:
  - name: storage
    persistentVolumeClaim:
      claimName: test-pvc-02
  containers:
  - name: hello-container
    image: busybox
    command:
       - sh
       - -c
       - 'while true; do echo "`date` [`hostname`] Hello from Longhorn PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
    volumeMounts:
    - mountPath: /mnt/store
      name: storage
EOF