How to configure certs for HA cluster for v1.15.0

In a Kubernetes cluster, there are several different components, such as etcd, api-service, scheduler, controller, Kube-proxy, Kubelet etc, lots of communication will happen among all these components, its a very important to make sure all these communication secured, so setup tls certs to secure all these communication is another important task when creating a K8s cluster, by default, K8s will generate tls certs automatically with only 1 year duration, this means 1 year later, all these certs will be expired, we need to follow some process to renew these certs otherwise your K8s cluster will stop working. in this blog, I will describe how to use cfssl tool to generate tls certs and use these certs for new K8s cluster.

install cfssl:

curl -s -L -o /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 curl -s -L -o /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 curl -s -L -o /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 chmod +x /bin/cfssl*

Write configuration file:

CA cert configuration root@ppydalbik0103:/etc/kubernetes/certs# more ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "server": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth" ] }, "client": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "client auth" ] }, "peer": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] }, "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } ca cert csr configuration(certificate signing request) root@ppydalbik0103:/etc/kubernetes/certs# more ca-csr.json { "CN": "Kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "US", "ST": "Dallas, TX", "L": "Dallas, TX", "O": "k8s", "OU": "System" } ] } kubernetes certs csr file root@ppydalbik0103:/etc/kubernetes/certs# more certs-csr.json { "CN": "kubernetes", "hosts": [ "127.0.0.1", "::1", "10.94.xxx.xxx", "10.95.xxx.xxx", "10.94.xxx.xxx", "10.94.xxx.xxx", "172.17.0.1", "ppydalbik0101.xxx.xxx.xxx.com", "ppydalbik0102.xxx.xxx.xxx.com", "ppydalbik0103.xxx.xxx.xxx.com", "ppydalbik0104.xxx.xxx.xxx.com", "ppydalbik0101", "ppydalbik0102", "ppydalbik0103", "ppydalbik0104", "localhost", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "US", "ST": "Dallas, TX", "L": "Dallas, TX", "O": "k8s", "OU": "System" } ] }

generate CA certs:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca cfssl gencert -initca ca-csr.json | cfssljson -bare front-proxy-ca cfssl gencert -initca ca-csr.json | cfssljson -bare etcd-ca ##generate all the srever certs: cfssl gencert -ca=ca.pem -ca-key=ca-key.pem --config=ca-config.json -profile=server certs-csr.json | cfssljson -bare apiserver cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem --config=ca-config.json -profile=kubernetes certs-csr.json | cfssljson -bare server cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem --config=ca-config.json -profile=peer certs-csr.json | cfssljson -bare peer ##generate all the clients certs: cfssl gencert -ca=ca.pem -ca-key=ca-key.pem --config=ca-config.json -profile=client certs-csr.json | cfssljson -bare apiserver-kubelet-client cfssl gencert -ca=front-proxy-ca.pem -ca-key=front-proxy-ca-key.pem --config=ca-config.json -profile=client certs-csr.json | cfssljson -bare front-proxy-client cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem --config=ca-config.json -profile=client certs-csr.json | cfssljson -bare apiserver-etcd-client cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem --config=ca-config.json -profile=client certs-csr.json | cfssljson -bare healthcheck-client ##deploy new certs to kuberenetes certs folder: mkdir -p /etc/kubernetes/pki/etcd cp /etc/kubernetes/certs/front-proxy-client.pem /etc/kubernetes/pki/front-proxy-client.crt cp /etc/kubernetes/certs/front-proxy-client-key.pem /etc/kubernetes/pki/front-proxy-client.key cp /etc/kubernetes/certs/front-proxy-ca.pem /etc/kubernetes/pki/front-proxy-ca.crt cp /etc/kubernetes/certs/front-proxy-ca-key.pem /etc/kubernetes/pki/front-proxy-ca.key cp /etc/kubernetes/certs/apiserver-kubelet-client.pem /etc/kubernetes/pki/apiserver-kubelet-client.crt cp /etc/kubernetes/certs/apiserver-kubelet-client-key.pem /etc/kubernetes/pki/apiserver-kubelet-client.key cp /etc/kubernetes/certs/apiserver.pem /etc/kubernetes/pki/apiserver.crt cp /etc/kubernetes/certs/apiserver-key.pem /etc/kubernetes/pki/apiserver.key cp /etc/kubernetes/certs/apiserver-etcd-client.pem /etc/kubernetes/pki/apiserver-etcd-client.crt cp /etc/kubernetes/certs/apiserver-etcd-client-key.pem /etc/kubernetes/pki/apiserver-etcd-client.key cp /etc/kubernetes/certs/ca.pem /etc/kubernetes/pki/ca.crt cp /etc/kubernetes/certs/ca-key.pem /etc/kubernetes/pki/ca.key cp /etc/kubernetes/certs/etcd-ca.pem /etc/kubernetes/pki/etcd/ca.crt cp /etc/kubernetes/certs/etcd-ca-key.pem /etc/kubernetes/pki/etcd/ca.key cp /etc/kubernetes/certs/healthcheck-client.pem /etc/kubernetes/pki/etcd/healthcheck-client.crt cp /etc/kubernetes/certs/healthcheck-client-key.pem /etc/kubernetes/pki/etcd/healthcheck-client.key cp /etc/kubernetes/certs/server.pem /etc/kubernetes/pki/etcd/server.crt cp /etc/kubernetes/certs/server-key.pem /etc/kubernetes/pki/etcd/server.key cp /etc/kubernetes/certs/peer.pem /etc/kubernetes/pki/etcd/peer.crt cp /etc/kubernetes/certs/peer-key.pem /etc/kubernetes/pki/etcd/peer.key

copy all the certs to all master nodes:

root@ppydalbik0102:/etc/kubernetes/pki# ls -lt total 52 drwx------ 2 root root 4096 Jul 19 02:59 etcd -rw------- 1 root root 1679 Jul 19 02:59 ca.key -rw------- 1 root root 1375 Jul 19 02:59 ca.crt -rw------- 1 root root 1675 Jul 19 02:59 apiserver-etcd-client.key -rw------- 1 root root 1838 Jul 19 02:59 apiserver-etcd-client.crt -rw------- 1 root root 1679 Jul 19 02:59 apiserver.key -rw------- 1 root root 1838 Jul 19 02:59 apiserver.crt -rw------- 1 root root 1838 Jul 19 02:59 apiserver-kubelet-client.crt -rw------- 1 root root 1675 Jul 19 02:59 apiserver-kubelet-client.key -rw------- 1 root root 1679 Jul 19 02:59 front-proxy-ca.key -rw------- 1 root root 1375 Jul 19 02:59 front-proxy-ca.crt -rw------- 1 root root 1679 Jul 19 02:59 front-proxy-client.key -rw------- 1 root root 1838 Jul 19 02:59 front-proxy-client.crt root@ppydalbik0102:/etc/kubernetes/pki# cd etcd root@ppydalbik0102:/etc/kubernetes/pki/etcd# ls -lt total 32 -rw------- 1 root root 1679 Jul 19 02:59 peer.key -rw------- 1 root root 1850 Jul 19 02:59 peer.crt -rw------- 1 root root 1675 Jul 19 02:59 server.key -rw------- 1 root root 1838 Jul 19 02:59 server.crt -rw------- 1 root root 1675 Jul 19 02:59 healthcheck-client.key -rw------- 1 root root 1838 Jul 19 02:59 healthcheck-client.crt -rw------- 1 root root 1375 Jul 19 02:59 ca.crt -rw------- 1 root root 1675 Jul 19 02:59 ca.key

then we can use Kubeadm to initialize cluster.

kubeadm init --config=/etc/kubernetes/k8s-cluster-bi.yaml --upload-certs

check certs status:

kubeadm alpha certs check-expiration CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED admin.conf Jul 17, 2020 08:46 UTC 364d no apiserver Jul 15, 2029 07:53 UTC 9y no apiserver-etcd-client Jul 15, 2029 08:38 UTC 9y no apiserver-kubelet-client Jul 15, 2029 08:25 UTC 9y no controller-manager.conf Jul 17, 2020 08:46 UTC 364d no etcd-healthcheck-client Jul 15, 2029 08:41 UTC 9y no etcd-peer Jul 15, 2029 08:40 UTC 9y no etcd-server Jul 15, 2029 08:39 UTC 9y no front-proxy-client Jul 15, 2029 08:35 UTC 9y no scheduler.conf Jul 17, 2020 08:46 UTC 364d no

attention there are 3 client certs with 1 year expiration in .conf files, these client certs will be refreshed automatically when they will be expired.