Update Kuberbetes certificates

Lars J├Ânsson, 2021-02-07

Information about how to replace expired certificates in a Kubernetes node. Normally most of the certificates will be replaced automatically in a running system and kubeadm can be also be used for for replacing the certificates. When the Kubernetes cluster is not running all the time, it may end up in a non-startable cluster and the certificates needs to be updated manually.

This instruction also include information about how to replace the HTTPS certificate for the Dashboard.

Kubelet certificates

The kubelet certificate needs to be udated manually if the it has expired.

Check the current certificate

Print the current kubelet certificate:

$ sudo su
# cd /etc/kubernetes/
# grep client-certificate-data kubelet.conf | awk '{print $2}' | base64 --decode | openssl x509 -text -noout
# exit

A printout similar to this will be printed. Check the Validity entry to see if the certificate is still valid.

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Nov 12 09:39:20 2020 GMT
            Not After : Sep  8 09:39:20 2021 GMT
        Subject: CN = kubelet
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:e1:9a:48:d0:d9:d9:4f:c3:19:7a:2c:bb:8a:c9:
                    90:9d:15:73:89:ca:70:c9:f1:3a:e0:28:41:87:4a:
                    49:71:da:b1:ec:5d:e1:df:79:bc:bc:0f:bd:29:3e:
                                   ..............
                    08:95:01:b1:8d:9e:79:67:00:04:2f:06:59:a3:87:
                    23:6a:37:bf:07:db:e9:61:dd:06:ae:a2:a8:5e:71:
                    dd:21
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         3d:9d:eb:b1:fe:20:32:b4:03:ba:a3:36:93:79:31:78:e7:72:
         6d:5e:76:01:78:6c:8d:8d:b2:28:26:bc:b5:ed:5d:0d:04:9e:
         5e:eb:6e:d0:27:86:45:61:ee:fa:b3:e1:3e:72:c7:31:a9:9d:
                        ..............
         fd:a1:79:7e:16:42:8a:72:e8:6a:ff:60:44:e6:7a:99:c1:fd:
         89:ad:55:8c:56:e2:6f:af:f6:2f:c9:11:39:62:a5:41:f6:f2:
         49:9c:52:21

Create and install new kubelet key and certificate

As a root, jump to the Kubernetes configuration folder.

$ sudo su
# cd /etc/kubernetes/

If the file with the serial number (ca.srl) of the CA certificate is missing, it needs to be created with the current serial number.

# openssl x509 -in pki/ca.crt -serial -noout
# echo 00 > pki/ca.srl 

Create a new key and certificate:

# openssl genrsa -out kubelet.key 2048
# openssl req -new -key kubelet.key -subj "/CN=kubelet" -out kubelet.csr
# openssl x509 -req -in kubelet.csr -CA /etc/kubernetes/pki/ca.crt \
  -CAkey /etc/kubernetes/pki/ca.key -out kubelet.crt -days 300

Convert the key and certificate into base64 format:

# echo `base64 -w 0 kubelet.key`
# echo `base64 -w 0 kubelet.crt`

Replace the xxxclient-certificate-data resp. client-key-data entries in the kubelet.conf file with the converted key and certificate:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJ - - - - - - 96ND0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.121.20:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:kub-master
  name: system:node:kub-master@kubernetes
current-context: system:node:kub-master@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:kub-master
  user:
    client-certificate-data: LS0tLS1CRUdJTiB - - - - - - 4VzRtK3Y5aS9KRVRsaXBVSDI4a21jVWlFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJ - - - - - - scUFKbkNSa2VJZEp5Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

End the root user session:

# exit

Kubernetes certificates

NOTE: Remember to add copying of new admin.conf file to uabjos!

Most of the Kubernetes certificates can be renewed by using kubeadm.

Update the certificates:

$ sudo kubeadm alpha certs renew all

Update the users Kubernetes configuration, if the admin.conf file is updated.

$ cd ~/.kube
$ mv config old-config
$ sudo cp /etc/kubernetes/admin.conf config
$ sudo chown `id -un`:`id -gn` config 

Kuberneters Dashboard certificate

The Kubernetes Dashboard HTTPS certificate is signed by a CA to prevent Web browsers from complaining about invalid certificates. If it is signed by a local CA, the root and intermediate certificates of the CA needs to installed in the Web browser.

It is only the Kubernetes part of the certificate that is covered, not the signing process at the CA.

More information about certificates is available in the Certificates guide.

Generate a key:

$ openssl genrsa -aes256 -out 192.168.121.20-dashboard.key 4096
Generating RSA private key, 4096 bit long modulus (2 primes)
....................................................................................................................................++++
....................................................................++++
e is 65537 (0x010001)
Enter pass phrase for 192.168.121.20-dashboard.key:
Verifying - Enter pass phrase for 192.168.121.20-dashboard.key:

Generate the signing request:

$ openssl req -new -key 192.168.121.20-dashboard.key \
  -addext "subjectAltName = IP:192.168.121.20" \
  -out 192.168.121.20-dashboard.csr

Enter pass phrase for 192.168.121.20-dashboard.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:SE
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:MyCity
Organization Name (eg, company) [Default Company Ltd]:MyCompany
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:192.168.121.20
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Copy the signing request to the CA:

$ scp 192.168.121.20-dashboard.csr <ca-server>:.

Sign the CSR. It can be signed according these instructions if you have your own private CA.

Get the signed certificate from the CA:

$ scp <ca-server>:192.168.121.20-dashboard.crt .

Remove the password from the key file:

$ openssl rsa -in 192.168.121.20-dashboard.key -out 192.168.121.20-dashboard-unencrypted.key
Enter pass phrase for 192.168.121.20-dashboard.key:
writing RSA key

Store the password free key and certificate at a location where it will be installed into the Kubernetes cluster:

$ cp 192.168.121.20-dashboard-unencrypted.key ~/certs/dashboard.key
$ cp 192.168.121.20-dashboard.crt ~/certs/dashboard.crt

To update the Kubernetes Dashboard certificate, it is easiest to just uninstall it and install it again with the new key and certificate. The procesure to install Kubernetes Dashboard is covered by a separate instruction called Install Kubernetes Dashboard.