TLS in Kubernetes

Intro TLS Certificates Generating TLS Certificates CA Certificate Client Certificate for Admin User Client Certificate for K8s Components Server Certificate for K8s Components Using Client Certificates View certificate details Certificates API

Intro

The communication between the different nodes in the cluster, between the different components of k8s, between a user and the cluster, etc. must be encrypted using TLS.

There must be at least one CA in the cluster for signing TLS certificates.

The CA private key must be stored on a secure server (CA server). Anyone who has access to it can create as many authenticated users as they want. When setting up the cluster using KubeAdmin, it is stored on the master node. So, in this case, the master node is also the CA server.

TLS Certificates

KubeAPI Server,

ETCD, and

Kubelet act like servers that are contacted by clients. So, they generate server certificates.

Kube Scheduler,

Kube Controller Manager and

Kube Proxy, contact the KubeAPI server. So, they generate client certificates.

The KubeAPI server also contacts ETCD and Kubelet for which it can either use its server certificates or generate a client certificate for itself. Similarly, the Kubelet service also contacts the KubeAPI server for which it can generate its own client certificate or use its server certificate.

If a user has to access the cluster via the KubeAPI server, they will need to generate their own client-side certificates.

💡

Whenever a server and client communicate, they both need a copy of the CA root certificate to verify each other’s certificate as every other certificate is signed by the CA.

Generating TLS Certificates

When setting up the cluster from scratch, you need to generate and configure each certificate manually. When setting up the cluster using KubeAdmin, it automatically generates and configures the certificates. All the certificates are present at /etc/kubernetes/pki.

CA Certificate

Generate CA’s private key - openssl genrsa -out ca.key 2048

Generate a CSR (certificate without the signature) using the private key - openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr where /CN is the common name field (which component in k8s we are creating certificates for). The command will output a ca.csr file.

Sign the CA’s CSR using its own private key generated in the first step (self-signing) - openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt. The command will output the self-signed CA certificate ca.crt.

Now that we have the CA private key and the ca.crt certificate. They both can be used together to sign other CSRs in the cluster.

Client Certificate for Admin User

Generate admin user’s private key - openssl genrsa -out admin.key 2048

Generate a CSR for the admin user - openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr. /O=system:masters adds the user to the system:masters group to provide them with admin privilege.

Sign the admin’s CSR using the CA’s private key and certificate - openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

Client Certificate for K8s Components

Generating client TLS certificates for K8s components follows the same procedure. However, the common name /CN field must be prefixed with system:. The /CN field should be as follows for the following:

Kube Scheduler - system:kube-scheduler

Kube Controller Manager - system:kube-controller-manager

Kube Proxy - system:kube-proxy

Kubelet - system:node:<node-name>

The Kubelet service needs to act as a client to connect with the API server. For that, it needs to generate separate certificates for each node (kubelet service runs on every node). For this the certificate needs to be a part of system:nodes group.

Eg. to create a CSR for node-1 - openssl req -new -key node-1.key -subj "/CN=system:node:node-1/O=system:nodes" -out node-1.csr

Server Certificate for K8s Components

Generating server TLS certificates for k8s components follows the same procedure. Since the servers run continuously, we need to configure the key and certificate along with the CA certificate before starting the server.

The /CN field should be as follows for the following:

ETCD - etcd-server

The key, certificate and the CA certificate need to be configured while starting the ETCD server.

Peer certificates need to be configured when ETCD server has multiple instances for high availability.

KubeAPI Server - kube-apiserver

The KubeAPI server is referred to by many names. All these names along with the pod’s IP running it must be specified under the alt_names section of the openssl.cnf file, when generating the CSR.

openssl req -new -key apiserver. key -subj "/CN=kube-apiserver" -out apiserver.csr -config openssl.cnf

The key, certificate and the CA certificate needs to be configured while starting the KubeAPI server. We also need to specify the key, certificate and the CA certificate when the KubeAPI server acts as a client to connect to the ETCD server and Kubelet service.

Kubelet - <node-name>

The kubelet service runs on every node. So, separate key-certificate pairs must be created for every node. Since the Kubelet is an HTTP server, the key and certificate must be configured before starting it.

Using Client Certificates

When making an API request to the KubeAPI server, the HTTP client (curl in this case) needs the admin key and certificate to authenticate the request. It also needs the CA certificate to validate the server’s certificate.


curl https://kube-apiserver:6443/api/v1/pods \
--key admin.key \
--cert admin.crt \
--cacert ca.crt

Otherwise, the certificates and keys can be configured in

KubeConfig from where kubectl can automatically pick it and attach it with every request.


apiVersion: v1
clusters:
- cluster:
		certificate-authority: ca.crt
		server: https://kube-apiserver:6443
	name: kubernetes
kind: Config
users:
- name: kubernetes-admin
	user:
		client-certificate: admin.crt
		client-key: admin.key

View certificate details

Run the openssl x509 -in certificate.crt -text -noout command to view the certificate details including the subject, validity, issuer and alternative DNS names and IP addresses.

Certificates API

K8s has a built in certificates API to easily sign and manage certificates. This is handled by the

Kube Controller Manager using the CA certificate and key.

Let’s consider the case where a new user needs access to the cluster. She’ll first create a private key and then a CSR (CN=<user-name>) using openssl commands. She’ll then send the .csr file to the cluster admin. The admin will create a CertificateSigningRequest object using the base64 encoded contents of the .csr file (generate using cat filename.csr | base64 -w 0) along with the info about the groups and usages.


apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
	name: <user-name>
spec:
	signerName: kubernetes.io/kube-apiserver-client
	groups:
		- system:authenticated
	usages:
		- digital signature
		- key encipherment
		- server auth
	request: <base64-encoded-csr>

All the CertificateSigningRequest objects can be viewed by the cluster admins using k get csr command. Then admin can approve any CSR by running k certificate approve <csr-name>.