Kubernetes Networking

Cluster (Node) Networking Pod Networking CNI Standards WeaveNet - Pod Networking Solution Service Networking Important Commands

Cluster (Node) Networking

Each node must have at least one interface connected to a common network. Each interface must have an IP address configured. Every node must have a unique hostname as well as a unique MAC address.

Various ports need to be opened (firewall and security group settings must be configured) on the master and worker nodes as shown in the diagram. The worker nodes expose services for external access on ports 30000 - 32767 (for NodePort services).

In a multi-master setup, the ETCD clients need to communicate on port 2380, so that needs to be open as well.

Pod Networking

Now that we have configured networking at the cluster level as explained in the previous section, all the nodes can communicate with each other each other. But we also need to establish networking at the pod level. K8s doesn’t come with a native pod networking solution. Instead it requires us to set it up ourselves.

We can use a CNI compatible networking solution (plugin) to achieve this. Basically, the CNI plugin ensures that every pod on the cluster (irrespective of the node it is running on), gets a unique IP address.

After setting up the networking solution:

Every pod in the cluster should have a unique IP Address.

Every pod should be able to communicate with every other POD on the same node by using IP addresses.

Every pod should be able to communicate with every other POD on other nodes by directly using their IP addresses, without using a NAT.

Networking solutions create a bridge network on each node with a different subnet mask and attach every pod to the bridge network on its node. This way, every pod in the cluster gets a unique IP address. Also, pods on the same node can now communicate with each other using IP addresses.

At this stage, pods cannot communicate across nodes. If the blue pod (10.244.1.2) tries to ping the purple pod (10.244.2.2), it won’t be able to since it is on a different subnet. It then routes the request to NODE1’s IP (192.168.1.11) as it is the default gateway for the bridge network on NODE1. Even NODE1 has no idea where the subnet 10.244.2.0/24 is as it is a private network on NODE2. So, we need a router configured as the default gateway on each node to tell them where to forward the requests going to pods on various bridge networks.

This way, the bridge networks running on each node coalesce together to form a large network with CIDR 10.244.0.0/16.

CNI Standards

Networking solutions for K8s must follow the Container Networking Interface (CNI) standards. CNI defines how networking solutions (plugins) should be developed and how container runtimes should invoke them. Docker does not support CNI. K8s uses a workaround to achieve networking when using Docker as the container runtime.

CNI is configured on the

Kubelet service. Here, we define the cni-bin-dir (default /opt/cni/bin) which contains the supported CNI plugins as executable binaries. We also define cni-conf-dir (default /etc/cni/net.d/) which contains the different config files for CNI to be used by Kubelet.

WeaveNet - Pod Networking Solution

WeaveNet is a CNI compatible networking solution plugin for K8s. Instead of configuring a router to route traffic between nodes, it deploys an agent on each node. These agents communicate with each other to share information about their node. Each agent stores the entire network topology, so they know the pods and their IPs on the other nodes.

Weave creates a separate bridge network on each node called weave and connects the pods to it. The pods are connected to the docker bridge network and the weave bridge network at the same time. Weave configures the routing on the pods to ensure that all packets destined for other nodes are delivered to the agent. The agent then encapsulates the packet and sends it to the agent running on the other node, which then decapsulates the original packet and sends it to the required pod.

Weave allocates the CIDR 10.32.0.0/12 for the entire cluster. This allows for IP range 10.32.0.1 to 10.47.255.254 (over a million IPs). Weave peers (agents) split this range among themselves.

Weave can be deployed as a deamonset on each node. For installation steps, refer Integrating Kubernetes via the Addon (weave.works) and

Service Networking

In a cluster, pods communicate with each other through services, using their DNS names or IP addresses, instead of using the pod’s IP which can change if it restarts. While a pod runs on a single node, a service spans the entire cluster and can be reached from any of the nodes. In the above image, the purple service is of type NodePort, which means it can be accessed at a given port from any node in the cluster.

Services are allocated IPs from a range configured in the KubeAPI server using the service-cluster-ip-range parameter. The service networking CIDR should not overlap with the pod networking CIDR.

Services in K8s are not resources like pods. When a service object is created, the KubeProxy running on each node configures the IPTable rules on all of the nodes to forward the requests coming to the IP:port of the service to the IP:port of the backend pod configured in the service.

Important Commands

Find which ports on the node is being used by what process - netstat -nplt

To find the interface used by k8s on the node, run ip link or ip address and search for node’s internal IP (found using k get nodes -o wide).

View the route table on a node - route

Find the interface of the bridge network created by the container runtime on a node - ip address show type bridge

To find the range of IP addresses for the nodes in the cluster, run ip address and deduce the range for eth0 interface.

To find the range of IP addresses for the pods in the cluster running a networking solution like WeaveNet, run ip address and deduce the range for weave interface.