While Docker popularized the concept of containerization to the world, its network interface assumed a single host environment, a situation which most often does not reflect reality.
Modern architectures employ a multitude of containers, services, and endpoints, possibly deployed across multiple datacenters, hybrid-cloud or even multi-cloud infrastructures.
Kubernetes, as an infrastructure orchestration system, defined concepts on how containers and pods should interact, but left out the implementation instead, relying on a plug-in interface defined by the Container Network Interface (CNI).
Between the multiple options implementing the CNI, Calico is well known for its performance, flexibility, and advanced network features. Project Calico is a good choice for environments that support its requirements and when performance and features like network policy are important.
Docker and Single Host Containers Network Configurations Options
Before Docker, the applications and their required components resided directly on physical systems.
System management and networking was done at the OS level which would allow an administrator to configure systems in such a way that the application could be isolated and yet still be able to communicate in a secure fashion.
With Docker, we are now able to package the software and related dependencies into a dedicated container. This provides numerous advantages ranging from isolation of the software itself, higher utilization of the underlying compute resource, and the ability to run the same software and configuration on multiple systems.
Regardless of the number of containers running on a system, they still need to be able to talk with each other and the outside world. Docker supports the following network configurations to provide this functionality:
- Bridge mode: Allow for docker containers to communicate amongst themselves
- Host mode: Utilizes the host’s networking stack for communication
- Overlay mode: Allows for containers to talk cross systems
- macvlan mode: Assigns a MAC address to the container allowing it to act as a pseudo-vm
- none: No networking whatsoever
- miscellaneous: This consists of plug-ins to Docker through their store
Each networking configuration has its advantages and disadvantages.
In bridge mode, there is a default bridge that acts differently from a user defined bridge. This difference is due to a need to explicitly specify which ports to open up between containers versus exposing all ports sharing the same bridge.
In host mode, ports being exposed by the container would be exposed to all network ports on the system.
Overlay mode, which requires Docker Swarm, exposes several ports to assist with managing the interfaces.
The macvlan approach offers the most flexibility, however most systems are restricted in the number of unique MAC addresses per host.
Kubernetes Networking Principles and the CNI
In the single host context, you are constrained in the number of available ports that would need to be coordinated or IP addresses to manage.
What happens when there is a need to scale beyond a single system? Even when keeping containers on a single system, one can easily imagine how the container configurations can become quite tedious when scaling to multiple services that will expose their own collection of ports, may need to talk to each other, or even need to be isolated from each other.
Kubernetes attempts to address this by adding a higher level abstraction which provides a mechanism allowing for containers to communicate with each other across system (and even network) boundaries.
Kubernetes does this by adhering to several properties:
- All containers can communicate with each other without a NAT
- All nodes can communicate with all containers without a NAT
- The IP address the container sees is the same that others see
At the lowest level of a pod, which can have multiple containers running inside, Kubernetes provides a single real IP address through a virtual network device and treat all containers running inside the pod as if they are running on a single host.
The IP addresses being created inside the Kubernetes cluster denote a private routable subnet that is accessible within the rest of the Kubernetes cluster regardless of where the pod happens to reside.
To expose the applications hosted inside Kubernetes to the outside world, there is the concept of a service which defines which ports should be load balanced to their respective pods through an ephemeral IP address.
Each Kubernetes node runs a daemon called kube-proxy that handles some of the port mapping and forwarding between the host’s network interface and the underlying containers.
Bear in mind, this is an edge proxy for the Kubernetes network. For the rest of the Kubernetes networking, Kubernetes provides a plug-in infrastructure that relies on the Container Network Interface CNI.
The CNI makes it possible to exposes a common interface for Kubernetes to handle the network related operations between containers while being able to leverage the underlying virtualization platform’s unique implementation.
This will come into play with components such as VPCs as implemented by Amazon and Google’s Compute Engine where network resources are managed by the cloud provider. It can also come in with self-managed platforms such as Infoblox.
Note that Kubernetes allows for a lot of flexibility when it comes to managing the networking fabric in the cluster. Each underlying infrastructure will have its limits such as the number of addresses one can have in their VPCs, configuring high availability between different availability zones, or even per subnet and port firewalling.
This is where a dedicated Software Defined Network (SDN) can come into play, and one of the common SDNs in use with Kubernetes is Calico.
Enabling Application Connectivity and Network Policy With Calico
Project Calico provides a Level-3 software defined gateway combined with a network policy engine.
The purpose behind Calico is to provide a virtual network that can securely scale between 10s and 100s of thousands of systems regardless of whether they are containers, virtual machines, or bare metal systems.
Calico can be used on its own, or combined with Flannel. In either case, Calico provides a network policy layer that can be used to define and apply ACLs and routing rules.
Calico also has a plug-in style architecture for their interface to the higher level clustering environment via an orchestrater plug-in.
These plug-ins implement the CNI standard and act as the glue between Kubernetes (or your other higher level clustering environment) and Calico itself.
Calico’s Routing and Gateway Management
Installed on each node is a Calico daemon called Felix which, roughly similar to kube-proxy, is responsible for programming the routes and ACL rules at the operating system level. Felix is responsible for monitoring the overall health of the network configuration.
In addition to Felix, Calico relies on etcd to act as a datastore for the overall network state data or etcd can act as a cache of a master view held in a chosen cloud provider.
It also requires some kind of Borderline Gateway Protocol (BGP) Router to distribute the appropriate routing information across the network.
By default Calico uses BIRD and maintains their own fork, however the implementer can chose their own client.
For larger networks, the implementer may require a BGP reflector, for which BIRD is also used by default.
As a Software defined network, Calico can be used to provide an interconnect layer between a Kubernetes Cluster (or any cloud provider for that matter) and any other network the implementer needs to talk to.
How Calico does this is by encapsulating the IP packet from the underlying container interface to masquerade as coming from the host itself. From there, the packet is routed to the target node where Calico will decapsulate the packet and forward it to the target node’s underlying container interface.
This IPIP and BGP mechanism allows for a small number of direct routes between the nodes themselves to be specified in the router, and offloads the container/pod specific subnet routes to reside in the nodes themselves speeding up route resolution and reducing the amount of overhead in the underlying cloud infrastructure.
This is where the real power of Calico comes out.
One other place where Calico shines is in defining the network policy such as which ports are accessible to what subnets, or even which subnets are routable to other subnets. All in a layer that sits above the underlying cloud or data center providing a centralized source of truth.
Why Choose Calico
While Docker popularized the concept of containerization to the world however its network interface assumed a single host environment.
Kubernetes, as an infrastructure orchestration system, defined concepts on how containers and pods should interact, but left out the implementation instead relying on a plug-in interface defined by the CNI.
Between the multiple options implementing the CNI, Calico provides one common implementation of the networking fabric that can allow it to meet the needs of not just Kubernetes, but the underlying cloud provider, and the administrator or network engineer that needs to tie sometimes very different pieces of infrastructure together.
Calico is currently used internally on Exoscale: it allows us to interconnect our multiple Kubernetes clusters via our existing network via BGP, and to benefit from an increased security layer by leverage Network Policies.
When you want to be able to control your network and enforce tight policies at scale without sacrificing performance Calico makes for a great choice.
References
- Kubernetes Networking Design: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/network/networking.md
- Kubernetes Networking Policy Guide: https://kubernetes.io/docs/concepts/services-networking/network-policies
