--- title: "How to Install Calico on Kubernetes or Bare Metal" description: "How to Install Calico on Kubernetes or Bare Metal" date: 2019-10-31 author: "Chris Baumbauer" tags: ["network"] url: https://www.exoscale.com/blog/how-to-install-calico/ --- # How to Install Calico on Kubernetes or Bare Metal **Quick summary of this article** - Install and configure calicoctl and etcd v3, with kubectl optional, Kubernetes datastore preferred for small clusters. - Provide a calicoctl configuration specifying datastoreType and endpoints or kubeconfig, use --config or environment variables. - For bare metal deployments use etcd v3, single-node is possible but lacks redundancy and needs secure systemd. - Install Calico Felix via package, Docker image, or binary, ensure ipset is installed and Felix registers in etcd. - Apply the Calico manifest after adjusting POD_CIDR, remove master taint if needed, validate with iptables and calicoctl. In our recent [introduction to Calico](https://www.exoscale.com/blog/calico-introduction/) we have gone trough a very high level overview. This time we'll go more in depth, installing Calico itself. While there are several getting started guides on [Calico's website](https://projectcalico.docs.tigera.io/about/about-calico), we'll attempt to combine them into a single guide to show how everything fits together. ![A complete guide for install Calico on Kubernetes or Bare Metal](https://www.exoscale.com/blog/how-to-install-calico/exoscale-install-calico.jpg) ## Prerequisites to Install Calico Regardless of how Calico will be used within the installation, there are several common prerequisites: 1. `calicoctl` needs to be installed and configured 1. `etcd v3` may need to be installed and configured 1. `kubectl` is an optional dependency to interact with a Kubernetes cluster Note in the case for Kubernetes, it is recommend that the Kubernetes datastore is used in lieu of etcd for small clusters. Otherwise, etcd can be used. This article will assume a single `etcd` instance for the examples cited that look at an external etcd installation. ## The Calico CLI The `calicoctl` interface can be downloaded from [Calico's project page](https://github.com/projectcalico/calicoctl/releases). Optionally, Project Calico provides a [Docker image](https://projectcalico.docs.tigera.io/getting-started/bare-metal/installation/container) and [Kubernetes manifest](https://docs.projectcalico.org/v3.7/getting-started/kubernetes/installation/calico) which can be installed in a target environment where direct access may be difficult to obtain. In any case, a Calico configuration file is required for the `calicoctl` CLI to be usable, and the configuration file must contain enough information to allow `calicoctl` to connect to the etcd cluster. A sample configuration file for etcd, assuming a single etcd host, would resemble the following: ```yaml apiVersion: projectcalico.org/v3 kind: CalicoAPIConfig metadata: spec: datastoreType: "etcdv3" etcdEndpoints: "http://localhost:2379" ``` > [!NOTE] > The `datastoreTime` will default to `etcdv3` if not present._ For the Kubernetes datastore, the configuration file would resemble: ```yaml apiVersion: projectcalico.org/v3 kind: CalicoAPIConfig metadata: spec: datastoreType: "kubernetes" kubeconfig: "/path/to/.kube/config" ``` While the default location for the client configuration file is in `/etc/calico/calicoctl.cfg`, the `--config` flag can be used to specify an alternate location. In addition, some of the `calicoctl` properties can be passed as an environment variable to ensure private data such as credentials are not exposed. ## Calico Datastore (etcd) If Calico will be managing the networking stack on a bare metal system or a dedicated Docker installation, then etcd will need to be installed and configured. For very small installations, a single node can be used, however there is the loss of redundancy in case of failure. Installing etcd will require [downloading](https://github.com/etcd-io/etcd/releases) the system appropriate archive, and extracting it. The archive will contain the etcd documentation as well as two binaries: `etcd` as the server component, and `etcdctl` as the control program. Assuming a simple bring-up scenario, to start `etcd` on a host called `etcd1`: ```bash $ export ETCDCTL_API=3 $ THIS_IP=0.0.0.0 # This really should be accessible to the Calico nodes only $ etcd --data-dir=data.etcd --name etcd1 \ --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \ --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \ --initial-cluster etcd1=http://${THIS_IP}:2380 \ --initial-cluster-state new --initial-cluster-token initaltoken ``` > [!NOTE] > This will start etcd in the foreground. To run it in a more permanent scenario, a systemd service manifest would need to be created. In addition, this is an insecure configuration. ## Calico Node (Felix) With the pre-requisites taken care of, the last thing to do is to install Felix, which is Calico's runtime service. This will be broken up into two scenarios: bare metal/Docker, and Kubernetes. > [!NOTE] > etcd is required to be reachable by Felix. ### Bare Metal and Docker Installation of Calico The bare metal installation can be done using Calico's [pre-built packages](https://projectcalico.docs.tigera.io/getting-started/bare-metal/installation/binary-mgr). Felix can be installed by hand using the Docker image using the following steps: ```bash $ docker pull calico/node:v3.7.2 $ docker create --name container calico/node:v3.7.2 $ docker cp container:/bin/calico-node calico-node $ docker rm container $ chmod +x calico-node $ sudo mv calico-node /usr/local/bin ``` Once the binary is copied and migrated into place, an init script will need to be created. Assuming systemd is used: ```systemd [Unit] Description=Calico Felix agent After=syslog.target network.target [Service] User=root Environment=ETCD_ENDPOINTS=http://127.0.0.1:2379 ExecStartPre=/bin/mkdir -p /var/run/calico ExecStart=/usr/local/bin/calico-node -felix KillMode=process Restart=on-failure LimitNOFILE=32000 [Install] WantedBy=multi-user.target ``` Lastly, before Felix can be started `ipset` needs to be installed on the node. In the above cases, Felix can be configured by creating a configuration file at `/etc/calico/felix.cfg`. Details on how to tune Felix can be found at: https://docs.projectcalico.org/v3.6/reference/felix/configuration. Note that by default, Felix is looking for etcd at `127.0.0.1:2379`. To run Felix inside a Docker container, run `calicoctl node run --node-image=calico/node:v3.6.1`. Lastly, Felix needs to be configured in etcd. While the Docker command will handle the automatic registration, the bare metal approaches require the Felix daemon to register itself upon first invocation: ```bash calicoctl create -f - < spec: bgp: ipv4Address: /24 EOF ``` Note that the name should reflect the unique hostname for the node. ### Install Calico on Kubernetes Installing Calico on Kubernetes can be simple or complex depending whether to use Calico for policy and network management or just policy. The installation consists of applying a Kubernetes manifest file against the cluster. By default, Calico assumes the internal IP address inside the Kubernetes cluster to be `192.168.0.0/24`. If this is not the case, then the Calico manifest will need to be updated: ```bash # To use the Kubernetes datastore manifest (recommended by the Calico Project) $ curl \ https://docs.projectcalico.org/v3.7/manifests/calico.yaml \ -O # Otherwise, for the etcd datastore manifest $ curl \ https://docs.projectcalico.org/v3.7/manifests/calico-etcd.yaml \ -O # This will be required for both $ POD_CIDR="" \ sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml $ kubectl apply -f ./calico.yaml ``` Lastly, depending on how the Kubernetes cluster is initialized, the taint mode on the master node may need to be removed. First, ensure that all pods are up, and then remove the taint flags. ```bash $ watch kubectl get pods --all-namespaces # ^C to kill the watch mode once all services are up # May be required if kubeadm was used to bring up a new Kubernetes cluster $ kubectl taint nodes --all node-role.kubernetes.io/master- ``` ## Validating the Calico Setup At this point, Calico should be up and running. If Calico is managing the network stack in a Kubernetes cluster, then the Kubernetes [NetworkPolicy API](https://kubernetes.io/docs/concepts/services-networking/network-policies/) can be used to manage Calico with `kubectl`. If you are running on bare-metal, you can confirm by running `sudo iptables --list`. A small snippet of the results can be seen: ``` vagrant@ubuntu:~/calico$ sudo iptables --list Chain INPUT (policy ACCEPT) target prot opt source destination cali-INPUT all -- anywhere anywhere /* cali:Cz_u1IQiXIMmKD4c */ Chain FORWARD (policy DROP) target prot opt source destination cali-FORWARD all -- anywhere anywhere /* cali:wUHhoiAYhphO9Mso */ DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination cali-OUTPUT all -- anywhere anywhere /* cali:tVnHkvAo15HuiPy0 */ Chain DOCKER (1 references) target prot opt source destination ``` Using the `calicoctl` command for either bare metal or Kubernetes: ``` ./calicoctl get nodes -o wide NAME ASN IPV4 IPV6 ubuntu (unknown) 192.168.16.96/24 ``` ## Next Up We've seen how to install Calico on either bare metal/Docker or Kubernetes. Now that your Calico setup is up and running we'll soon see how to deploy a multi-Zone deployment blueprint on Exoscale, implementing a resilient and redundant high availability architecture. [Follow Exoscale on Twitter](https://twitter.com/exoscale) to stay up to date! #### References 1. Calico configuration properties: [https://projectcalico.docs.tigera.io/reference/calicoctl/](https://projectcalico.docs.tigera.io/reference/calicoctl/) 2. etcd: [https://etcd.io](https://etcd.io)