---
title: "HAProxy Elastic IP Automatic failover"
description: "HAProxy Elastic IP Automatic failover. Cloud-native failover with exoip."
date: 2017-02-07
author: "Pierre-Yves Ritschard"
url: https://www.exoscale.com/blog/haproxy-elastic-ip-automatic-failover/
---
# HAProxy Elastic IP Automatic failover

> [!NOTE]
> **Ubuntu Bionic Beaver changes**
>
> Ubuntu Bionic deprecates `ifupdown` in favor of
> [netplan.io](https://netplan.io) (see [Bionic release
> notes](https://wiki.ubuntu.com/BionicBeaver/ReleaseNotes#Network_configuration)).
> The _loopback_ interface configuration has been updated [within our
> documentation](https://community.exoscale.com/documentation/compute/eip/).
> The *up* and *down* hooks may also be achieved via `networkd-dispatcher` as
> explained on the netplan FAQ entry: [Use pre-up, post-up, etc. hook
> scripts](https://netplan.io/faq#use-pre-up-post-up-etc-hook-scripts)

> A look a the canonical Elastic IP use-case: highly available load balancers

Exoscale recently introduced Elastic IP addresses, which provide
a way to share a reserved IP address across several instances.
In this article we will explore the canonical use of Elastic IPs:
a shared IP on two load-balancers with automatic failover. We will
make use of the Exoscale API to infer most configuration details.

Elastic IP addresses can be held by any number of instances, but in
the case of instance failure, they must be revoked through the
Exoscale API. Automatic failover typically involves using tools such as
[keepalived](https://www.keepalived.org/),
[heartbeat](http://www.linux-ha.org/wiki/Heartbeat), and
[corosync](http://corosync.github.io/corosync/).

In this article, we will take advantage of our own tool, 
[**exoip**](https://github.com/exoscale/exoip)
to ensure liveness of load-balancer instances and to ensure the
Elastic IP is held by a functioning instance.
We will perform the following, step-by-step:

1. [Defining our sample architecture](#defining-our-sample-architecture)
1. [Setting-up security groups](#setting-up-security-groups)
1. [Creating instances](#creating-instances)
1. [Application server configuration](#application-server-configuration)
1. [Load-balancer configuration](#load-balancer-configuration)
1. [Reserving an Elastic IP address](#reserving-an-elastic-ip-address)
1. [Automatic failover](#automatic-failover)
1. [Setting-up **exoip**](#setting-up-exoip)

## Defining our sample architecture

For the purpose of the article, let's assume we have 2 load-balancers
distributing requests to 3 application servers:

- lb1.example.com: 198.51.100.11
- lb2.example.com: 198.51.100.12
- app1.example.com: 198.51.100.21
- app2.example.com: 198.51.100.22
- app3.example.com: 198.51.100.23


Both load-balancer and application server instances will expose
their service on port 80. Setting-up a proper TLS environment is
beyond the scope of this article. For an in-depth description of how
to correctly deploy a secure TLS web serving environment,
[this article from NGINX](https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/)
is a must-read.

![architecture](https://www.exoscale.com/blog/haproxy-elastic-ip-automatic-failover/archi1.png)

Through-out the article, we will assume that DNS records exist for the
above list of instances. Setting-up records won't be covered in this article. 
For an in-depth description on how to set-up DNS zones and records, you may
refer to the Exoscale [DNS documentation](https://community.exoscale.com/documentation/dns/).

Setting-up the `default` security group is not described below, for an introductory
guide on how to do initial security group configuration, please refer to the [security group documentation](https://community.exoscale.com/documentation/compute/security-groups/).

## Setting-up security groups

We will start by taking care of firewalling between instances by creating rules in
three security groups: `default`, `load-balancer`, and `app-server`:

- `default` will authorize `SSH`, and `ICMP` access to all instances.
- `load-balancer` will allow external access to port 80.
- `app-server` will allow port 80 access from the `load-balancer` security group.

The `load-balancer` security group contents should be as follows:

![load balancer security group](https://www.exoscale.com/blog/haproxy-elastic-ip-automatic-failover/sg-lb1.png)

And the `app-server` security group contents should be as follows:

![app server security group](https://www.exoscale.com/blog/haproxy-elastic-ip-automatic-failover/sg-app.png)

Using our [cs](https://github.com/exoscale/cs) utility, rules may be provisioned as follows
(provided an `EXOSCALE_ACCOUNT` environment variable is set to the target organization's name):

```
cs createSecurityGroup name=load-balancer \
                       description="Load balancer instances"
                       
cs createSecurityGroup name=app-server \
                       description="Application server instances"
                       
cs authorizeSecurityGroupIngress startPort=80 \
                                 endPort=80 \
                                 protocol=TCP \
                                 cidrlist=0.0.0.0/0 \
                                 securitygroupname=load-balancer
                                 
cs authorizeSecurityGroupIngress startPort=80 \
                                 endPort=80 \
                                 securitygroupname=app-server \
                                 protocol=TCP \
                                 'usersecuritygrouplist[0].account'=$EXOSCALE_ACCOUNT \
                                 'usersecuritygrouplist[0].group'=load-balancer
```

## Creating instances

Now that our firewalling configuration is ready, we can create instances as described above.
We will create instances using *Ubuntu 16.04* as the operating system template.

This gives us the following instance list:

![instance list](https://www.exoscale.com/blog/haproxy-elastic-ip-automatic-failover/instance_list.png)


## Application server configuration

For the purpose of the article, we will stick to the basics and our
application server will not serve much of an application... On `app1`,
`app2`, and `app3` we will simply install `nginx` and run the
corresponding service:

```
sudo apt install nginx
```

## Load-balancer configuration

Our load-balancer instances will run **HAProxy**, first, let's install it on both
`lb1` and `lb2`:

```
sudo apt install haproxy
```

**HAProxy** comes bundled with good defaults in `/etc/haproxy/haproxy.cfg`.
We will keep those and add our additional configuration at the end of the file:

```
cat << EOF | sudo tee -a /etc/haproxy/haproxy.cfg
listen web
  bind 0.0.0.0:80
  mode http
  server app1 app1.example.com:80 check
  server app2 app2.example.com:80 check
  server app3 app3.example.com:80 check
EOF
```

Once the configuration is provisioned on both `lb1` and `lb2`, **HAProxy** will need to be reloaded:

```
sudo systemctl reload haproxy
```


We can now verify that we are able to see the default **nginx**
landing page on the IPs held by `lb1` and `lb2`.  At this point, we
have a valid load-balancing configuration to our three application
servers.

Up until now, DNS records pointing to both `lb1`
and `lb2` as well as relying n HTTP clients' DNS resolving libraries were
needed to perform failover for H/A.

For the remainder of this article, we will focus on using Exoscale
Elastic IP addresses and our [**exoip**](https://github.com/exoscale/exoip)
tool to host both of these load-balancers behind a single IP address,
with automatic failover in case of failure.

## Reserving an Elastic IP address

Allocating a new Elastic IP address is a simple operation from our
console or API. For details about how to perform console operations
pleaser refer to
[the EIP documentation](https://community.exoscale.com/documentation/compute/eip/).

![elastic ip](https://www.exoscale.com/blog/haproxy-elastic-ip-automatic-failover/elastic_ip.png)

At Exoscale, Elastic IPs are additional IPs which need to be
be provisioned on instances.
To help share a single Elastic IP across
several instances without worrying too much about configuration, we have
written a small tool called [**exoip**](https://github.com/exoscale/exoip).

## Automatic failover

Now that we have an Elastic IP, we can assign it to any or all of
our load-balancers, but we need to take care of removing it from
instances in the case of failure. In practice, this would incur
downtime and be burdensome.

Fortunately, there are tools available to provide automatic failover
of IP assignment.  In the Linux world, popular tools include
[keepalived](https://www.keepalived.org/),
[heartbeat](http://www.linux-ha.org/wiki/Heartbeat), and
[corosync](http://corosync.github.io/corosync/).

A typical approach would be to couple one of these tools with scripts
to be ran during state transitions. While these tools are good
solutions, they are generic and do not take advantage of the API
capabilities a cloud environment can offer. On the other-hand, the
network protocol used to ensure peer liveness and perform state
transitions is very simple and well documented.

To ease the configuration of automatic failover, we have introduced
[**exoip**](https://github.com/exoscale/exoip) which implements a protocol
similar in nature to
[VRRP](https://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Protocol)
or
[CARP](https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol), but
takes advantage of the Exoscale API to infer most configuration.

The basic idea behind **exoip** is that any number of hosts may
participate in the ownership of an Elastic IP. When hosts fail to
report their status, their Elastic IP association is revoked by
other cluster members. Additionally, **exoip** provides support for the
most common use-case: when all owners of an Elastic IP share a common
Security Group.

## Setting-up **exoip**

By default, **exoip** uses UDP port 12345 to communicate, so we will
add this to our `load-balancer` security group:

```bash
cs authorizeSecurityGroupIngress startPort=12345 \
                                 endPort=12345 \
                                 securitygroupname=load-balancer \
                                 protocol=UDP \
                                 'usersecuritygrouplist[0].account'=$EXOSCALE_ACCOUNT \
                                 'usersecuritygrouplist[0].group'='load-balancer'
```

Now, let's retrieve **exoip**, which is available at
https://github.com/exoscale/exoip. The latest release
is always available at https://github.com/exoscale/exoip/releases/latest:

```bash
VERSION=0.3.6
wget https://github.com/exoscale/exoip/releases/download/$VERSION/exoip
```

Trusting executables directly fetched from the internet is never a great idea,
we can verify **exoip** comes from a trusted source using the following:

```
wget https://github.com/exoscale/exoip/releases/download/$VERSION/exoip.asc

gpg --recv-keys E458F9F85608DF5A22ECCD158B58C61D4FFE0C86
gpg --verify exoip.asc
```

Provided the signature verification process succeeded, the binary can
be moved to an executable location:

```
sudo mv exoip /usr/local/bin
sudo chmod +x /usr/local/bin/exoip
```

Once **exoip** is installed all is left to do is to configure a new
interface, let's put its configuration in
`/etc/network/interfaces.d/51-exoip.cfg`:

```
auto lo:1
iface lo:1 inet static
  address 198.51.100.50
  netmask 255.255.255.255
  exoscale-peer-group load-balancer
  exoscale-api-key EXO.......
  exoscale-api-secret LZ.....
  up /usr/local/bin/exoip -W &
```

This ensures the following steps are taken:

- Configure a new *loopback*, holding our allocated IP.
- Use members of the `load-balancer` security group as peers.
- Configure Exoscale API credentials.
- Launch **exoip** in watchdog mode when this interface comes up.

The interface may now be brought up:

```
sudo ifup lo:1
```

We can confirm in the console that our *EIP* is correctly assigned:

![elastic ip assignment](https://www.exoscale.com/blog/haproxy-elastic-ip-automatic-failover/eip_assign.png)

We can now try stopping and starting instances to see the assignment
operation performed by **exoip**.

**exoip** logs all its operations through *syslog*, we can access its event log with `journalctl`:

```
journalctl -xe -t exoip
```

## Wrapping-up

With this guide, we described a simple way of setting-up robust
automatic failover for load-balancers or other applications needing to
share IPs.

It must be said that **exoip** is only one tool, and will not cater to
all possible use-cases. When implementing automatic failover, other
approaches should also be investigated to find the best fit. In any case,
**exoip** might come in handy since it can also be ran to perform one-off associations and dissociations
between Elastic IPs and instances.

We also have written in-depth documentation on subjects pertaining to EIP:

- [EIP section in our documentation](https://community.exoscale.com/documentation/compute/eip/)
- API Documentation for [addresses](https://community.exoscale.com/api/compute/#addresses) and [NICs](https://community.exoscale.com/api/compute/#nic)
- **exoip** [documentation](https://github.com/exoscale/exoip)

That being said, we understand that you would rather focus on writing your
application than configuring **HAProxy**. Rest assured that we work hard
to expand our catalog of network services this year to provide you
with a programmable load-balancing service. Stay tuned!