---
title: "Deploy ELK with Docker"
description: "Capturing and processing your application's logs is a job for specialised tools. Here we look at how to deploy the ELK stack using Docker and configure your application's container to send its logs to Elasticsearch."
date: 2016-07-11
author: "Matthieu Buffenoir"
url: https://www.exoscale.com/blog/elk-docker/
---
# Deploy ELK with Docker

When developing a web app, I like to refer to the [twelve factor app](https://12factor.net/) recommendations. Their approach to handling application logging is in the eleventh "commandment":

> *It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout.*

It seems that the folks at [Docker](https://www.docker.com/) also read this manifesto and thus provided us with great tools to manage logs produced by applications in containers. In this tutorial we are going to see how to use those tools to collect logs and manage them with Elasticsearch + Logstash + Kibana (aka ELK).

![Deploy ELK with Docker](https://www.exoscale.com/blog/elk-docker/cover.png)

## What you'll learn

In this post I'll cover:

 * A quick description of ELK.
 * How to create the machines you need on Exoscale account: one to host the ELK stack and a second to host a demo web application.
 * How to start ELK using Docker compose.
 * How to use the Docker logging driver to route the web application logs to ELK.

## Prerequisites

**Mac users:** you'll need the latest [Docker Desktop](https://www.docker.com/products/docker-desktop/).

**Linux users:** you'll need Docker [engine](https://docs.docker.com/engine/install/), [machine](https://docs.docker.com/desktop/) and [compose](https://docs.docker.com/compose/install/).

On both systems you'll need to install Exoscale's `cs` tool, which you can grab with the Python ecosystem's `pip`:

    pip install cs

We'll use `docker-machine` to create the instances we need on Exoscale and then `cs` to modify the security group to allow connections to our newly installed ELK stack. 

`cs` and `docker-machine` both relying on the Exoscale API, so they will need to access your [API credentials](https://portal.exoscale.com/account/profile/api).

Export the credentials in your shell with the following:

```bash
export EXOSCALE_ACCOUNT_EMAIL=<your exoscale account email>
export CLOUDSTACK_ENDPOINT=https://api.exoscale.ch/compute
export CLOUDSTACK_KEY="<your exoscale API Key>"
export CLOUDSTACK_SECRET_KEY="<your exoscale Secret Key>"
```

Lastly, you'll need a few files that I've provided in a git repo. Clone it locally:

```bash
git clone https://github.com/MBuffenoir/elk.git
cd elk
```

## First up, what is ELK?

[ELK](https://wikitech.wikimedia.org/wiki/Logstash) is a collection of open source tools, maintained by [Elastic.co](https://www.elastic.co/) that combines:

 * **Elasticsearch:** an indexer and search engine that we will use to store our logs.
 * **Logstash:** a tool that collects and parses the logs, then stores them in Elasticsearch indexes.
 * **Kibana:** a tool that queries the Elasticsearch REST API for our data and helps to create insightful visualizations.

## Creating instances

Let's use `docker-machine` to create an Ubuntu instance on Exoscale that will host ELK:

```bash
docker-machine create --driver exoscale \
    --exoscale-api-key $CLOUDSTACK_KEY \
    --exoscale-api-secret-key $CLOUDSTACK_SECRET_KEY \
    --exoscale-instance-profile small \
    --exoscale-disk-size 10 \
    --exoscale-security-group elk \
    elk
```

This will take a little time to run and it will give you a result similar to this:

```
Running pre-create checks...
Creating machine...
(elk) Querying exoscale for the requested parameters...
(elk) Security group elk does not exist, create it
[...]
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
```

For this demo I've mostly used the default parameters, but of course you can customize this to fit your needs. Take a look at the [driver documentation](https://docs.docker.com/desktop/) to learn more. You can also scale up your instance later (either manually through the web interface or using `cs` and the Exoscale API).

At this point you should be able to connect to the remote Docker daemon with:

```bash
eval $(docker-machine env elk)
```

Let's make sure that everything is okay:

```bash
docker version
docker info
```

Now let's create a second machine on which we will run the demo application:

```bash
docker-machine create --driver exoscale \
    --exoscale-api-key $CLOUDSTACK_KEY \
    --exoscale-api-secret-key $CLOUDSTACK_SECRET_KEY \
    --exoscale-instance-profile tiny \
    --exoscale-disk-size 10 \
    --exoscale-security-group demo \
    demo
```

`docker-machine` has already opened all the necessary ports for you to connect to your new remote Docker daemon. You can review those ports in the newly created elk [security group](https://portal.exoscale.com/compute/firewalling).

Now you need to open the ports necessary to access your running applications and have them communicate together.

Let's use `cs` to do that. We'l; start with opening the port to connect to Kibana:

```bash
cs authorizeSecurityGroupIngress protocol=TCP startPort=5600 endPort=5600 securityGroupName=elk cidrList=0.0.0.0/0
```

Now let's enable our application to send its logs to Logstash:

```bash
cs authorizeSecurityGroupIngress protocol=UDP startPort=5000 endPort=5000 securityGroupName=elk 'usersecuritygrouplist[0].account'=$EXOSCALE_ACCOUNT_EMAIL 'usersecuritygrouplist[0].group'=demo
```

And finally, we'll open port 80 to our demo web server:

```bash
cs authorizeSecurityGroupIngress protocol=TCP startPort=80 endPort=80 securityGroupName=demo cidrList=0.0.0.0/0
```

## Launching ELK

Now that our instances and security group are ready to use, it is time to launch ELK.

First, you will need to upload the configuration files needed by ELK. We'll do that with:

```bash
docker-machine scp -r conf-files/ elk:
```

Note that the certificate used here is self-signed. If you prefer you can replace or regenerate it with:

```bash
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./kibana-cert.key -out ./kibana-cert.crt
```

Now you can connect to your remote Docker daemon and start ELK with:

```bash
eval $(docker-machine env elk)
docker-compose -f docker-compose-ubuntu.yml up -d
```

Enjoy the beauty of a job well done with:

```bash
docker-compose -f docker-compose-ubuntu.yml ps
```

Which should give you an output similar to this one:

```bash
    Name                   Command               State                     Ports
------------------------------------------------------------------------------------------
elasticdata     /docker-entrypoint.sh chow ...   Exit 0
elasticsearch   /docker-entrypoint.sh elas ...   Up       9200/tcp, 9300/tcp
kibana          /docker-entrypoint.sh kibana     Up       5601/tcp
logstash        /docker-entrypoint.sh -f / ...   Up       0.0.0.0:5000->5000/udp
proxyelk        nginx -g daemon off;             Up       443/tcp, 0.0.0.0:5600->5600/tcp, 80/tcp
```

You can see here the three applications that constitute the ELK stack and a reverse nginx proxy that secures them.

##Running a demo web server and sending its logs to ELK

Let's use Docker again to run a simple nginx web server that will simulate a typical web application:

```bash
eval $(docker-machine env demo)
docker run -d --name nginx-with-syslog --log-driver=syslog --log-opt syslog-address=udp://$(docker-machine ip elk):5000 -p 80:80 nginx:alpine
```

Ensure your web server is running properly with:

```bash
docker ps
```
Connect to it with your favorite web browser or by using curl:

```bash
curl http://$(docker-machine ip demo)
```

##Using Kibana to view the logs

Now is the moment you've been waiting for! 

Browse to your ELK installation: `https://<elk instance ip>:5600`

The default username and password are `admin` / `Kibana05`. 

As this is the first you're connecting to it, it will take a few seconds to get set up. 

Once it's ready, create your first index by hitting the green `create` button and then the `Discover` tab. You should now be able to see the logs of your nginx demo web server:

![image](https://www.exoscale.com/blog/elk-docker/kibana-dashboard.png)

##Conclusion

Here we've seen here the minimum you need to collect logs with ELK and Docker.

The next steps might be to enhance your Logstash configuration to detect special fields in your logs, maybe prune some sensitive information and, of course, create nice visualizations in Kibana. We can take a look at that another time. 

Have fun collecting logs!