Logging is a vital part of any IT infrastructure. Central logging gives you ease of use, availability and enhanced security. It is the same for your local network and for your virtual machines in the Exoscale cloud.

You can learn from this blog how to implement centralized logging for your machines at Exoscale and gives you a few hints about archiving and possible next steps.

cover image

Central logging

Most sysadmins understand/appreciate the importance of logging. Many times logging is required by law or compliance regulations. But even if you if you are not required to log your systems, you have no other ways to learn how your services are used or what problems your software has.

Logging is important, but why to log to a centralized location? The three keywords here are convenience, availability and security. Let me examine them one by one.

  • Convenience. There is a good chance that you have more than one machine running in the cloud. If there is a problem logging in to each of the machines and checking many different log files is inconvenient. And not just that, it slows down the debugging process considerably. Having all logs at a centralized location can speed up locating a problem.

  • Availability. Your server can sometimes crash or even be deleted when not needed any more. If you have the logs from the server you can use it to figure out what happened and start the recovery process accordingly or find traces of a security incident even long after a redundant machine is deleted.

  • Security. When a security breach occurs, more often than not, the perpetrators take steps to remove traces of the hack by deleting logfiles. But if you forward your log messages to a central server as they are created you have the evidence to figure out what happened.

Logging as a Service

Your machines are in the cloud. Why not utilize a Logging as a Service (LaaS) provider for centralized logging? The answer is not easy. There are many things to consider:

  • Getting started. If you use LaaS, all you need is to register yourself at one of the providers. Many even have a free tier if you have modest logging requirements. Just install a client or append a line or two to your syslog configuration and you are ready to go. If you do centralized logging yourself, you also need to setup a dedicated machine, install and configure software, a lot longer process.

  • Price. Except if you use just some basic, limited services, you need to pay to the LaaS provider. If you have lots of logs you also have to pay to Exoscale for the outgoing Internet traffic. If you create a centralized logging server at Exoscale you do not have to pay for the network traffic, but only for the virtual machine and storage used. And of course your time for setting up and maintaining the machine(s).

  • Regulations. GDPR and many other privacy regulations suggest or require that personal data never leaves the country or the EU. Log messages often contain sensitive personal data. Unfortunately I am not aware of any LaaS providers who can assure you that log messages do not cross the borders. From the privacy point of view handling your own log data is the safest solution.

Creating a simple centralized log server at Exoscale

There are plans at Exoscale to provide LaaS through partners. Until that is available here is how you can create your own centralized log server running in the Exoscale cloud alongside your other virtual machines.

New instance

As a first step login to the Exoscale portal and create a new instance. Even the smallest instance can handle hundreds of thousands of messages a second for simple log storage. If you’re limited by CPU, memory, storage, you can upgrade your machine any time.

CentOS is used in the examples below, but but installation and configuration steps are fairly similar on other Linux distributions.

Installing and configuring syslog-ng

There are many log servers available but syslog-ng was the first one designed for centralized logging right from the beginning. And it is also developed in Hungary, where I was born :-)

The syslog-ng application is available for most Linux distributions either in the base distribution or in 3rd party repositories. For CentOS it is available from the EPEL repository. In the following steps you download and install the repo file for the EPEL repository, install/enable/start syslog-ng and erase rsyslog:

wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -Uvh epel-release-latest-7.noarch.rpm 
yum install syslog-ng
yum erase rsyslog
systemctl enable syslog-ng
systemctl start syslog-ng

Next you need to create a configuration snippet for syslog-ng to listen to syslog-ng messages on port 514 and save them to a file under /var/log. “fromnet” is just an example file name, using macros you can create complete directory hierarchies to store files based on time and/or source host name. The syslog-ng documentation has many examples. Place the following snippet into a file under the /etc/syslog-ng/conf.d/ directory with the file name ending with .conf:

source s_net {
    tcp(ip( port(514));
destination d_fromnet { file("/var/log/fromnet"); };
log {source(s_net); destination(d_fromnet); };

And finally restart syslog-ng:

systemctl restart syslog-ng

Configuring a client

Installing and configuring syslog-ng on the client side is pretty similar. The only difference is the content of the configuration snippet:

destination d_remote {
  tcp("central-syslog.example.com" port("514"));
log {source(s_sys); destination(d_remote);};

Make sure, that you use the right IP address or host name in the configuration file. Once ready with configuration, restart syslog-ng:

systemctl restart syslog-ng

Testing centralized logging

Once both sides are ready you should test your centralized log server. On the client enter:

logger this is a test

On the server you should see a similar line appearing at the end of /var/log/fromnet:

Nov 21 14:29:28 czpclient root: this is a test

But you do not see it. Actually you do not even find the file, as due to lack of incoming log messages it is not yet even created.

By default Exoscale machines are strongly firewalled even from each other. Before you can send logs from the client to the server you need to drill a hole in it. Go to the firewalling menu in the Exoscale portal and choose the security group used both by your clients and server. In the upper right corner choose “New rule”. As “source type” choose “Security group” and select one in the drop down menu below it. Enter “514” both for start and end of port numbers. Click “Add” at the end.

Go back now to you syslog client and server and test it again. This time you should see logs from the client on the server.

What is next?

You have now a simple but working and compliant log server running next to your other virtual machines in the Exoscale cloud. How you can enhance it? The possibilities are almost endless. You can sort logs to different files, parse them, create alerts, archive them, store them to databases for easier searching and many more.