Centralized Logging with ELK, Beaver and Swarm 1


ELK stack running on docker

ELK stack running on docker

The ELK stack is a powerful centralized logging solution, that can store and analyse tons of logs from different services located on multiple hosts. This can save you hours of debugging and troubleshooting, when something gets screwed. It can be compared to powerful solutions like splunk – but it is completely open source.

 

In this post, I will deploy the ELK stack using Docker and will use the Beaver tool to send logs privately over TCP to Logstash, and to add more fun – I will use Docker Swarm to deploy a small cluster of Docker hosts.

 

What is the ELK Stack?

The ELK stack is consisting of 3 main components E(Elasticsearch) L(Logstash) K(Kibana), Elasticsearch is an open-source very efficient search server, besides its RESTful API it does offer many features including: sharded indexes, read operations only to the replicas, and Near Real time search.

 

Logstash is considered the key component for the ELK stack, it is a tool which manages and stores logs and events, Logstash offers multiple plugins which can enhance its capabilities including input, output, and processing plugins.

 

Kibana is a flexible reporting and visualization platform, that can analyze and monitors data gathered by logstash and indexed into elasticsearch, Kibana is easily integrated with Elasticsearch.

 

Starting a Swarm Cluster

 

Let’s get started! Swarm is a native clustering tool for Docker, which can create a cluster of Docker daemons on different hosts, that can act as a single Docker daemon, it supports the standard Docker API and therefore it can be used with any tool that can communicate with Docker (like docker-compose).

We assume that you are familiar with the principles of Docker and have a basic Docker installation on your system. If not – please consider following the basic Docker trainings first (https://training.docker.com/)

 

Each Docker daemon registers itself with service discovery backend to join the cluster, Docker swarm supports multiple service discovery backends like Etcd, Consul, Zookeeper, and its own hosted discovery backend the which is built in Docker Hub, in the case of using Docker Hub as the backend discovery, the cluster when created will have a unique token (cluster_id) which will be used later to join each Docker daemon with the swarm cluster.

 

The cluster is managed by a swarm manager, which must have access to each node on the cluster. The official documentation for Docker swarm recommends that you shouldn’t use Docker swarm in production yet.

 

I will use the hosted discovery service that built in Docker Hub, to create a new cluster you can either:

# docker run --rm swarm create
70d987c8f8c78065eb63c74eb9998ae3

 

Or you can use the backend API directly:

# curl -XPOST https://discovery-stage.hub.docker.com/v1/clusters
70d987c8f8c78065eb63c74eb9998ae3

 

The output token will be used later by the Swarm nodes to join the cluster, in order to join each node with the cluster you need first to start docker with different options than the defaults, edit the /etc/default/docker on the first node where we will deploy the ELK stack:

DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --label service=elk"

Note: For the sake of simplicity I started the Docker daemon to listen on a TCP socket – which is not secure by any way, in a real production environment we would use TLS to secure the connection between the client and the server.

 

One of the many Docker swarm features is the constraint filter which is basically running Docker daemon with specific tags (–label) to only schedule running or stopping containers on a subset of the swarm cluster. Thats why we started the previous Docker daemon with the –label which a service tag that indicates that this machine will host the ELK stack.

 

For the second node, start the daemon with app label to indicate that it will be used to host any other container:

DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --label service=apps"

 

Let’s now join each node with the cluster, to join any node with the cluster you will need the token generated earlier on creating the swarm cluster:

node-1:~# docker run -d swarm join --addr=104.236.117.230:2375 token://70d987c8f8c78065eb63c74eb9998ae3
node-2:~# docker run -d swarm join --addr=104.131.172.228:2375 token://70d987c8f8c78065eb63c74eb9998ae3

Now run the manager on any machine even your personal laptop, to manage the cluster:

$ docker run -d -p 2375:2375 swarm manage token://70d987c8f8c78065eb63c74eb9998ae3

 

Note that the swarm manager must have access on the Docker daemons to operate. To verify that your cluster is working properly, use the Docker cli to connect to the Docker daemon on the manager, for example:

$ docker -H tcp://localhost info
Containers: 2
Strategy: spread
Filters: affinity, health, constraint, port, dependency
Nodes: 2
node-1: 104.236.117.230:2375
└ Containers: 1
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 1.019 GiB
node-2: 104.131.172.228:2375
└ Containers: 1
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 514.5 MiB

ELK Image

The ELK Docker image is responsible for installing the 3 components of the ELK stack and start the Logstash to listen on port 5000 and the Kibana to listen on port 8000.

 

The Dockerfile of the ELK image:

FROM ubuntu:14.04
MAINTAINER Hussein Galal
ENV DEBIAN_FRONTEND=noninteractive

# Install Important tools
RUN apt-get -q update
RUN apt-get install -yqq wget software-properties-common

# Update the repos
RUN wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
RUN echo 'deb http://packages.elasticsearch.org/elasticsearch/1.4/debian stable main' | tee /etc/apt/sources.list.d/elasticsearch.list
RUN echo 'deb http://packages.elasticsearch.org/logstash/1.5/debian stable main' | tee /etc/apt/sources.list.d/logstash.list
RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt-get update

# Java 8
RUN apt-get install -yqq openjdk-8-jdk

# Elasticsearch 1.4.4
RUN apt-get -yqq install elasticsearch=1.4.4
RUN sed -i "s/#network.host.*/network.host: localhost/g" /etc/elasticsearch/elasticsearch.yml
VOLUME /var/lib/elasticsearch

# Logstash 1.5 Server
RUN apt-get install -yqq logstash
ADD logstash/input.conf /etc/logstash/conf.d/01-input.conf
ADD logstash/syslog.conf /etc/logstash/conf.d/10-syslog.conf
ADD logstash/output.conf /etc/logstash/conf.d/40-output.conf

# Kibana 4.0.1
RUN wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz
RUN tar xf kibana-4*.tar.gz
RUN mkdir -p /opt/kibana
RUN cp -R kibana-*/* /opt/kibana
ADD kibana/kibana.yml /opt/kibana/config/kibana.yml

# Start script
ADD run.sh /tmp/run.sh
RUN chmod +x /tmp/run.sh
WORKDIR /opt/kibana

EXPOSE 8000

ENTRYPOINT /tmp/run.sh

The Docker image adds the syslog filter, it also exports the volume /var/lib/elasticsearch to persist the data gathered from the log files. We will run this Docker image using our swarm manager:

$ docker -H tcp://<manager-ip>:2375 run --name elk -p 8000:8000 -p 10.132.6.35:5000:5000 -d  -e "constraint:service==elk" husseingalal/elk

The previous command maps the port 5000 privately, so that it can be accessible from other hosts in  the swarm cluster.

 

Configuring Logstash

The logstash tool has 3 configuration components, input, output, and filters, the input plugin enables a specific source of events to  be read by Logstash, there are a lot of available input plugins that can be used with logstash like TCP, UDP, Redis, lumberjack, etc.

 

configuring logstash

configuring logstash

 

In our case, I will use TCP input plugin to receive the logs in raw format from the beaver tool which we will use in the next section, the input configuration file of logstash will be like the following:

input {
  tcp {
    port => 5000
  }
}

 

The output plugin is used to send event data to specific destination, it can send the data to HTTP, TCP, UDP, email, Elasticsearch, or even stdout, In this case we will send data to Elasticsearch to be indexed and used later, the output configuration file will be something like this:

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

The middle stage of the event pipeline, is the filter stage, filters are intermediary processing devices used to perform action or event on the data being processed, one of the most popular filters out there, is the grok filter which is used to parse unstructured data into something structured and queryable.

 

Logstash comes with over than 120 grok patterns by default, which probably will match your needs, in our case we used syslog grok filter to structure the data gathered from system syslog:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

 

Beaver Image

Logstash can be fed with any kind of log files generated locally or remotely. For our case we’ll be generating our logs remotely and sending them to a central instance of Logstash.  here are a couple of good ways to send logs to Logstash remotely, for example you may use lumberjack protocol using the logstash-forwarder tool which requires you to setup an SSL certificate on both sides.

 

In our case, we’ll be using the tcp protocol with the help of a powerful tool called Beaver.  Beaver can do more than send logs via tcp, it also supports udp, Redis, HTTP, Mqtt, and Zeromq as a transport methods. and in this case Beaver tool will use TCP to send different logs of different types to the Logstash server over port 5000.

 

The following Docker image just installs beaver tool via pip and copy a configuration file of beaver that specify by default /var/log/syslog to send to Logstash:

FROM ubuntu
MAINTAINER Hussein Galal

RUN apt-get update -q && apt-get install -yqq rsyslog python python-pip
RUN pip install beaver
RUN mkdir -p /etc/beaver/conf.d
ADD beaver.conf /etc/beaver/beaver.conf

ENTRYPOINT beaver -c /etc/beaver/beaver.conf -C /etc/beaver/conf.d
CMD []

 

The previous image introduce an interesting concept called baseimage, baseimage is a Docker image that includes a general but basic service or process with its default configuration, and later this baseimage will be used as a base for another Docker image that will modify configuration or property of this baseimage base on your needs.

 

In our case we used Beaver image as a baseimage with the basic ini configuration (beaver.conf) which includes the ip and port of the Logstash server:

[beaver]
logstash_version: 1
tcp_host: localhost
tcp_port: 5000
format: raw

[/var/log/syslog]
type: syslog

 

Then you can use this image to build other services that can send their logs to Logstash server, for example to create an Apache image that sends its logs to Logstash server:

FROM husseingalal/beaver
MAINTAINER Hussein Galal
RUN apt-get update
RUN apt-get install -yqq apache2
ADD apache.conf /etc/beaver/conf.d/apache.conf
ADD start.sh /tmp/start_apache.sh
RUN chmod +x /tmp/start_apache.sh
ENTRYPOINT /tmp/start_apache.sh

And the start_apache script will edit the beaver configuration to send the logs to logstash server defined by the environment variable $LOGSTASH_SRV:

# Add the address of logstash server (env: LOGSTASH_SRV)
if env | grep -q "$LOGSTASH_SRV"
then
  sed -i "s/tcp_host.*/tcp_host: $LOGSTASH_SRV/g" /etc/beaver/beaver.conf
fi
######## START YOUR SERVICES HERE #########
service apache2 restart
################# END #####################
# starting beaver
beaver -c /etc/beaver/beaver.conf -C /etc/beaver/conf.d -t tcp

 

Apache Service

To start Apache service on the other node, start the container with the apps label constraint, also providing the IP of the logstash server started on the other node:

# docker -H tcp://:2375 run --name apache -p 80:80 -d -e "constraint:service==apps" -e "LOGSTASH_SRV=10.132.6.35" husseingalal/apache-beaver

 

To verify that the Apache is connected to the Logstash server:

$ docker -H tcp://localhost:2375 logs apache
 * Restarting web server apache2
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.4. Set the 'ServerName' directive globally to suppress this message
...done.
[2015-05-31 00:40:41,172] INFO Starting worker...
[2015-05-31 00:40:41,173] INFO Working...
[2015-05-31 00:40:41,177] INFO [28gd8] - watching logfile /var/log/apache2/access.log
[2015-05-31 00:40:41,178] INFO Starting queue consumer
[2015-05-31 00:40:41,193] INFO Connected

 

This means that Apache’s beaver is connected to the Logstash server, and as you can see that beaver is watching /var/log/apache2/access.log log file.

Finally you can access the Kibana server …

access your Kibana server

access your Kibana server

… and have all your logs in a central dashboard

search your logs in Kibana

search your logs in Kibana

 

Conclusion

 

Gathering and analyzing logs is a main task for any ops team, using Docker and beaver with a centralized logging stack like ELK simplifies the process of gathering, shipping, and analyzing the logs. We saw how to use Docker swarm to extend the solution to multiple clustered machines.


Leave a comment

Your email address will not be published. Required fields are marked *

One thought on “Centralized Logging with ELK, Beaver and Swarm