Update on 20th of July 2015! As the BusyBox image’s package manager opkg doesn’t work anymore I have switched to using Alpine instead. The example images and commands have been updated. At the same time the Dockerfiles have been modularized to help update the tool versions.
As versions of the tools have changed considerably from the time of the original blog post, I decided to republish this instead of updating the old post. Now the tool versions are:
- Elasticseach 1.7.0
- LogStash 1.5.2
- Kibana 4.1.0
Docker-machine (0.3.0) syntax is used in the examples.
As running the microboxes with the volume containers is quite cumbersome, I have concentrated on the miniboxes and dropped the microboxes from the examples. If you are interested in using the microbox versions, please drop a line in the comments and I’ll update them as well.
When I was using fat containers, I used rsyslog to pass the log messages to a centralized container which had Elasticsearch, Logstash and Kibana installed. This combination is also known as the ELK-stack. After moving towards minicontainers I wanted to separate the logging logic from the application containers. Orchestration is still an issue with using multiple simple containers, but at least the application containers shouldn’t need to know anything about the logging architecture as long as they can log to stdout and stderr so that Docker can pick the logs up.
All the containers in the examples have the corresponding DockerHub builds created for them, so running the scripts is enough without the need to build the containers locally.
I decided to install Logstash and Elasticsearch to the same container for simplicity and separate Kibana as an own container as it’s not needed for the logging to work. I use Logspout to push the logs from the Docker socket to the Logbox.
The Dockerfiles for both Logstash + Elasticsearch (logbox) container and the Kibana (kibanabox) container can be found from github. They are pretty straightforward, although they are based on BusyBox to minimize the size. The Logbox image is about 350 MB in size.
These examples expect that you have created the environment using Docker Machine 0.3.0 and it is called “dev”. If not, just substitute the IP address of your server to the commands. These commands start the Logbox, Kibanabox and Logspout on the active docker-machine controlled node.
Scripts called startLogging.sh and stopLogging.sh can be found here.
1 2 3 4 #!/bin/bash docker run -d --name logbox -h logbox -p 5000:5000/udp -p 9200:9200 sirile/minilogbox docker run -d -p 5601:5601 -h kibanabox --name kibanabox sirile/kibanabox http://`docker-machine ip $DOCKER_MACHINE_NAME`:9200 docker run -d --name logspout -h logspout -p 8100:8000 -v /var/run/docker.sock:/tmp/docker.sock progrium/logspout syslog://`docker-machine ip $DOCKER_MACHINE_NAME`:5000
After starting the services the Kibana UI can be found from the port 5601. For example http://192.168.99.100:5601.
When you access Kibana for the first time, it asks you to confirm the time field name. Choose @timestamp. Then you see the listing of the index and can choose Discover from the top bar which takes you to the Kibana dashboard showing the log messages.
Adding more nodes
On the subsequent nodes only the LogSpout container needs to be run and it should be pointed to the server which has Logbox running. If service discovery is handled by Consul (like described here), then DNS can be used to locate the logging server instead of relying to docker-machine.
Troubleshooting and Issues
At first after having everything running I still couldn’t get Logspout messages to go through to the Logstash container. I couldn’t find any error messages and if I used telnet to directly connect the Logstash container in port 5000, my messages showed up nicely.
After battling for a couple of hours I realized that UDP isn’t automatically supported when exposing ports to the host. Instead it needs to be manually specified with adding “/udp” after the -p parameter.
After adding the parameter the logging system began to work as expected.
Syslog message parsing
Another issue I faced were a lot of parsing failures for the log messages. It was caused by Logspout using ISO 8601 as the date format whereas the older rsyslog based parsing expected syslog format. After modifying the parser the _grokparsefailures stopped and the messages were properly handled.
Timestamps and timezones
For the timestamps to be the same across the host and containers, it may be
advisable to pass
-v /etc/localtime:/etc/localtime:ro -v
/etc/timezone:/etc/timezone:ro as parameters for all the container run
commands. Note! It seems that this is not necessary with the current Alpine
Separating logging using Logspout to direct the logs to Logstash and Elasticsearch works pretty well. One issue is that a lot of the programs log in a non-syslog friendly way by for example indenting the log messages. Parsing a Java stacktrace from multiple messages can be painful and requires further investigation. With rsyslog the messages can be combined before sending, but logspout forwards the messages as they appear, so that is not an option.
As Docker has started to support direct syslog logging the need for Logspout may not be there any more. I need to try it out and will update the blog based on the findings.