Introduction to Managing Docker Container Logs with the ELK Stack

Docker containers generate a high volume of log data—stdout, stderr, application logs, and system events. Without a centralized log management solution, diagnosing issues, monitoring performance, and ensuring security becomes chaotic. The ELK Stack—Elasticsearch, Logstash, and Kibana—is a battle-tested, open-source platform for aggregating, processing, and visualizing logs at scale. By combining Docker with ELK, you gain real-time visibility into container health, resource usage, and error patterns.

This guide walks you through a production-ready setup of ELK for Docker container logs. You’ll learn how to deploy each component, configure log ingestion, create meaningful visualizations, and follow best practices to keep your system reliable and secure.

Prerequisites

Before we begin, ensure you have the following:

  • Docker Engine (20.10+) and Docker Compose installed on a Linux server (Ubuntu 20.04/22.04 recommended).
  • At least 8 GB of RAM available (Elasticsearch and Kibana are memory-intensive).
  • Basic familiarity with the command line and YAML configuration.
  • Open ports: 9200 (Elasticsearch), 5601 (Kibana), and optionally 5044 (Logstash beats input).

Setting Up Elasticsearch

Elasticsearch is the heart of the ELK Stack—it stores indexed log data and provides near-real-time search capabilities. Because Docker container logs can grow quickly, Elasticsearch must be configured for stability and performance.

Pulling and Running Elasticsearch

Start by pulling the official Elasticsearch Docker image. Use version 8.x (or 7.x if compatibility is needed). The following command runs a single-node instance for development; for production, you would deploy a multi-node cluster.

docker run -d --name elasticsearch \
  -p 9200:9200 -p 9300:9300 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -e "ES_JAVA_OPTS=-Xms1g -Xmx1g" \
  docker.elastic.co/elasticsearch/elasticsearch:8.10.0

Important configuration notes:

  • Memory limits: Elasticsearch uses the -Xms and -Xmx heap size settings. For a production Docker environment, allocate at least 50% of available RAM to the heap, but never exceed 30 GB. Use the ES_JAVA_OPTS environment variable to set these values.
  • Data persistence: Mount a host directory to /usr/share/elasticsearch/data to prevent data loss when the container restarts.
  • Network security: In a real deployment, enable X-Pack security (xpack.security.enabled=true) and create secure passwords. For this walkthrough, we disable it for simplicity.
  • Resource limits: Use Docker’s --memory and --cpus flags to cap Elasticsearch’s resource consumption and avoid starving other containers.

Verify Elasticsearch is running by visiting http://<your-server-ip>:9200 in a browser or using curl. You should see a JSON response with cluster name and version details.

Configuring Logstash for Log Collection

Logstash ingests logs from Docker containers, transforms them with filters, and sends them to Elasticsearch. The challenge is accessing Docker logs—standard Docker logging drivers write logs to files or ship them via TCP/UDP. Below are three common input strategies.

Input Strategies for Docker Logs

  1. File Input (JSON logs): Docker writes container logs as JSON files to /var/lib/docker/containers/<container-id>/<container-id>-json.log. Mount the Docker log directory into Logstash and use the file input plugin to tail these files.
  2. TCP/UDP Input: Configure Docker’s syslog or gelf logging driver to forward logs directly to Logstash. This avoids file system access but requires driver configuration per container or globally.
  3. Beats Input (Filebeat): Deploy Filebeat as a sidecar container that scrapes Docker logs and forwards them to Logstash via the Beats protocol. This is the most scalable approach for production.

For this guide, we’ll use the file input method because it requires no extra dependencies and works out of the box with Docker’s default json-file driver.

Creating a Logstash Pipeline

Create a directory structure for Logstash configuration:

mkdir -p logstash/pipeline

Inside logstash/pipeline, create logstash.conf with the following content:

input {
  file {
    path => "/var/log/docker/containers/*/*-json.log"
    start_position => "beginning"
    codec => "json"
  }
}

filter {
  if [docker][container][image] =~ /^logstash/ {
    drop { }
  }
  mutate {
    add_field => { "[@metadata][index]" => "docker-logs-%{+YYYY.MM.dd}" }
  }
  date {
    match => [ "time", "ISO8601" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "%{[@metadata][index]}"
    user => "elastic"
    password => "changeme"
  }
}

Key configuration details:

  • Path: The file input monitors all JSON log files under the Docker containers directory. Adjust the path if you mount logs differently.
  • Codec: Docker’s JSON log format is parsed natively. Each line is a JSON object with fields like log, stream, time.
  • Filter: The date filter uses the time field (ISO8601 format) to set the @timestamp field. The mutate step adds a dynamic index name based on the date. We also drop Logstash’s own logs to avoid loops.

Running Logstash with Docker

Run the Logstash container with the configuration mounted and access to the Docker log files:

docker run -d --name logstash \
  --link elasticsearch \
  -v /var/lib/docker/containers:/var/log/docker/containers:ro \
  -v $(pwd)/logstash/pipeline:/usr/share/logstash/pipeline:ro \
  -e "LS_JAVA_OPTS=-Xms512m -Xmx512m" \
  docker.elastic.co/logstash/logstash:8.10.0

Important: Mounting /var/lib/docker/containers gives Logstash read access to all container logs, including those from other containers on the host. Ensure this is acceptable in your security policy. Use a network bridge and --link to allow Logstash to resolve the Elasticsearch container hostname.

Check Logstash logs (docker logs logstash) to verify it starts without errors and begins tailing logs. Elasticsearch should start receiving documents shortly after you run the container.

Visualizing Logs with Kibana

Kibana is the visualization and management layer. It connects to Elasticsearch and lets you build dashboards, create alerts, and perform ad-hoc searches.

Deploying Kibana

Pull and run the Kibana container, linking it to Elasticsearch:

docker run -d --name kibana \
  --link elasticsearch \
  -p 5601:5601 \
  -e "ELASTICSEARCH_HOSTS=http://elasticsearch:9200" \
  docker.elastic.co/kibana/kibana:8.10.0

Once the container starts, access Kibana at http://<your-server-ip>:5601. You’ll be prompted to define an index pattern.

Configuring Index Patterns

  1. In Kibana, go to Management > Stack Management > Index Patterns.
  2. Click Create index pattern and enter docker-logs-* (matching the index name we set in Logstash).
  3. Select @timestamp as the time filter field name.
  4. Click Create index pattern.

Kibana will now display all fields extracted from your Docker log documents. You can explore logs in the Discover tab using Lucene or Kibana Query Language (KQL).

Building Dashboards

Dashboards provide at-a-glance health of your container environment. Common visualization ideas include:

  • Total logs over time: A line chart showing log volume per minute, helpful for detecting spikes.
  • Logs by container name: A pie chart or data table grouped by docker.container.name.
  • Error rates: Filter logs where log field contains “error” or “ERROR” and display as a gauge or metric.
  • Top error messages: A tag cloud of recurring error strings.
  • Geo IP (if available): If your logs contain IP addresses, use a geoip filter in Logstash to add location coordinates and visualize on a map.

Create visualizations in the Visualize tab, then add them to a new dashboard. Kibana also supports Canvas for custom pixel-perfect layouts.

Best Practices for Managing Docker Logs with ELK

Running a log pipeline in production requires careful planning to avoid performance pitfalls, data loss, and security breaches. Follow these best practices to maintain a robust setup.

Log Rotation and Retention

Docker containers can produce tens of gigabytes of logs per day. Configure log rotation at the Docker level and index lifecycle management in Elasticsearch.

  • Docker log rotation: Use the --log-opt max-size=10m --log-opt max-file=3 flags when running containers to keep log files small. You can set these defaults globally in /etc/docker/daemon.json.
  • Elasticsearch ILM (Index Lifecycle Management): Create a policy that rolls over indices daily, transitions to warm phase after 7 days, and deletes after 30 days. This prevents Elasticsearch from accumulating stale data.

Securing the ELK Stack

Since logs often contain sensitive information, protection is non-negotiable.

  • Enable X-Pack Security: Set xpack.security.enabled=true on all ELK components. Create users for Elasticsearch, Logstash, and Kibana with least-privilege roles.
  • Use TLS/SSL: Encrypt communication between Logstash, Elasticsearch, and Kibana. Elastic provides certificates or you can use a reverse proxy (nginx) with Let’s Encrypt for Kibana.
  • Network segmentation: Place ELK containers on a dedicated Docker network and restrict inbound traffic to trusted IPs.
  • Audit Logs: Enable Elasticsearch audit logging to track who accessed what data.

Monitoring the ELK Infrastructure

Your logging stack itself needs logging. Use Heartbeat or Metricbeat to monitor the health of Elasticsearch and Logstash. Set up Kibana alerts (alerting requires a Platinum license or the free basic tier for core features) to notify you if log ingestion drops or error rates spike.

Scaling and Resource Management

As log volume grows, you may need to scale:

  • Elasticsearch: Add more nodes to the cluster and shard indices appropriately. Use index.number_of_shards based on expected data size (e.g., 1 shard per 20 GB per day).
  • Logstash: Increase the number of pipeline workers (pipeline.workers) and use multiple input pipelines for high throughput.
  • Kibana: Deploy behind a load balancer for high availability, though Kibana is stateless.

Using Docker Logging Drivers

Docker supports several logging drivers (json-file, syslog, gelf, fluentd, etc.). For ELK, the native gelf driver can send logs directly to Logstash via UDP, eliminating the need for file mounts. Example global daemon.json configuration:

{
  "log-driver": "gelf",
  "log-opts": {
    "gelf-address": "udp://logstash:12201"
  }
}

However, be aware that GELF bundles multiple messages per packet, which may cause data loss under high load. The file input method is more reliable for high-volume environments.

Field Mapping and Data Quality

Logstash’s default JSON parsing produces nested fields (e.g., docker.container.name). Elasticsearch applies dynamic mapping, which is fine for exploration but can lead to mapping explosions. Define explicit index templates with appropriate data types and limit the number of fields to 1000 or less using the index.mapping.total_fields.limit setting.

Conclusion

Implementing the ELK Stack for Docker container logs transforms raw, scattered log lines into structured, searchable, and visualizable data. Elasticsearch provides fast storage and retrieval, Logstash handles data transformation, and Kibana offers powerful dashboards for monitoring and troubleshooting. By following the setup steps and best practices outlined here—such as log rotation, security hardening, and performance tuning—you’ll build a logging infrastructure that scales with your containerized applications.

To dive deeper, explore the official Elastic documentation for Elasticsearch Docker, Logstash Docker, and Kibana Docker. For advanced Docker logging strategies, refer to the Docker logging driver documentation.