ELK

https://www.elastic.co/what-is/elk-stack

So, what is the ELK Stack? “ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch.

Kibana lets users visualize data with charts and graphs in Elasticsearch.

There in the wild are existing more complex deployments like HELK https://github.com/Cyb3rWard0g/HELK or S1EM https://github.com/V1D1AN/S1EM.

Security information and event management (SIEM) provide real-time analysis of security alerts generated by applications and network hardware. Events on a system could include and are not limited to credential changes, failed access attempts, role base or attribute changes to accounts, token-based use, access attempts, and failures, etc. While logging every system action to the system is possible, it is often not advised based on the volume of logs and actionable security-relevant data. Organizations can use AU-2 a through e, as the basis to build from while adhering to other controls that may require or call out specific security auditing requirements in more granular detail. NIST SP 800-53 SI-4 System Monitoring is the security control that specifies the monitoring of the system. This monitoring is focused on monitoring systems that monitor the system. This can include hardware and software in unison to detect events and anomalies, malware, connections, and any other pertinent mechanism that is used to detect attacks or indicators of potential attacks.

VPSFree

ELK is being deployed to virtual machine hosted at with operating system Rocky Linux 8.4.

Installation

# Installation of docker
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y docker-ce
sudo systemctl start docker
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

# Installation of docker compose
dnf install -y curl
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version

# Installation of gitlab runner, which is not neccessary for your build
# I'm using gitlab cicd to deploy elk in docker
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
sudo dnf install -y gitlab-runner
sudo systemctl enable --now gitlab-runner

sudo gitlab-runner register

export HTTP_PROXY=http://yourproxyurl:3128
export HTTPS_PROXY=http://yourproxyurl:3128

sudo -E gitlab-runner register

gitlab-runner register -h

sudo gitlab-runner register \
--non-interactive \
--name "docker-shared-runner-bilek" \
--url "https://gitlab.url.com/" \
--registration-token "token" \
--executor "docker" \
--docker-image docker:20.10.17-dind \
--maintenance-note "Sorry that your general pipelie isn't running." \
--tag-list "docker,shared" \
--run-untagged="true" \
--locked="false" \
--access-level="not_protected"


# Install ELK in docker
git clone https://github.com/sherifabdlnaby/elastdocker.git
make setup

# For Linux's docker hosts only. By default virtual memory is not enough
# so run the next command as root sysctl -w vm.max_map_count=262144

make elk #<OR>
docker-compose up -d

#Visit Kibana at https://localhost:5601 or https://<your_public_ip>:5601
#Default Username: elastic, Password: changeme

# Setup ngix reverse proxy
cat > /etc/nginx/sites-available/siem.lichnak.cz.conf <<"EOF"
upstream siem {
  server 127.0.0.1:5601 fail_timeout=0;
}

server {
  server_name siem.lichnak.cz;
  access_log /var/log/nginx/vhost/siem.lichnak.cz-access.log;
  error_log /var/log/nginx/vhost/siem.lichnak.cz-error.log;

  location / {
    proxy_read_timeout 300;
    proxy_connect_timeout 300;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header Host $host;
    proxy_set_header X-NginX-Proxy true;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass https://siem;
    proxy_redirect off;
  }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/lichnak.cz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/lichnak.cz/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = siem.lichnak.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  server_name siem.lichnak.cz;
    listen 80;
    return 404; # managed by Certbot

}
EOF

ln -s /etc/nginx/sites-available/siem.lichnak.cz.conf /etc/nginx/sites-enabled/siem.lichnak.cz.conf
nginx -t
systemctl restart nginx

Planning, configuration and hardening

1) Plan & size

  • Define retention, expected daily ingest volume (GB/day), and query/aggregation SLA.
  • Choose topology: single-node (dev), small cluster (3 master-eligible + data nodes), or separated roles for large workloads.

1) Proof-of-concept (single-node or docker-compose)

  • Start an all-in-one node locally or with the provided docker-compose to validate parsing pipelines and dashboards.
  • Ingest example logs using Filebeat and create index patterns and a small dashboard in Kibana.

1) Secure the stack

  • Enable TLS, set up built-in users and roles, require MFA for Kibana admins where possible.
  • Restrict network access with firewall rules and use internal networks for node-to-node traffic.

1) Reliability & storage management

  • Configure ILM policies: hot → warm → cold → delete phases to match retention requirements.
  • Implement snapshot schedules to remote storage and verify restores periodically.

1) Scale to production

  • Add dedicated master-eligible nodes (3) and scale data and ingest nodes horizontally.
  • Use shard sizing best practices: avoid too many small shards; target shard size ~30–50GB depending on use case.

1) Operationalize

  • Enable metrics collection (Metricbeat) and log monitoring for Elasticsearch, Logstash, and Kibana.
  • Create alerting rules for cluster health, disk watermarks, JVM pressure, and slow queries.
  • Automate configuration and deployment using Ansible, Terraform (for infra), and CI pipelines for pipeline/config promotion.

1) Dashboards, detection and retention

  • Build dashboards iteratively: start with critical KPIs and expand.
  • Implement detection rules (Elastic SIEM, custom alerts, or Wazuh/other detections) and tune to reduce false positives.

1) Continual improvement

  • Conduct periodic capacity planning, index audits, and ILM policy reviews.
  • Run disaster recovery drills for snapshot restore and test restoring Kibana saved objects.

Configuration

This section highlights important configuration and hardening points for a reliable ELK deployment.

Elasticsearch

  • Set cluster.name and node.name consistently across nodes.
  • Use node roles (master, data, ingest, coordinating, ml) and avoid colocating heavyweight roles on the same host.
  • Heap: set -Xms and -Xmx equal and to no more than ~50% of available RAM (leave room for OS/file cache).
  • Files and limits: raise vm.max_map_count to at least 262144, increase ulimit -n (open files) and ulimit -u (max user processes) for the elasticsearch user.
  • Paths: configure path.data and path.logs on fast, separate disks when possible.
  • Network: bind network.host to internal interfaces; use transport TLS and set xpack.security.enabled: true in production.
  • Authentication & TLS: enable built-in security (X-Pack) or place an authenticated reverse proxy in front of Kibana/Elasticsearch and secure transport layer between nodes with certificates.
  • Index lifecycle management (ILM): configure ILM policies to automate rollover, shrink, and deletion to control storage costs.

Logstash

  • Use pipelines.yml for managing multiple pipelines; enable persistent queues if reliability during peaks is required.
  • Keep filters efficient (avoid heavy grok where possible) and use grok with named patterns and on_failure handling.
  • Tune pipeline.workers and pipeline.batch.size according to CPU and memory.
  • Protect secrets (API keys, DB credentials) with environment variables or secret backends (Vault) rather than committing them into repositories.

Kibana

  • Set server.host and elasticsearch.hosts in kibana.yml to connect only to trusted endpoints.
  • Enable xpack.security and configure TLS between Kibana and Elasticsearch.
  • Set server.ssl.enabled and provide certificates for public-facing instances, and enable kibana.encryptionKey for session encryption.

Beats (Filebeat/Metricbeat/Auditbeat)

  • Use central configuration management and enroll beats with tokens/keys.
  • Configure output to Logstash or Elasticsearch depending on filter needs, and enable SSL/TLS and authentication.

Docker / Docker Compose specifics

  • Persist Elasticsearch data directories as named volumes or host volumes mapped to stable storage.
  • Expose only necessary ports; use internal overlay networks for inter-container communication.
  • Set JVM options via environment variables or mounted config files; monitor memory usage in containers.

Monitoring & Observability

  • Enable monitoring (Metricbeat or Elastic’s monitoring features) to collect node metrics and alert on cluster health, disk utilization, JVM pressure, and shard allocation issues.
  • Use Kibana and watcher/alerting (or external alert manager) for health and SLA alerts.

Security hardening checklist (minimum)

  • Enable TLS for transport and HTTP layers.
  • Require authentication for all APIs; disable anonymous access.
  • Restrict Elasticsearch API access to internal networks or service accounts only.
  • Regularly rotate service account credentials and TLS certificates.

Deployment Scenario: Docker Compose with Nginx Reverse Proxy

A complete, production-ready ELK deployment using Docker Compose and Nginx as reverse proxy is available in the project repository at deployments/elk/.

Architecture Overview

┌─────────────────────────────────────────────────────┐
│                     nginx (reverse proxy)            │
│                  (HTTP → HTTPS redirect)             │
│            (Port 80, 443 exposed to host)            │
└──────────────────────┬──────────────────────────────┘
                       │
         ┌─────────────┴──────────────┐
         │                            │
    ┌────────────┐             ┌─────────────┐
    │  Kibana    │             │ Elasticsearch
    │ (Port      │             │ (Port 9200, │
    │  5601)     │             │  internal)  │
    └────────────┘             └─────────────┘
         │                            △
         └────────────┬───────────────┘
                      │
               ┌──────────────┐
               │  Logstash    │
               │ (Processes   │
               │   logs)      │
               └──────────────┘

Quick Start

cd deployments/elk

# Generate self-signed certificates (development)
mkdir -p certs
openssl req -x509 -newkey rsa:4096 -keyout certs/server.key -out certs/server.crt \
  -days 365 -nodes -subj "/CN=localhost"

# Update .env with a strong password (optional)
# ELASTIC_PASSWORD=your_secure_password

# Start all services
docker compose up -d

# Wait for Elasticsearch to be healthy
docker compose logs elasticsearch | grep "green"

# Access Kibana
# https://localhost (or https://<your-hostname>)
# Username: elastic
# Password: (from .env ELASTIC_PASSWORD)

# View onion service hostname (if running behind Tor)
docker compose exec tor cat /var/lib/tor/hidden_service/hostname

Components

  • Elasticsearch (8.10.0): Search and analytics engine with X-Pack security enabled.
  • Logstash (8.10.0): Data processing pipeline that reads JSON logs from logs/ directory.
  • Kibana (8.10.0): Visualization and analytics UI.
  • Nginx (Alpine): Reverse proxy with TLS termination, HTTP→HTTPS redirect, and security headers.

Key Features

  • TLS/SSL: HTTPS by default with self-signed or CA-signed certificates.
  • Authentication: Built-in Elasticsearch security with username/password.
  • Persistent Storage: Named volume for Elasticsearch data.
  • Sample Data: Pre-populated logs/app.log with realistic JSON log entries for testing.
  • Security Headers: Strict-Transport-Security, X-Frame-Options, X-Content-Type-Options, etc.
  • Networking: Isolated overlay network; only Nginx exposes ports to the host.

Configuration Files

  • docker-compose.yml — Service definitions, volumes, networks.
  • nginx/default.conf — Reverse proxy config with health checks and upstream servers.
  • logstash/pipeline/logstash.conf — Log parsing, filtering, and enrichment.
  • .env — Environment variables (passwords, versions, Java heap options).
  • logs/app.log — Sample JSON logs for ingestion.

Ingesting Logs

File-based (Default)

Place .log files in logs/ directory:

echo '{"timestamp":"2026-02-16T10:00:00Z","level":"ERROR","message":"Test error"}' >> logs/app.log

TCP Input

Uncomment TCP input in logstash/pipeline/logstash.conf and send data:

echo 'test message' | nc localhost 5000

Syslog Input

Configure remote systems to forward syslog to the stack; uncomment syslog input in the Logstash config.

Creating Kibana Dashboards

  1. Access Kibana at https://localhost.
  2. Go to Management → Dev Tools → Console.
  3. Verify indices: GET _cat/indices.
  4. Create an index pattern: Management → Index Patterns → Create index pattern.
  5. Explore logs in the Discover tab.
  6. Build visualizations and dashboards.

Backup and Snapshots

# Create a snapshot repository
docker compose exec elasticsearch curl -X PUT -u elastic:changeme -k \
  -H "Content-Type: application/json" \
  https://localhost:9200/_snapshot/my_backup \
  -d '{"type":"fs","settings":{"location":"/usr/share/elasticsearch/snapshots"}}'

# Trigger a snapshot
docker compose exec elasticsearch curl -X PUT -u elastic:changeme -k \
  https://localhost:9200/_snapshot/my_backup/snapshot_1

# List snapshots
docker compose exec elasticsearch curl -u elastic:changeme -k \
  https://localhost:9200/_snapshot/my_backup/_all?pretty

Hardening & Production Readiness

  • Replace certificates: Use CA-signed certificates; store them securely.
  • Strong passwords: Change ELASTIC_PASSWORD in .env to a strong, unique value.
  • Networking: Restrict Nginx ports via firewall; isolate ELK network from untrusted clients.
  • Memory tuning: Adjust ES_JAVA_OPTS and LS_JAVA_OPTS based on available RAM.
  • Persistence: Back up the elasticsearch_data volume regularly.
  • Scaling: Add more Logstash and Kibana replicas, and scale Elasticsearch to a multi-node cluster.
  • Monitoring: Export metrics to Prometheus or use Elastic’s monitoring features.

Troubleshooting

Elasticsearch won’t start:

docker compose logs elasticsearch
# Check Java heap size; adjust ES_JAVA_OPTS in .env

Logstash not forwarding logs:

docker compose logs logstash
# Verify logs/ directory has files and Elasticsearch is healthy

Nginx 502 Bad Gateway:

docker compose logs nginx
docker compose ps  # Verify all services are running

For detailed setup, configuration, and troubleshooting, see deployments/elk/README.md.

Backup and restore

Backups for Elasticsearch are best performed using snapshot/restore. Store snapshots on a durable repository (S3, GCS, Azure, or a stable NFS filesystem).

1) Create a snapshot repository (example: filesystem)

# register a filesystem repository
curl -X PUT "http://localhost:9200/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/mnt/elastic_backups",
    "compress": true
  }
}
'

1) Create a snapshot (can be automated via cron/curator/ILM policies or external scheduler)

curl -X PUT "http://localhost:9200/_snapshot/my_backup/snapshot_$(date +%F)?wait_for_completion=true"

1) List snapshots

curl -s "http://localhost:9200/_snapshot/my_backup/_all?pretty"

1) Restore a snapshot

# stop ingestion and lock index aliases if needed
curl -X POST "http://localhost:9200/_snapshot/my_backup/snapshot_2026-02-16/_restore" -H 'Content-Type: application/json' -d'
{
  "indices": "logstash-*",
  "ignore_unavailable": true,
  "include_global_state": false
}
'

Additional items to backup and restore

  • Kibana saved objects (dashboards, visualizations): export via Management → Saved Objects or use the Saved Objects API to export/import JSON.
  • Logstash pipelines and configuration: persist /etc/logstash or container volume and include in file-level backups.
  • Beats configuration: keep central configs and enrollment tokens backed up.

Restore guidance

  • Always stop ingest (Filebeat/Logstash) before restoring indices to avoid write conflicts.
  • Restore snapshots into a maintenance cluster or restore into index names with new prefixes, then reindex into production if necessary.
  • After restore, verify index mappings, ILM policies, and aliases; re-enable ingestion once verified.

From Zero to Hero

This concise path outlines how to go from an empty environment to a basic but production-aware ELK deployment.

me

My name is Adam Lichonvsky and I'm proud father and researcher.

         

Recent Posts