Awesome Open Source
Awesome Open Source

ansible-elk

Ansible Playbook for setting up the ELK/EFK Stack and Filebeat client on remote hosts

ELK

CI

What does it do?

  • Automated deployment of a full 6.x series ELK or EFK stack (Elasticsearch, Logstash/Fluentd, Kibana)
    • 5.6 and 2.4 ELK versions are maintained as branches and master branch will be 6.x currently.
    • Uses Nginx as a reverse proxy for Kibana, or optionally Apache via apache_reverse_proxy: true
    • Generates SSL certificates for Filebeat or Logstash-forwarder
    • Adds either iptables or firewalld rules if firewall is active
    • Tunes Elasticsearch heapsize to half your memory, to a max of 32G
    • Deploys ELK clients using SSL and Filebeat for Logstash (Default)
    • Deploys rsyslog if Fluentd is chosen over Logstash, picks up the same set of OpenStack-related logs in /var/log/*
    • All service ports can be modified in install/group_vars/all.yml
    • Optionally install curator
    • Optionally install Elastic X-Pack Suite
    • This is also available on Ansible Galaxy

Requirements

  • RHEL7 or CentOS7 server/client with no modifications
  • RHEL7/CentOS7 or Fedora for ELK clients using Filebeat
  • ELK/EFK server with at least 8G of memory (you can try with less but 5.x series is quite demanding - try 2.4 series if you have scarce resources).
  • You may want to modify vm.swappiness as ELK/EFK is demanding and swapping kills the responsiveness.
    • I am leaving this up to your judgement.
echo "vm.swappiness=10" >> /etc/sysctl.conf
sysctl -p

Notes

  • Current ELK version is 6.x but you can checkout the 5.6 or 2.4 branch if you want that series
  • I will update this playbook for major ELK versions going forward as time allows.
  • Sets the nginx htpasswd to admin/admin initially
  • nginx ports default to 80/8080 for Kibana and SSL cert retrieval (configurable)
  • Uses OpenJDK for Java
  • It's fairly quick, takes around 3minutes on a test VM
  • Fluentd can be substituted for the default Logstash
    • Set logging_backend: fluentd in group_vars/all.yml
  • Install curator by setting install_curator_tool: true in install/group_vars/all.yml
  • Install Elastic X-Pack Suite for Elasticsearch, LogStash or Kibana via:
    • install_elasticsearch_xpack: true
    • install_kibana_xpack: true
    • install_logstash_xpack: true
    • Note: Deploying X-Pack will wrap your ES with additional authentication and security, Kibana for example will have it's own credentials now - the default is username: elastic and password: changeme

ELK/EFK Server Instructions

  • Clone repo and setup your hosts file
git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
sed -i 's/host-01/elkserver/' hosts
sed -i 's/host-02/elkclient/' hosts
  • If you're using a non-root user for Ansible, e.g. AWS EC2 likes to use ec2-user then set the follow below, default is root.
ansible_system_user: ec2-user
  • Run the playbook
ansible-playbook -i hosts install/elk.yml

Create your Kibana Index Pattern

  • Next you'll login to your Kibana instance and create a Kibana index pattern.

ELK

  • Note: Sample data can be useful, you can try it later however.

ELK

ELK

ELK

ELK

  • At this point you can setup your client(s) to start sending data via Filebeat/SSL

ELK Client Instructions

  • Run the client playbook against the generated elk_server variable
ansible-playbook -i hosts install/elk-client.yml --extra-vars 'elk_server=X.X.X.X'
  • Once this completes return to your ELK and you'll see log results come in from ELK/EFK clients via filebeat

ELK

5.6 ELK/EFK (Deprecated)

  • The 5.6 series of ELK/EFK is also available, to use this just use the 5.6 branch
git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
git checkout 5.6

2.4 ELK/EFK (Deprecated)

  • The 2.4 series of ELK/EFK is also available, to use this just use the 2.4 branch
git clone https://github.com/sadsfae/ansible-elk
cd ansible-elk
git checkout 2.4
  • You can view a deployment video here:

Ansible Elk

File Hierarchy

.
ā”œā”€ā”€ hosts
ā”œā”€ā”€ install
ā”‚Ā Ā  ā”œā”€ā”€ elk_client.yml
ā”‚Ā Ā  ā”œā”€ā”€ elk.yml
ā”‚Ā Ā  ā”œā”€ā”€ group_vars
ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ all.yml
ā”‚Ā Ā  ā””ā”€ā”€ roles
ā”‚Ā Ā      ā”œā”€ā”€ apache
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā”œā”€ā”€ 8080vhost.conf.j2
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ kibana.conf.j2
ā”‚Ā Ā      ā”œā”€ā”€ curator
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ files
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ curator.repo
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā”œā”€ā”€ curator-action.yml.j2
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ curator-config.yml.j2
ā”‚Ā Ā      ā”œā”€ā”€ elasticsearch
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ files
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ elasticsearch.in.sh
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ elasticsearch.repo
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ elasticsearch.yml.j2
ā”‚Ā Ā      ā”œā”€ā”€ elk_client
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ files
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ elk.repo
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”œā”€ā”€ filebeat
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ meta
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā”œā”€ā”€ filebeat.yml.j2
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ rsyslog-openstack.conf.j2
ā”‚Ā Ā      ā”œā”€ā”€ firewall
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ handlers
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”œā”€ā”€ fluentd
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ files
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ filebeat-index-template.json
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ fluentd.repo
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā”œā”€ā”€ openssl_extras.cnf.j2
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ td-agent.conf.j2
ā”‚Ā Ā      ā”œā”€ā”€ heartbeat
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ meta
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ heartbeat.yml.j2
ā”‚Ā Ā      ā”œā”€ā”€ instructions
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”œā”€ā”€ kibana
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ files
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ kibana.repo
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ kibana.yml.j2
ā”‚Ā Ā      ā”œā”€ā”€ logstash
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ files
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā”œā”€ā”€ filebeat-index-template.json
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ logstash.repo
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā”œā”€ā”€ 02-beats-input.conf.j2
ā”‚Ā Ā      ā”‚Ā Ā      ā”œā”€ā”€ logstash.conf.j2
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ openssl_extras.cnf.j2
ā”‚Ā Ā      ā”œā”€ā”€ metricbeat
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ meta
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ metricbeat.yml.j2
ā”‚Ā Ā      ā”œā”€ā”€ nginx
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā”œā”€ā”€ kibana.conf.j2
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ nginx.conf.j2
ā”‚Ā Ā      ā”œā”€ā”€ packetbeat
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ meta
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā”œā”€ā”€ tasks
ā”‚Ā Ā      ā”‚Ā Ā  ā”‚Ā Ā  ā””ā”€ā”€ main.yml
ā”‚Ā Ā      ā”‚Ā Ā  ā””ā”€ā”€ templates
ā”‚Ā Ā      ā”‚Ā Ā      ā””ā”€ā”€ packetbeat.yml.j2
ā”‚Ā Ā      ā””ā”€ā”€ xpack
ā”‚Ā Ā          ā””ā”€ā”€ tasks
ā”‚Ā Ā              ā””ā”€ā”€ main.yml
ā””ā”€ā”€ meta
    ā””ā”€ā”€ main.yml

56 directories, 52 files


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
shell (10,394)Ā 
elasticsearch (547)Ā 
ansible (426)Ā 
centos (98)Ā 
kibana (89)Ā 
logstash (73)Ā 
elk (61)Ā 
playbook (55)Ā 
fluentd (43)Ā 
rhel (30)Ā