Monday, August 03, 2015

Notes on setting up an ELK stack and logstash-forwarder

I set up the ELK stack a while ago and I want to jot down some notes on installing and configuring it.  I was going to write "before I forget how to do it", but that's not true anymore, because I have ansible playbooks and roles for this setup. As I said before, using ansible as executable documentation has been working really well for me. I still need to write this blog post though just so I refresh my memory about the bigger picture of ELK when I revisit it next.

Some notes:

  • Used Jeff Geerling's ansible-role-logstash for the main setup of the ELK server I have
  • Used logstash-forwarder (used to be called lumberjack) on all servers that need to send their logs to the ELK server
  • Wrapped the installation and configuration of logstash-forwarder into a simple ansible role which installs the .deb file for this package and copies over a templatized logstash-forwarder.conf file; here is my ansible template for this file
  • Customized the lumberjack input config file on the ELK server (still called lumberjack, but actually used in conjunction with the logstash-forwarder agents running on each box that sends its logs to ELK); here is my /etc/logstash/conf.d/01-lumberjack-input.conf file
  • Added my app-specific config file on the ELK server; here is my /etc/logstash/conf.d/20-app.conf file with a few things to note
    • the grok stanza applies the 'valid' tag only to the lines that match the APPLOGLINE pattern (see below for more on this pattern)
    • the 'payload' field of any line that matches the APPLOGLINE pattern is parsed as JSON; this is nice because I can change the names of the fields in the JSON object in the log file and all these fields will be individually shown in ELK
    • all lines that are not taggeed as 'valid' will be dropped
  • Created a file called myapp in the /opt/logstash/patterns directory on the ELK server; this file contains all my app-specific patterns referenced in the 20-app.conf file above, in this example just 1 pattern: 
    • APPLOGLINE \[myapp\] %{TIMESTAMP_ISO8601:timestamp}Z\+00:000 \[%{WORD:severity}\] \[myresponse\] \[%{NUMBER:response}\] %{GREEDYDATA:payload}
    • this patterns uses predefined logstash patterns such as TIMESTAMP_ISO8601, WORD, NUMBER and GREEDYDATA
    • note the last field called payload; this is the JSON payload that gets parsed by logstash



Using AWS CloudWatch Logs and AWS ElasticSearch for log aggregation and visualization

If you run your infrastructure in AWS, then you can use CloudWatch Logs and AWS ElasticSearch + Kibana for log aggregation/searching/visuali...