Apache access logs in Kibana – part 2

KibanaAfter writing my previous article about shipping Apache logs to Elasticsearch via Logstash, I changed my setup for various reasons. But I still visualize them using Kibana.

I used to use nc to ship the logs to logstash, as I ship my logs from a Raspberry Pi. Back in the days when I wrote this original article, logstash-shipper was the program to use for this purpose. However, this application was not available for arm architecture, and that’s the reason I used nc.

But quickly I noticed this was not a very good solution. When my ELK machine was not available (back in the days logstash used to crash because of a DNS lookup bug), nc started to use 100% CPU. As I was running a lot of virtual hosts, all using a seperate nc instance, the CPU became quite overloaded. So I started looking for some other solution.

Meanwhile, logstash-shipper was replaced by filebeat. And as you can read in this article, I runs perfectly on a Raspberry Pi, without too much configuration. A benefit of using filebeat is also that when your ELK server in unreachable, it will cache the events until it is available again and inserts them with the original timestamp.

Well, enough talking, let’s set this thing up!

First of all, follow the setup I describe here.  The filebeat.yml I provide in that article is already “listening” to the apache access log, so we’re done on this side.

The input for logstash is not much different from the one in the previous article, we only need to use a separate input, the beats input.

This is what the input will look like:

input {
 beats {
   port => 5045
   type => "apache_access"
 }
}

filter {
 if [type] == "apache_access" {
    grok {
        match => { "message" => "%{HOSTNAME:vhost}\:%{NUMBER:port} %{COMBINEDAPACHELOG}"}
    }

geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
    }
    mutate {
      convert => [ "[geoip][coordinates]", "float"]
    }
    # do GeoIP lookup for the ASN/ISP information.
    geoip {
      database => "/etc/logstash/GeoIPASNum.dat"
      source => "clientip"
    }
 }
}

output {
 if [type] == "apache_access" {
  elasticsearch { hosts => localhost }
}
}

One sidenote: I have apache log to the accesslog in vhost_combined format. Out of the box apache logs in combined format. If you use this format, the match line will have to look like this:

match => { "message" => "%{COMBINEDAPACHELOG}"}

Well, that will be all. Your setup should work now. Happy logging!

 

One thought on “Apache access logs in Kibana – part 2”

Leave a Reply

Your email address will not be published. Required fields are marked *