Elastic Beats on Raspberry Pi

Beats are the new (log) shippers by Elastic. They’re available for various architectures, installable via repositories, but not for ARM architecture. So how do we install them on a Raspberry Pi? Read on to find out!

I use beats on almost al my machines, including some Raspberry Pis. There’s not a very easy way of installing it, no repository is available for ARM architecture. But in the end it’s not so hard to get it running.

I wanted to save myself from compiling it myself. So I found out Elastic posts nightly builds of their beats, for various architectures, including ARM! This makes it a lot more easy to get it running on a Raspberry.

For the rest of this article I focus on filebeat, but you can use it for any beat, if you adjust the naming accordlingly.

First, we need the filebeat binary. Head to https://beats-nightlies.s3.amazonaws.com/index.html?prefix=jenkins/ to get to the nightly builds. Go to the filebeat directory (or any orther beat if you like) and open the most recent directory underneath that (that is with the highest number), so we have the most recent build. Look for the file ending on “-linux-arm” and download it via wget.

$ wget https://beats-nightlies.s3.amazonaws.com/jenkins/filebeat/642-9cd369649054d0184adb0ec54ab4f1679b8c4293/filebeat-linux-arm

Don’t use this URL, a more recent will be available by the time you read this!

Now we create some directories and move the binary to its own directory to have it all look neat.

$ sudo mkdir /opt/filebeat
$ sudo mkdir /etc/filebeat
$ sudo mv filebeat-linux-arm /opt/filebeat/

Don’t forget to make the binary exectuable.

$ sudo chmod +x /opt/filebeat/filebeat-linux-arm

I made a sample configuration file, but feel free to adjust it. Using this sample we monitor the auth.log file, so you can see who is logging on to the system, perform sudo commands, etcetera. Also we monitor the apache access and error logs.

$ sudo vi /etc/filebeat/filebeat.yml

And paste this sample config file into it:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
      input_type: log
      document_type: auth
      scan_frequency: 1s

    -
      paths:
        - /var/log/apache2/access.log
      input_type: log
      document_type: apache_access
      scan_frequency: 1s

    -
      paths:
        - /var/log/apache2/error.log
      input_type: log
      document_type: apache_error
      scan_frequency: 1s

output:
  elasticsearch:
    hosts: ["elk-hostname:9200"]
    index: "filebeat"

Replace elk-hostname by your host running Elasticsearch.

Now we have the binary in place, we have a sample config, we only need to get it running. We could start it on the command line, but it would be much nicer to have it running as a service. No worries, I also created a service script for that.

$ cd /lib/systemd/system
$ sudo vi filebeat.service

Now, paste this script:

[Unit]
Description=filebeat
Documentation=https://www.elastic.co/guide/en/beats/filebeat/current/index.html
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/opt/filebeat/filebeat-linux-arm -c /etc/filebeat/filebeat.yml
Restart=always

[Install]
WantedBy=multi-user.target

We only need to tell the system to have this service start at boot.

$ systemctl enable filebeat

The service is now enabled to start again after you reboot the system. But it’s still not running at this moment.  You can start it by using

$ sudo service filebeat start

That’s all folks!

You can use this also for other beats, like topbeat, to monitor the cpu usage. Only adjust the names accordingly.

Have fun using the beats on your Raspberry Pi!

22 thoughts on “Elastic Beats on Raspberry Pi”

  1. Thanks, René. I was also surprised to see that Elastic does nightly ARM builds even though they don’t provide ARM binaries in their stable releases. This is great for personal use or testing, but for production environments, we’ll still have to build the latest stable release from source. Hopefully there will eventually be official ARM support.

  2. Thank you Rene, nice tutorial. Have followed and checked thoroughly but cannot get the filebeat.service to run. The error I get is:-
    Exiting: error initialising publisher: Error loading template /etc/filebeat/filebeat.template.json: open /etc/filebeat/filebeat.template.json: no such file or directory.

    I have checked the filesystem and this json file does not exist. Is there a dependency I am missing or some additional commandfu I need to do?

  3. using pi 3 … get these error messages:

    Dec 17 01:38:45 pihole systemd[1]: filebeat.service holdoff time over, scheduling restart.
    Dec 17 01:38:45 pihole systemd[1]: Stopping filebeat…
    Dec 17 01:38:45 pihole systemd[1]: Starting filebeat…
    Dec 17 01:38:45 pihole systemd[1]: filebeat.service start request repeated too quickly, refusing to start.
    Dec 17 01:38:45 pihole systemd[1]: Failed to start filebeat.
    Dec 17 01:38:45 pihole systemd[1]: Unit filebeat.service entered failed state.

    set the filebeat.yml as

    output:
    elasticsearch:
    hosts: [“http://my-elkhost-ip:9200”]
    index: “filebeat”
    template.enabled: false

    after set the template.enabled: false, it stop looking of the template, but the service still can be started.

    1. Did you make the filebeat binary executable? Also did you adjust the path to the binary in /lib/systemd/system/filebeat.service?

      1. Rene: Thanks for reply me question.

        Yes. I follow the precisely of your blog, edit the file in /lib/systemd/system/filebeat.service .

        Heere is the full error message I get from syslog on the pi:

        Dec 17 16:18:01 pihole systemd[1]: Started filebeat.
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: panic: runtime error: invalid memory address or nil pointer dereference
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0xb2834]
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: goroutine 34 [running]:
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: panic(0x4fe308, 0x10b1c008)
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/tools/org.jenkinsci.plugins.golang.GolangInstallation/1.7/src/runtime/panic.go:500 +0x33c
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: sync/atomic.addUint64(0x10c523ac, 0x1, 0x0, 0x1, 0x0)
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/tools/org.jenkinsci.plugins.golang.GolangInstallation/1.7/src/sync/atomic/64bit_arm.go:31 +0x68
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: github.com/elastic/beats/filebeat/prospector.(*Prospector).startHarvester(0x10c52320, 0x10c26d20, 0x11, 0x0, 0x0, 0x0, 0x75dbc8, 0x10cecab0, 0x347f, 0x0, …)
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/workspace/Filebeat/src/github.com/elastic/beats/filebeat/prospector/prospector.go:239 +0x30c
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: github.com/elastic/beats/filebeat/prospector.(*ProspectorLog).scan(0x10c4c320)
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/workspace/Filebeat/src/github.com/elastic/beats/filebeat/prospector/prospector_log.go:232 +0x3f4
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: github.com/elastic/beats/filebeat/prospector.(*ProspectorLog).Run(0x10c4c320)
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/workspace/Filebeat/src/github.com/elastic/beats/filebeat/prospector/prospector_log.go:77 +0xd8
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: github.com/elastic/beats/filebeat/prospector.(*Prospector).Run(0x10c52320, 0x0)
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/workspace/Filebeat/src/github.com/elastic/beats/filebeat/prospector/prospector.go:140 +0x1c4
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: github.com/elastic/beats/filebeat/crawler.(*Crawler).Start.func1(0x10c25320, 0x0, 0x0, 0x10c52320)
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/workspace/Filebeat/src/github.com/elastic/beats/filebeat/crawler/crawler.go:59 +0xf4
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: created by github.com/elastic/beats/filebeat/crawler.(*Crawler).Start
        Dec 17 16:18:02 pihole filebeat-linux-arm[2861]: /home/jenkins/workspace/Filebeat/src/github.com/elastic/beats/filebeat/crawler/crawler.go:60 +0x498

        it seems that I got a runtime error from the filebeat ?

        Any idea?

        1. What version of filebeat are you using? I mean, what number did you download from the nightly builds? Looks like this one is not working correctly. Can you try to download from 2 days previous to the one you are using now?

          1. Rene:
            Thanks again. I went back to 2016-11-12T09 (the very last version listed 1009). It works this time.

          2. Great! Glad you have it working now 🙂 This is the downside of the nightly builds, they’re not always that stable, although 99% of the time they are.

  4. Hey René, I have exactly the same problem as “sam” with build 1084 and 1091. I got the same output/errors.

    I have also read that the problem does not occure in build 1009 – but it is not available on Jenkins anymore. Do you have the source from build 1009 somewhere?

    Thanks a lot,
    Thomas

  5. Hi Rene
    Thanks for this. Like others I did not know Elastic had compiled versions for ARM available (even if not official release) – very useful. As you suggest I would advise people to try another download revision if the first does not work. I used 1097 which failed to start as a service. However 1034 works fine on my Pi 3.
    regards – david

  6. Hello,

    I’ve this error on /opt/filebeat/logs/filebeat :
    2017-05-10T08:56:41+02:00 ERR Connecting error publishing events (retrying): Get http://elk_server:9200: dial tcp xxx.xx.xx.xxx:9200: getsockopt: connection refused
    2017-05-10T08:57:02+02:00 INFO Non-zero metrics in the last 30s: beat.memstats.memory_alloc=73824 beat.memstats.memory_total=73824

    What I can do please?

    1. Seems like the ELK server is actively terminating the connection. Do you have a firewall active on the ELK server? Did you adjust the configuration of Elasticsearch to accept connections from the network?

  7. im using elk docker, and all using output like this (eg: my ES) ;
    hosts: [“localhost:9200”]

    my question is, for filebeat, the output, am i should put it like this?
    hosts: [“localhost:5044”]
    *5044 because googling

    my second question is, is there any way to create systemctl service? it seems that, after i followed all the steps given, i cannot see the port 5044 to be LISTEN

    1. You can configure filebeat to send its data to port 5044, but this port is usually used by logstash. So make sure logstash is running on your Elasticsearch machine. Beware that you are sending to localhost in the example!

      Filebeat sending its output to 9200 on localhost only works if filebeat is running on the ELK machine itself. Otherwise you will have to adjust localhost to the IP address of your ELK server. Using port 9200 filebeat will insert the data directly into elasticsearch, bypassing logstash. Also make sure you configured elasticsearch to listen on ports other than localhost.

  8. hi rene,
    so I set my beat output to
    hosts=”localhost:5044″

    but, when I do netstat -an | grep LISTEN , returns nothing. Please help.

    another question, can you show me on filebeat passing data into logstash? specifically suricata eve.json

    Thanks

  9. thank you so much Rene! I’m using raspberry pi as my ELK + Suricata project. I don’t know what to do, without your Filebeat tutorial!
    Thank you so much!
    *warm hugs

Leave a Reply

Your email address will not be published. Required fields are marked *