Report generator for Logstash parse failures
While Logstash is great for this job, sometimes it cannot parse some log events because the events have an unknown formatting or my parsing rules don’t match well enough.
I used to manually search for such parse failures in the stored events from time to time. While this worked basically, it required me to remember “ah, maybe I should have a look for parse failures”. This happened not that often, so sometimes I had many parse failures for a long time although they were easy to fix.
Finally, I wrote a simple script to generate a simple report of parse failures happened
in the last seven days. This script is executed as cronjob every seven days and will send
the report via email to me.
Yay, now it’s really easy: I just need to read that tiny mail with parse failures and then decide whether I want to fix them or rather not.
In case anyone is interested, the script can be downloaded here: report_logstash_parse_failures.py
Additionally, for my convenience, I set up a Saved Search in Kibana:
The Saved Search can be imported directly into Kibana after downloading it from here: kibana_saved_search_parse_failures.json