Logstash: mutate nested field

Ah, the ELK stack. It seems like everybody is using it now. Tons of data in and lightning fast searches. Yelp even created a monitoring solution that works pretty well. This is a solution for developers and data architects. It’s not quite right for the average systems professional as it’s missing some [monitoring] features. I’m getting used to it.

I recently had an issue with an Elastic Search index where I wanted a field to be a float and the first value being sent by Logstash was a long, 99% of the time. Since Elastic Search types dynamically based on the first value, it’s a long.

The solution to this problem is to add a mutate section to the logstash config to catch this value before it gets to Elastic Search and force the float. I tried a number of methods, even tried using the Elastic Search mapping API, but that does not work if you use a daily index as the new index gets created with the (99% of the time) long vs float.

The specific issue was that Topbeat was sending the volume details for each volume and the filesystem.fs.used_p value was nearly always 0 as most volumes are pseudo-filesystems if all systems have a single disk mounted.

The logstash configuration that finally worked for me was the following:

filter {
        if [type] == "filesystem" {
                mutate {
                        convert => [ "[fs][used_p]" , "float" ]
                }   
        }   
}
  • https://www.elastic.co/
  • https://www.elastic.co/products/logstash
  • https://www.elastic.co/products/kibana
  • https://github.com/Yelp/elastalert

Posted

in

by

Tags:

Comments

2 responses to “Logstash: mutate nested field”

  1. Semaev Avatar
    Semaev

    It just does not work on logstash 6.2.4+ versions 🙁

  2. Josh Avatar

    Hey Semaev,

    The correct syntax appears to be:

    filter {
      mutate {
        convert => {
          "[fs][used_p]"  => "float"
        }
      }
    }
    

Leave a Reply

Your email address will not be published. Required fields are marked *