Categorizing IPs with Logstash – Private, Public, and GeoIP Enrichment

In the world of network monitoring and log analysis, understanding the type and origin of IP addresses is essential. Are they private, coming from internal networks, or public, coming from external sources?

The Logstash configuration we’re exploring today automates this process efficiently. It identifies private IPs, categorizes public IPs, and enriches data with geographical details for public addresses.

Let’s explore into how it works and how you can use it in your workflows.

Logs often contain source IP addresses (`src_ip`) that can provide valuable insights when properly categorized and enriched. For instance:

Private IPs indicate traffic within internal or reserved networks.
Public IPs signify external interactions, useful for analyzing user behaviors or potential threats.
GeoIP enrichment for public IPs adds geographical data, such as the country, region, and city.

Manually managing this process can be tedious. That’s where the power of Logstash comes in.

Here’s the configuration at a glance:

if [src_ip] {
  cidr {
    address => [ "%{src_ip}" ]
    network => [
      "10.0.0.0/8", "127.0.0.0/8", "169.254.0.0/16", "172.16.0.0/12", "192.0.0.0/24", "192.0.0.0/29", "192.0.0.8/32", "192.0.0.9/32", "192.0.0.10/32", "192.0.0.170/32", "192.0.0.171/32", "192.0.2.0/24", "192.31.196.0/24", "192.52.193.0/24", "192.168.0.0/16", "192.88.99.0/24", "224.0.0.0/4", "100.64.0.0/10", "192.175.48.0/24", "198.18.0.0/15", "198.51.100.0/24", "203.0.113.0/24", "240.0.0.0/4", "::1", "FE80::/10", "FF00::/8"
      ]
    add_field => {"ip_type" => "private_ip"}
  }
  if [src_ip] =~ /^\b(?:\d{1,3}.){3}\d{1,3}\b/ and "private_ip" not in [ip_type] {
    if [src_ip] !~ /^(0.0.0.0)/ {
      mutate { add_field => {"ip_type" => "public_ip"} }
      geoip {
        source => "src_ip"
        target => "src_ip_geoip"
        database => "/etc/logstash/GeoLite2-City.mmdb"
      }
    }
  }
}

The configuration begins by verifying if the `src_ip` field is present in the event. Without this, no further processing occurs.

Using the `cidr` filter, the `[src_ip]` is compared against a comprehensive list of private and reserved IP ranges. Matches are tagged as `private_ip`.

cidr {
  address => [ "%{src_ip}" ]
  network => [ "10.0.0.0/8", "127.0.0.0/8", ... ]
  add_field => {"ip_type" => "private_ip"}
}

If the `src_ip` does not match the private IP list, additional checks are applied:

– Regex Validation: Ensures the IP is in a valid IPv4 format.
– Special IP Exclusion: Skips invalid or special addresses like `0.0.0.0`.

Valid IPs are tagged as `public_ip`.

if [src_ip] =~ /^\b(?:\d{1,3}.){3}\d{1,3}\b/ and "private_ip" not in [ip_type] {
  if [src_ip] !~ /^(0.0.0.0)/ {
  mutate { add_field => {"ip_type" => "public_ip"} }

For public IPs, the `geoip` filter retrieves geographical data from the MaxMind GeoLite2 database, storing it under the `src_ip_geoip` field. This data includes:

– Country
– Region
– City
– Latitude and longitude

geoip {
  source => "src_ip"
  target => "src_ip_geoip"
  database => "/etc/logstash/GeoLite2-City.mmdb"
}

This Logstash configuration is a powerful tool for categorizing IPs and enriching public IP data with geographical insights. Whether you’re monitoring security, analyzing traffic, or optimizing network performance, this setup gives you the actionable data you need.

Implement it today to make your logs more insightful and your analytics more effective!

You may also like: