y-ken/fluent-plugin-geoip y-ken/fluent-plugin-geoip

  • Stars
    star
    103
  • Rank 333,046 (Top 7 %)
  • Language
    Ruby
  • License
    Other
  • Created over 11 years ago
  • Updated over 4 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
y-ken/fluent-plugin-geoip
github

y-ken

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Fluentd output plugin to geolocate with geoip.

fluent-plugin-geoip Build Status

Fluentd Filter plugin to add information about geographical location of IP addresses with Maxmind GeoIP databases.

fluent-plugin-geoip has bundled cost-free GeoLite2 Free Downloadable Databases and GeoLite City database by default.
Also you can use purchased GeoIP City database (lang:ja) which costs starting from $50.

The accuracy details for GeoLite City (free) and GeoIP City (purchased) has described at the page below.

Requirements

fluent-plugin-geoip fluentd ruby
>= 1.0.0 >= v1.0.2 >= 2.1
< 1.0.0 >= v0.12.0 >= 1.9

If you want to use this plugin with Fluentd v0.12.x or earlier use 0.8.x.

Compatibility notice

We've removed GeoipOutput since 1.3.0, because GeoipFilter is enough to add information about geographical location of IP addresse.

Dependency

before use, install dependent library as:

# for RHEL/CentOS
$ sudo yum groupinstall "Development Tools"
$ sudo yum install geoip-devel --enablerepo=epel
$ sudo yum install libmaxminddb-devel

# for Ubuntu/Debian
$ sudo apt-get install build-essential
$ sudo apt-get install libgeoip-dev
$ sudo apt-get install libmaxminddb-dev

# for OS X
$ brew install geoip
$ bundle config build.geoip-c --with-geoip-dir=/usr/local/include/

See geoip2_c, if you failed to install geoip2_c.

Installation

install with gem or td-agent provided command as:

# for fluentd
$ gem install fluent-plugin-geoip

# for td-agent
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-geoip

# for td-agent2
$ sudo td-agent-gem install fluent-plugin-geoip

Usage

For GeoipFilter

Note that filter version of geoip plugin does not have handling tag feature.

<filter access.apache>
  @type geoip

  # Specify one or more geoip lookup field which has ip address (default: host)
  geoip_lookup_keys host

  # Specify optional geoip database (using bundled GeoLiteCity databse by default)
  # geoip_database    "/path/to/your/GeoIPCity.dat"
  # Specify optional geoip2 database
  # geoip2_database   "/path/to/your/GeoLite2-City.mmdb" (using bundled GeoLite2-City.mmdb by default)
  # Specify backend library (geoip2_c, geoip, geoip2_compat)
  backend_library geoip2_c

  # Set adding field with placeholder (more than one settings are required.)
  <record>
    city            ${city.names.en["host"]}
    latitude        ${location.latitude["host"]}
    longitude       ${location.longitude["host"]}
    country         ${country.iso_code["host"]}
    country_name    ${country.names.en["host"]}
    postal_code     ${postal.code["host"]}
  </record>

  # To avoid get stacktrace error with `[null, null]` array for elasticsearch.
  skip_adding_null_record  true

  # Set @log_level (default: warn)
  @log_level         info
</filter>

Tips: how to geolocate multiple key

<filter access.apache>
  @type geoip
  geoip_lookup_keys user1_host, user2_host
  <record>
    user1_city      ${city.names.en["user1_host"]}
    user2_city      ${city.names.en["user2_host"]}
  </record>
</filter>

Tips: Modify records without city information

<filter access.apache>
  @type geoip
  geoip_lookup_keys remote_addr
  <record>
    city         ${city.names.en["remote_addr"]}     # skip adding fields if this field is null
    latitude     ${location.latitude["remote_addr"]}
    longitude    ${location.longitude["remote_addr"]}
    country      ${country.iso_code["remote_addr"]}
    country_name ${country.names.en["remote_addr"]}
    postal_code  ${postal.code["remote_addr"]}
  </record>
  skip_adding_null_record true
</filter>

Skip adding fields if incoming remote_addr's GeoIP data is like following:

# the record does not have "city" field
{"continent"=>
  {"code"=>"NA",
   "geoname_id"=>6255149,
   "names"=>
    {"de"=>"Nordamerika",
     "en"=>"North America",
     "es"=>"Norteamรฉrica",
     "fr"=>"Amรฉrique du Nord",
     "ja"=>"ๅŒ—ใ‚ขใƒกใƒชใ‚ซ",
     "pt-BR"=>"Amรฉrica do Norte",
     "ru"=>"ะกะตะฒะตั€ะฝะฐั ะะผะตั€ะธะบะฐ",
     "zh-CN"=>"ๅŒ—็พŽๆดฒ"}},
 "country"=>
  {"geoname_id"=>6252001,
   "iso_code"=>"US",
   "names"=>
    {"de"=>"USA",
     "en"=>"United States",
     "es"=>"Estados Unidos",
     "fr"=>"ร‰tats-Unis",
     "ja"=>"ใ‚ขใƒกใƒชใ‚ซๅˆ่ก†ๅ›ฝ",
     "pt-BR"=>"Estados Unidos",
     "ru"=>"ะกะจะ",
     "zh-CN"=>"็พŽๅ›ฝ"}},
 "location"=>
  {"accuracy_radius"=>1000, "latitude"=>37.751, "longitude"=>-97.822},
 "registered_country"=>
  {"geoname_id"=>6252001,
   "iso_code"=>"US",
   "names"=>
    {"de"=>"USA",
     "en"=>"United States",
     "es"=>"Estados Unidos",
     "fr"=>"ร‰tats-Unis",
     "ja"=>"ใ‚ขใƒกใƒชใ‚ซๅˆ่ก†ๅ›ฝ",
     "pt-BR"=>"Estados Unidos",
     "ru"=>"ะกะจะ",
     "zh-CN"=>"็พŽๅ›ฝ"}}}

We can avoid this behavior changing field order in <record> like following:

<filter access.apache>
  @type geoip
  geoip_lookup_keys remote_addr
  <record>
    latitude     ${location.latitude["remote_addr"]} # this field must not be null
    longitude    ${location.longitude["remote_addr"]}
    country      ${country.iso_code["remote_addr"]}
    country_name ${country.names.en["remote_addr"]}
    postal_code  ${postal.code["remote_addr"]}
    city         ${city.names.en["remote_addr"]}     # adding fields even if this field is null
  </record>
  skip_adding_null_record true
</filter>

Tips: nested attributes for geoip_lookup_keys

See Record Accessor Plugin Helper

NOTE Since v1.3.0 does not interpret host.ip as nested attribute.

Advanced config samples

It is a sample to get friendly geo point recdords for elasticsearch with Yajl (JSON) parser.

<filter access.apache>
  @type                  geoip
  geoip_lookup_keys      host
  <record>
    # lat lon as properties
    # ex. {"lat" => 37.4192008972168, "lon" => -122.05740356445312 }
    location_properties  '{ "lat" : ${location.latitude["host"]}, "lon" : ${location.longitude["host"]} }'
  
    # lat lon as string
    # ex. "37.4192008972168,-122.05740356445312"
    location_string      ${location.latitude["host"]},${location.longitude["host"]}
    
    # GeoJSON (lat lon as array) is useful for Kibana's bettermap.
    # ex. [-122.05740356445312, 37.4192008972168]
    location_array       '[${location.longitude["host"]},${location.latitude["host"]}]'
  </record>

  # To avoid get stacktrace error with `[null, null]` array for elasticsearch.
  skip_adding_null_record  true
</filter>

On the case of using td-agent3 (v1-config), it have to quote { ... } or [ ... ] block with quotation like below.

<filter access.apache>
  @type                  geoip
  geoip_lookup_keys      host
  <record>
    location_properties  '{ "lat" : ${location.latitude["host"]}, "lon" : ${location.longitude["host"]} }'
    location_string      ${location.latitude["host"]},${location.longitude["host"]}
    location_array       '[${location.longitude["host"]},${location.latitude["host"]}]'
  </record>
  skip_adding_null_record  true
</filter>

Tutorial

For GeoipFilter

configuration

<source>
  @type forward
</source>

<filter test.geoip>
  @type    geoip
  geoip_lookup_keys  host
  <record>
    city  ${city.names.en["host"]}
    lat   ${location.latitude["host"]}
    lon   ${location.longitude["host"]}
  </record>
</filter>

<match test.**>
  @type stdout
</match>

result

# forward record with Google's ip address.
$ echo '{"host":"66.102.9.80","message":"test"}' | fluent-cat test.geoip

# check the result at stdout
$ tail /var/log/td-agent/td-agent.log
2016-02-01 12:04:37 +0900 test.geoip: {"host":"66.102.9.80","message":"test","city":"Mountain View","lat":37.4192008972168,"lon":-122.05740356445312}

You can check geoip data format using utils/dump.rb.

$ bundle exec ruby utils/dump.rb geoip2 66.102.3.80
$ bundle exec ruby utils/dump.rb geoip2_compat 66.102.3.80
$ bundle exec ruby utils/dump.rb geoip 66.102.3.80

Placeholders

GeoIP2

You can get any fields in the GeoLite2 database and GeoIP2 Downloadable Databases.

For example(geoip2_c backend):

placeholder attributes output example note
${city.names.en[lookup_field]} "Mountain View" -
${location.latitude[lookup_field]} 37.419200000000004 -
${location.longitude[lookup_field]} -122.0574 -
${country.iso_code[lookup_field]} "US" -
${country.names.en[lookup_field]} "United States" -
${postal.code[lookup_field]} "94043" -
${subdivisions.0.iso_code[lookup_field]} "CA" -
${subdivisions.0.names.en[lookup_field]} "California" -

For example(geoip2_compat backend):

placeholder attributes output example note
${city[lookup_field]} "Mountain View" -
${latitude[lookup_field]} 37.419200000000004 -
${longitude[lookup_field]} -122.0574 -
${country_code[lookup_field]} "US" -
${country_name[lookup_field]} "United States" -
${postal_code[lookup_field]} "94043"
${region[lookup_field]} "CA" -
${region_name[lookup_field]} "California" -

NOTE: geoip2_compat backend supports only above fields.

Related configurations:

  • backend_library: geoip2_compat or geoip2_c
  • geoip2_database: path to your GeoLite2-City.mmdb

GeoIP legacy

Provides these placeholders for adding field of geolocate results.
For more example of geolocating, you can try these websites like Geo IP Address View or View my IP information.

placeholder attributes output example type note
${city[lookup_field]} "Ithaca" varchar(255) -
${latitude[lookup_field]} 42.4277992248535 decimal -
${longitude[lookup_field]} -76.4981994628906 decimal -
${country_code3[lookup_field]} "USA" varchar(3) -
${country_code[lookup_field]} "US" varchar(2) A two-character ISO 3166-1 country code
${country_name[lookup_field]} "United States" varchar(50) -
${dma_code[lookup_field]} 555 unsigned int only for US
${area_code[lookup_field]} 607 char(3) only for US
${region[lookup_field]} "NY" char(2) A two character ISO-3166-2 or FIPS 10-4 code

Further more specification available at http://dev.maxmind.com/geoip/legacy/csv/#GeoIP_City_Edition_CSV_Database_Fields

Related configurations:

  • backend_library: geoip (default)
  • geoip_database: path to your GeoLiteCity.dat

Parameters

GeoipFilter

Note that filter version of geoip plugin does not have handling tag feature.

Plugin helpers

See also Filter Plugin Overview

Supported sections

Parameters

Plugin Common Paramteters

geoip_database (string) (optional)

  • Default value: bundled database GeoLiteCity.dat

Path to GeoIP database file.

geoip2_database (string) (optional)

  • Default value: bundled database GeoLite2-City.mmdb.

Path to GeoIP2 database file.

geoip_lookup_keys (array) (optional)

  • Default_value: ["host"]

Specify one or more geoip lookup field which has IP address.

geoip_lookup_key (string) (optional) (deprecated)

  • Default value: nil.

Use geoip_lookup_keys instead.

skip_adding_null_record (bool) (optional)

  • Default value: nil

Skip adding geoip fields when this valaues to true. On the case of getting nothing of GeoIP info (such as local IP), it will output the original record without changing anything.

backend_library (enum) (optional)

  • Available values: geoip, geoip2_compat, geoip2_c
  • Default value: geoip2_c.

Set backend library.

Articles

TODO

Pull requests are very welcome!!

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Copyright

Copyright (c) 2013- Kentaro Yoshida (@yoshi_ken)

License

Apache License, Version 2.0

This product includes GeoLite data created by MaxMind, available from http://www.maxmind.com.

More Repositories

1

mysql-udf-http

fork of MySQL User-defined function (UDF) for HTTP REST
Shell
86
star
2

fluent-plugin-mysql-replicator

Fluentd input plugin to track insert/update/delete event from MySQL databases.
Ruby
56
star
3

fluent-plugin-anonymizer

Fluentd filter output plugin to anonymize records with MD5/SHA1/SHA256/SHA384/SHA512 algorithms. This data masking plugin protects privacy data such as ID, email, phone number, IPv4/IPv6 address and so on.
Ruby
54
star
4

fluent-plugin-twitter

Fluentd Input/Output plugin to process tweets with Twitter Streaming API.
Ruby
53
star
5

mod_rpaf

Apache-2.2 module for reverse proxy. Set the header for REMOTE_ADDR, HTTPS, and HTTP_PORT from upstream proxy environment variables.
C
42
star
6

fluent-plugin-sentry

fluent-plugin-sentry sends aggregated errors/exception events to Sentry. Sentry is a event logging and aggregation platform.
Ruby
40
star
7

yamabiko

Yamabiko (Fluentd based MySQL/MariaDB Replicator for Elasticsearch/Solr)
Shell
30
star
8

fluent-logger-mruby

A structured logger class for Fluentd. (mruby)
Ruby
26
star
9

elasticsearch_mysql_importer

bulk import file generator as well as nested document from MySQL for elasticsearch bulk api
Ruby
25
star
10

fluent-plugin-mysql-query

Fluentd Input plugin to execute mysql query and fetch rows. It is useful for stationary interval metrics measurement.
Ruby
21
star
11

fluent-plugin-munin

Fluentd Input plugin to fetch munin-node metrics data with custom intervals.
Ruby
20
star
12

mysql-udf-base64

It provides base64 encode/decode UDF (user defined function) for MySQL-5.1/5.5/5.6
C
20
star
13

setup-sentry-centos

Setup Sentry on CentOS 6.x with MySQL and Redis as Queue/Buffer/Cache so quickly.
Shell
15
star
14

growthforecast-quick-install

growthforecast-quick-install
Shell
9
star
15

setup-server-hp

HP Proliant server setup repository.
Perl
8
star
16

fluent-plugin-twilio

Fluentd Output plugin to make a call with twilio. This Multilingual speech synthesis system supports these languages for en, en-gb, es, fr, de, it, ja-jp.
Ruby
8
star
17

fluent-plugin-watch-process

Fluentd Input plugin to collect continual process information via ps command. It is useful for cron/barch process monitoring.
Ruby
8
star
18

package

My public rpm packages. It is available to use for CentOS5/6 (64bit)
CSS
7
star
19

docker-centos-general-rpmbuild-environment

immutable RPM build machine/infrastructure with Docker for CentOS6.x environment.
Shell
7
star
20

milkode-web

Milkode web application template for apache passenger
Ruby
6
star
21

fluent-plugin-pagerduty

Fluentd Output plugin to replay alert notification for PagerDuty.
Ruby
6
star
22

fluent-cron

Wraps commands and reports failing ones to fluentd.
Python
6
star
23

bonnie

The fork of Bonnie++ is a benchmark suite that is aimed at performing a number of simple tests of hard drive and file system performance.
Shell
5
star
24

mysqlftppc-bigram

It's mysqlftppc-bigram-1.6.1 fork repository which has get compatible with MySQL-5.5 and MySQL-5.6
C
5
star
25

twilio-twimlbin

Twimlbin wrapper class. You can use twimlbin.com to host twilio markup language without your own server.
Ruby
4
star
26

fluent-mixin-rewrite-tag-name

Fluentd mixin plugin to rewrite tag like placeholder function of rewrite-tag-filter for your any plugins.
Ruby
4
star
27

mysql_casual_talks_vol4

It is for MySQL Casual Talks Vol.4
Shell
3
star
28

fluent-plugin-guardian

Fluentd service to monitoring and automatic alerting suite.
Ruby
3
star
29

fluent-plugin-gamobile

Fluentd Output plugin to send access report with "Google Analytics for mobile".
Ruby
3
star
30

fluent-mixin-type-converter

Fluentd mixin plugin to provides type conversion function as like as in_tail plugin. It acts calling Fluent::TextParser::TypeConverter as mixin. It will let you get easy to implement type conversion for your own plugins.
Ruby
3
star
31

docker-mroonga-centos

MySQL with Mroonga docker environment for CentOS 6.x
Shell
2
star
32

mysqlftppc-mecab

It's mysqlftppc-mecab-1.6.1 fork repository which has get compatible with MySQL-5.5 and MySQL-5.6
C
2
star
33

itunes-bulk-tag-editor

This apple script could change songs names , artists names , album names , composers names and genres of iTunes songs.
2
star
34

mod_dosdetector-fork

A fork of mod_dosdetector http://sourceforge.net/projects/moddosdetector/
C
1
star
35

mysqlftppc-space

It's mysqlftppc-space-1.6.1 fork repository which has get compatible with MySQL-5.5 and MySQL-5.6
C
1
star
36

fluentd_parser_benchmark

Fluentd Benchmark Tool for optimize parsing speed of logs to get more faster.
Ruby
1
star
37

mysql-udf-mecab

It provides mecab_wakati and mecab_yomi UDF for MySQL-5.1/5.5/5.6
C++
1
star
38

mysqlftppc-suffix

It's mysqlftppc-suffix-1.6.1 fork repository which has get compatible with MySQL-5.5 and MySQL-5.6
C
1
star
39

fluentd-skelton

quick test skelton for Fluentd.
Ruby
1
star
40

multiple_ruby_version_in_apache_passenger

Sample vagrant image to execute multiple versions of ruby in one Apache Passenger
Shell
1
star
41

fluent-parser-elasticsearch

Fluentd Parser Plugin to add choice for `format` option for elasticsearch logs with `in_tail` plugin.
Ruby
1
star
42

fluent-plugin-feedly

Fluentd input plugin to fetch RSS/ATOM feed via feedly Cloud API.
Ruby
1
star
43

mysqlftppc-snowball

It's mysqlftppc-snowball-1.6.1 fork repository which has get compatible with MySQL-5.5 and MySQL-5.6
C
1
star
44

fluent-plugin-boundio

Fluent Output plugin to make a call with boundio by KDDI. This Multilingual speech synthesis system uses VoiceText. NOTE: Boundio will be closed on the 30th Sep 2013.
Ruby
1
star