Skip to content

Network flow Monitoring (Netflow, sFlow and IPFIX) with the Elastic Stack

License

Notifications You must be signed in to change notification settings

davama/elastiflow

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

47 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ElastiFlow™

ElastiFlow™ provides network flow data collection and visualization using the Elastic Stack. As of version 2.0.0 it supports Netflow v5/v9, sFlow and IPFIX flow types (1.x versions support only Netflow v5/v9).

Overview

I was inspired to create ElastiFlow™ following the overwhelmingly positive feedback received to an article I posted on Linkedin... WTFlow?! Are you really still paying for commercial solutions to collect and analyze network flow data?

Getting Started

ElastiFlow™ is built using the Elastic Stack, including Elasticsearch, Logstash and Kibana. Refer to the following compatibility chart to choose a release of ElastiFlow™ that is compatible with the version of the Elastic Stack you are using.

Elastic Stack ElastiFlow™ 1.x ElastiFlow™ 2.x
6.2
6.1
6.0
5.6
5.5
5.4

NOTE: The instructions that follow are for ElastiFlow™ 2.x.

Setting up Elasticsearch

Currently there is no specific configuration required for Elasticsearch. As long as Kibana and Logstash can talk to your Elasticsearch cluster you should be ready to go. At high ingest rates (>10K flows/s), or for data redundancy and high availability, a multi-node cluster is recommended.

Setting up Logstash

To use ElastiFlow™ you will need to install the community supported sFlow codec for Logtsash. It is also recommended that you always use the latest version of Netflow codec. This can achieved by running the following commands:

$ LS_HOME/bin/logstash-plugin install logstash-codec-sflow
Validating logstash-codec-sflow
Installing logstash-codec-sflow

$ LS_HOME/bin/logstash-plugin update logstash-codec-netflow
Updating logstash-codec-netflow
Updated logstash-codec-netflow 3.10.0 to 3.11.2

There are four sets of configuration files provided within the logstash/elastiflow folder:

logstash
+-- elastiflow
    |-- conf.d  (contains the logstash pipeline)
    |-- dictionaries (yaml files used to enrich raw flow data)
    |-- geoipdbs  (contains GeoIP databases)
    +-- templates  (contains index templates)

Copy the elastiflow directory to the location of your Logstash configuration files (e.g. on RedHat/CentOS this would be /etc/logstash/elastiflow.

Rather than directly editing the pipeline configuration files for your environment, environment variables can be used. The supported environment variables are:

Environment Variable Description Default Value
ELASTIFLOW_GEOIP_DB_PATH The path where the GeoIP DBs are located /etc/logstash/geoipdbs
ELASTIFLOW_DICT_PATH The path where the dictionary files are located /etc/logstash/dictionaries
ELASTIFLOW_TEMPLATE_PATH The path to where index templates are located /etc/logstash/templates
ELASTIFLOW_RESOLVE_IP2HOST Enable/Disable DNS requests false
ELASTIFLOW_NAMESERVER The DNS server to which the dns filter should send requests 127.0.0.1
ELASTIFLOW_KEEP_ORIG_DATA If set to false the original netflow, ipfix and sflow objects will be deleted prior to indexing. This can save disk space without affecting the provided dashboards. However the original flow fields will no longer be available if they are desired for additional analytics. true
ELASTIFLOW_ES_HOST The Elasticsearch host to which the output will send data 127.0.0.1:9200
ELASTIFLOW_ES_USER The password for the connection to Elasticsearch elastic
ELASTIFLOW_ES_PASSWD The username for the connection to Elasticsearch changeme
ELASTIFLOW_NETFLOW_HOST[6] The IPv4/6 address from which to listen for Netflow messages 0.0.0.0/::1
ELASTIFLOW_NETFLOW_PORT The UDP port on which to listen for Netflow messages 2055
ELASTIFLOW_NETFLOW_LASTSW_TIMESTAMP Enable/Disable setting @timestamp with the value of netflow.last_switched false
ELASTIFLOW_NETFLOW_TZ The timezone of netflow.last_switched UTC
ELASTIFLOW_SFLOW_HOST[6] The IPv4/6 address from which to listen for sFlow messages 0.0.0.0/::1
ELASTIFLOW_SFLOW_PORT The UDP port on which to listen for sFlow messages 6343
ELASTIFLOW_IPFIX_TCP_HOST[6] The IPv4/6 address from which to listen for IPFIX messages via TCP 0.0.0.0/::1
ELASTIFLOW_IPFIX_TCP_PORT The port on which to listen for IPFIX messages via TCP 4739
ELASTIFLOW_IPFIX_UDP_HOST[6] The IPv4/6 address from which to listen for IPFIX messages via UDP 0.0.0.0/::1
ELASTIFLOW_IPFIX_UDP_PORT The port on which to listen for IPFIX messages via UDP 4739

The files profile.d/elastiflow.sh and logstash.service.d/elastiflow.conf are provided to help you setup the environment variables. For example, if you will be running Logstash using systemd, simply complete following steps...

  1. Edit the provided logstash.service.d/elastiflow.conf for your environment and needs.
  2. Copy the file to /etc/systemd/system/logstash.service.d/elastiflow.conf
  3. Run systemctl daemon-reload to pickup the new systemd configuration files
  4. Start Logstash by running systemctl start logstash

WARNING: DNS resolution of IP addresses to hostnames is controlled by the ELASTIFLOW_RESOLVE_IP2HOST environment variable. Within the pipeline caching is disabled for the dns filter as this causes performance issues due to the blocking nature of the cache lookups. For the best DNS performance it is recommended to use a local dnsmasq process to handle caching and to forward any unresolved lookups to upstream DNS servers. This is the reason that ELASTIFLOW_NAMESERVER defaults to 127.0.0.1. When receiving very high rates of flow records, leaving DNS resolution disabled will ensure the best performance.

Logstash can be configured to load the ElastiFlow™ pipeline using one of the following methods:

  • From the command line using the path.config option.
$ LS_HOME/bin/logstash --path.config=/etc/logstash/elastiflow/conf.d
  • By setting path.config in logstash.yml
path.config: /etc/logstash/elastiflow/conf.d
  • If using Logstash 6.0 or above, ElastiFlow™ can also be started ba adding it to pipelines.yml
- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d"

Setting up Kibana

As of Kibana 5.6 an API (yet undocumented) is available to import and export Index Patterns. The JSON file which contains the Index Pattern configuration is kibana/elastiflow.index_pattern-json. To setup the elastiflow-* Index Pattern run the following command:

curl -X POST -u USERNAME:PASSWORD http://KIBANASERVER:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/PATH/TO/elastiflow.index_pattern.json

Finally the vizualizations and dashboards can be loaded into Kibana by importing the elastiflow.dachboards.json file from within Kibana. This is done from the Management - > Saved Objects page.

Dashboards

The following dashboards are provided.

NOTE: The dashboards are optimized for a monitor resolution of 1920x1080.

Overview

Overview

Top-N

There are separate Top-N dashboards for Top Talkers, Top Services and Top Conversations. Conversation Partners

Geo Location

There are separate Geo Loacation dashboards for Client/Server and Source/Destination perspectives. Geo Location

Autonomous Systems

Provides a view of traffic to and from Autonomous Systems (public IP ranges) Autonomous Systems

Flow Exporters

Flow Exporters

Traffic Analysis

Traffic Analysis

Flow Records

Flow Records

Attribution

This product includes GeoLite data created by MaxMind, available from (http://www.maxmind.com)

About

Network flow Monitoring (Netflow, sFlow and IPFIX) with the Elastic Stack

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Shell 100.0%