NetFlow for Cybersecurity

NetFlow Analysis Tools

There are many different commercial and open source NetFlow analysis tools in the industry. The following two sections cover several examples of commercial and open source NetFlow analysis tools.

Commercial NetFlow Analysis Tools

Table 4-25 lists the most popular commercial NetFlow monitoring and analysis software packages in the industry today.

Table 4-25 Examples of Commercial NetFlow Monitoring and Analysis Software

Two of the most popular commercial products are Lancope’s StealthWatch solution and Plixer Scrutinizer, as described in greater detail in the sections that follow.

Cisco’s Lancope StealthWatch Solution

Cisco acquired Lancope, a company who produced the StealthWatch solution, which is a key component of the Cisco Cyber Threat Defense (CTD) solution. One of the key benefits of Lancope’s StealthWatch is its capability to scale in large enterprises. It also provides integration with the Cisco Identity Services Engine (ISE) for user identity information. Cisco ISE is a security policy management and control system that you can use for access control and security compliance for wired, wireless, and virtual private network (VPN) connections.

One other major benefit of Lancope’s StealthWatch is its graphical interface, which includes great visualizations of network traffic, customized summary reports, and integrated security and network intelligence for drill-down analysis.

Figure 4-16 shows a screenshot of Lancope’s StealthWatch Management Console (SMC).

Figure 4-16 SMC Web Management Application

Figure 4-17 shows a report of the top applications observed in the network. You can drill down into each application and host to get more detailed information about what is happening in the network.

Figure 4-17 Network Summary Report

Lancope used to have a security research initiative that tracked emerging threat information from around the world, called the StealthWatch Labs Intelligence Center (SLIC). Nowadays, it’s integrated with the Cisco Talos security research team.

Figure 4-18 illustrates the major components of Lancope’s StealthWatch solution.

Figure 4-18 The Lancope’s StealthWatch Solution Components

The following are the primary components of the Lancope StealthWatch solution shown in Figure 4-18:

• StealthWatch Management Console: Provides centralized management, configuration, and reporting of the other StealthWatch components. It can be deployed in a physical server or a virtual machine (VM). The StealthWatch Management Console provides high-availability features (failover), as shown in Figure 4-18.

• FlowCollector: A physical or virtual appliance that collects NetFlow data from infrastructure devices.

• FlowSensor: A physical or virtual appliance that can generate NetFlow data when legacy Cisco network infrastructure components are not capable of producing line-rate, unsampled NetFlow data. Alternatively, the Cisco NetFlow Generator Appliance (NGA) can be used.

• FlowReplicator: A physical appliance used to forward NetFlow data as a single data stream to other devices.

• StealthWatch IDentity: Provides user identity monitoring capabilities. Administrators can search on usernames to obtain a specific user network activity. Identity data can be obtained from the StealthWatch IDentity appliance or through integration with the Cisco ISE.

Lancope’s StealthWatch solution supports a feature called network address translation (NAT) stitching. NAT stitching uses data from network devices to combine NAT information from inside a firewall (or a NAT device) with information from outside the firewall (or a NAT device) to identify which IP addresses and users are part of a specific flow. A great feature of the StealthWatch solution is its ability to perform “NetFlow deduplication.” This feature allows you to deploy several NetFlow collectors within your organization without worrying about double or triple counting the traffic.

Plixer’s Scrutinizer

Plixer’s Scrutinizer is another commercial NetFlow monitoring and analysis software package that has gone through interoperability tests by Cisco. Scrutinizer is used for incident response and network monitoring. Just like several components of Lancope’s StealthWatch solution, Scrutinizer is available as a physical or virtual appliance. Plixer also sells two other products that provide additional network visibility: FlowPro and Flow Replicator.

FlowPro is an appliance that can be deployed in a specific area of the corporate network to perform deep packet inspection (DPI) by combining NetFlow/IPFIX data. Plixer’s Flow Replicator allows several sources of network device and server log data to be replicated to different destinations. Flow Replicator can also be configured as a syslog-to-IPFIX gateway. It converts syslog messages and forwards them on inside IPFIX datagrams.

Open Source NetFlow Monitoring and Analysis Software Packages

The number of open source NetFlow monitoring and analysis software packages is on the rise. You can use these open source tools to successfully identify security threats within your network.

Table 4-26 lists the most popular open source NetFlow monitoring and analysis software packages.

Table 4-26 Examples of Open Source NetFlow Monitoring and Analysis Software

NFdump

NFdump is a set of Linux-based tools that support NetFlow Versions 5, 7, and 9. You can download NFdump from http://nfdump.sourceforge.net and install it from source. Alternatively, you can easily install NFdump in multiple Linux distributions such as Ubuntu using sudo apt-get install nfdump.

Table 4-27 lists all the components of the NFdump toolkit.

Table 4-27 NFdump Components

The command to capture the NetFlow data is nfcapd. All processed NetFlow records are stored in one or more binary files. These binary files are read by nfdump and can be displayed in plaintext to standard output (stdout) or written to another file. Example 4-10 demonstrates how the nfcapd command is used to capture and store NetFlow data in a directory called netflow. The server is configured to listen to port 9996 for NetFlow communication.

Example 4-10 Using the nfcapd Command

omar@server1:~$ />nfcapd -w -D -l netflow -p 9996/>
omar@server1:~$ />cd netflow/>
omar@server1:~/netflow$ />ls -l/>
total 544
-rw-r--r-- 1 omar omar  20772 Jun 18 00:45 nfcapd.201506180040
-rw-r--r-- 1 omar omar  94916 Jun 18 00:50 nfcapd.201506180045
-rw-r--r-- 1 omar omar  84108 Jun 18 00:55 nfcapd.201506180050
-rw-r--r-- 1 omar omar  78564 Jun 18 01:00 nfcapd.201506180055
-rw-r--r-- 1 omar omar 106732 Jun 18 01:05 nfcapd.201506180100
-rw-r--r-- 1 omar omar  73692 Jun 18 01:10 nfcapd.201506180105
-rw-r--r-- 1 omar omar  76996 Jun 18 01:15 nfcapd.201506180110
-rw-r--r-- 1 omar omar    276 Jun 18 01:15 nfcapd.current

Flows are read either from a single file or from a sequence of files. In Example 4-10, a series of files were created by the nfcapd daemon. Example 4-11 shows the command options of the nfcapd daemon command.

Example 4-11 nfcapd Daemon Command Options

omar@ server1:~$ />nfcapd  -h/>
usage nfcapd [options]
-h             this text you see right here
-u userid      Change user to username
-g groupid     Change group to groupname
-w             Sync file rotation with next 5min (default) interval
-t interval    set the interval to rotate nfcapd files
-b host        bind socket to host/IP addr
-j mcastgroup  Join multicast group <mcastgroup>
-p portnum     listen on port portnum
-l basdir      set the output directory. (no default)
-S subdir      Sub directory format. see nfcapd(1) for format
-I Ident       set the ident string for stat file. (default ‘none’)
-H             Add port histogram data to flow file.(default ‘no’)
-n Ident,IP,logdir  Add this flow source - multiple streams
-P pidfile     set the PID file
-R IP[/port]   Repeat incoming packets to IP address/port
-s rate        set default sampling rate (default 1)
-x process     launch process after a new file becomes available
-z             Compress flows in output file.
-B bufflen     Set socket buffer to bufflen bytes
-e             Expire data at each cycle.
-D             Fork to background
-E             Print extended format of netflow data. for debugging purpose only.
-T             Include extension tags in records.
-4             Listen on IPv4 (default).
-6             Listen on IPv6.
-V             Print version and exit.

Example 4-12 demonstrates how to use the nfdump command to process and analyze all files that were created by nfcapd in the netflow directory.

Example 4-12 Processing and Displaying the nfcapd Files with nfdump

omar@server1::~$ />nfdump -R netflow -o extended -s srcip -s ip/flows/>
Top 10 Src IP Addr ordered by flows:
Date first seen          Duration Proto       Src IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2017-01-2222:35:10.805     2.353 any       192.168.1.140     1582(19.5)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.829     2.380 any       192.168.1.130      875(10.8)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.404 any       192.168.1.168      807( 9.9)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:11.219     1.839 any       192.168.1.142      679( 8.4)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.258 any       192.168.1.156      665( 8.2)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.297 any       192.168.1.205      562( 6.9)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.404 any        192.168.1.89      450( 5.5)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:11.050     1.989 any      10.248.91.231      248( 3.1)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:11.633     1.342 any       192.168.1.149      234( 2.9)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:11.040     2.118 any       192.168.1.157      213( 2.6)        0(-nan)        0(-nan)        0        0     0

Top 10 IP Addr ordered by flows:
Date first seen          Duration Proto           IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2017-01-2222:35:10.805     2.353 any       192.168.1.140     1582(19.5)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.353 any             10.8.8.8     1188(14.6)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.297 any         192.168.1.1     1041(12.8)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.829     2.380 any       192.168.1.130      875(10.8)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.404 any       192.168.1.168      807( 9.9)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:11.219     1.839 any       192.168.1.142      679( 8.4)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.258 any       192.168.1.156      665( 8.2)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.297 any       192.168.1.205      562( 6.9)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.825     2.277 any        10.190.38.99      467( 5.8)        0(-nan)        0(-nan)        0        0     0
2017-01-2222:35:10.805     2.404 any        192.168.1.89      450( 5.5)        0(-nan)        0(-nan)        0        0     0

Summary: total flows: 8115, total bytes: 0, total packets: 0, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: 2017-01-2222:35:10 - 2017-01-2222:35:13
Total flows processed: 8115, Blocks skipped: 0, Bytes read: 457128
Sys: 0.009s flows/second: 829924.3   Wall: 0.008s flows/second: 967222.9

In Example 4-12, you can see the top talkers (top hosts that are sending the most traffic in the network). You can refer to the nfdump man pages for details about usage of the nfdump command (using the man nfdump command). Example 4-13 shows an excerpt of the output of the nfdump man pages showing several examples of the nfdump command usage.

Example 4-13 nfdump Man Pages Excerpt

EXAMPLES
       nfdump -r /and/dir/nfcapd.201107110845 -c 100 ‘proto tcp and ( src ip 172.16.17.18 or
dst ip 172.16.17.19 )’ Dumps the first 100 netflow records which match the given filter:
       nfdump -r /and/dir/nfcapd.201107110845 -B Map matching flows as bin-directional single flow.
       nfdump -R /and/dir/nfcapd.201107110845:nfcapd.200407110945 ‘host 192.168.1.2’ Dumps all netflow records of host 192.168.1.2 from July 11 08:45 - 09:45
       nfdump -M /to/and/dir1:dir2 -R nfcapd.200407110845:nfcapd.200407110945 -s record -n 20 Generates the Top 20 statistics from 08:45 to 09:45 from 3 sources
       nfdump -r /and/dir/nfcapd.201107110845 -s record -n 20 -o extended Generates the Top 20 statistics, extended output format
       nfdump -r /and/dir/nfcapd.201107110845 -s record -n 20 ‘in if 5 and bps > 10k’ Generates the Top 20 statistics from flows coming from interface 5
       nfdump -r /and/dir/nfcapd.201107110845 ‘inet6 and proto tcp and ( src port > 1024
and dst port 80 ) Dumps all port 80 IPv6 connections to any web server.

NOTES
       Generating the statistics for data files of a few hundred MB is no problem. However be careful if you want to create statistics of several GB of data. This may consume a lot of memory and can take a while. Flow anonymization has moved into nfanon.

SEE ALSO
       nfcapd(1), nfanon(1), nfprofile(1), nfreplay(1)

NfSen

NfSen is the graphical web-based frontend for NFdump. You can download and obtain more information about NFSen at http://nfsen.sourceforge.net.

SiLK

The SiLK analysis suite is a very popular open source command-line “Swiss army knife” developed by the Computer Emergency Response Team (CERT) at Carnegie Mellon University. Administrators and security professionals combine these tools in various ways to perform detailed NetFlow analysis. SiLK includes numerous tools and plug-ins.

The SiLK packing system includes several applications (daemons) that collect NetFlow data and translate them into a more space-efficient format. SiLK stores these records into service-specific binary flat files for use by the analysis suite. Files are organized in a time-based directory hierarchy.

Elasticsearch, Logstash, and Kibana Stack

Elasticsearch ELK stack is a very powerful open source analytics platform. ELK stands for Elasticsearch, Logstash, and Kibana.

Elasticsearch is the name of a distributed search and analytics engine, but it is also the name of the company founded by the folks behind Elasticsearch and Apache Lucene. Elasticsearch is built on top of Apache Lucene, which is a high-performance search and information retrieval library written in Java. Elasticsearch is a schema-free, full-text search engine with multilanguage support. It provides support for geolocation, suggestive search, auto-completion, and search snippets.

Logstash offers centralized log aggregation of many types, such as network infrastructure device logs, server logs, and also Netflow. Logstash is written in JRuby and runs in a Java virtual machine (JVM). It has a very simple message-based architecture. Logstash has a single agent that is configured to perform different functions in combination with the other ELK components. The following are the four major components in the Logstash ecosystem:

• The shipper: Sends events to Logstash. Typically, remote agents will only run this component.

• The broker and indexer: Receive and index the events.

• The search and storage: Allow you to search and store events.

• The web interface: A web-based interface called Kibana.

Logstash is very scalable because servers running Logstash can run one or more of these aforementioned components independently.

Kibana is an analytics and visualization platform architected for Elasticsearch. It provides real-time summary and charting of streaming data, with the ability to share and embed dashboards.

Marvel and Shield are two additional components that can be integrated with ELK:

• Marvel: Provides monitoring of an Elasticsearch deployment. It uses Kibana to visualize the data. It provides a detailed explanation of things that are happening within the ELK deployment that are very useful for troubleshooting and additional analysis. You can obtain information about Marvel at http://www.elasticsearch.org/overview/marvel.

• Shield: Provides security features to ELK such as role-based access control, authentication, IP filtering, encryption of ELK data, and audit logging. Shield is not free, and it requires a license. You can obtain more information about Shield at http://www.elasticsearch.org/overview/shield.

Elasticsearch also provides integration with Big Data platforms such as Hadoop.