This chapter covers the following topics:
Packet Capture: Sniffer
Nexus Platform Tools
NetFlow
Network Time Protocol (NTP)
Embedded Event Manager (EEM)
Troubleshooting is an art that requires both in-depth knowledge on the subject and the ability to verify operations and isolate the incorrect behavior. If a network problem arises and an engineer has a topology of hundreds or thousands of devices, troubleshooting seems difficult at first glance. When part of the problematic topology is presented, troubleshooting the network issue becomes much easier. Proper tools and the right view of the problematic topology can quickly isolate the problem, thereby reducing large-scale network impact. This chapter focuses on the various tools available on the Nexus platform that can help in troubleshooting and day-to-day operation.
Packet Capture: Network Sniffer
NX-OS provides a command-line interface (CLI) that assists with troubleshooting various complex issues. However, in some scenarios, the show and debug commands do not yield sufficient information to isolate the problematic direction of the packet flow. In such situations, performing a packet capture helps. Forwarding issues require isolating the direction of the problem and understanding whether the packet is actually reaching the far end device. Understanding the packet flow between two directly connected devices requires taking three perspectives:
Determining whether the originating router is transmitting the packet across the network medium
Determining whether the packet is being received on the destination router
Examining packets flowing across the network medium
This is where concept of network sniffing comes into play. Network sniffing is the technique of intercepting the traffic that passes over the transmission medium for the protocol and for deep packet analysis. Not only does packet sniffing help with troubleshooting packet forwarding issues, but security experts also heavily use it to perform deep analysis of the network and find security holes.
Performing a network sniffer capture requires a PC with a packet capture tool, such as Wireshark, attached to the switch. A mirror copy of the relevant traffic is copied and sent to the destination interface, where it is captured by the packet capture tool and is available for analysis. Figure 2-1 shows a Nexus switch connected between two routers and a capture PC that has Wireshark installed to capture the traffic flowing between routers R1 and R2.
)
Figure 2-1 Sniffer Setup on Nexus Switch
On Cisco devices, the sniffing capability is called a Switched Port Analyzer (SPAN) feature. The source port is called the monitored port and the destination port is called the monitoring port. The SPAN feature on NX-OS is similar in Cisco IOS, but different Nexus switches have different capabilities, based on the hardware support. The following source interfaces can be used as SPAN source interfaces:
Ethernet
Fabric Expander (FEX) ports/Fabric port-channels
Port-channel
VLAN, or VLAN-based SPAN (VSPAN)
Remote SPAN (RSPAN) VLAN
Inband interfaces to the control plane CPU (on Nexus 7000, this feature is supported only on default virtual device context [VDC])
FCoE ports
To enable a port to forward the spanned traffic to the capture PC, the destination interface is enabled for monitoring with the interface parameter command switchport monitor. The destination ports are either an Ethernet or Port-Channel interface configured in access or trunk mode. The SPAN session is configured using the command monitor session session-number, under which the source interface is specified with the command source interface interface-id [rx|tx|both]. The rx option is used to capture the ingress (incoming) traffic, whereas the tx option is used to capture the egress (outgoing) traffic. By default, the option is set to both, which captures both ingress and egress traffic on the configured source interface. The destination interface is specified with the command destination interface interface-id. By default, the monitor session is in shutdown state and must be manually un-shut for the SPAN session to function.
Example 2-1 illustrates a SPAN session configuration on a Nexus switch. Notice that, in this example, the source interface is a range of interfaces, along with the direction of the capture.
Example 2-1 SPAN Configuration on NX-OS
NX-1(config)# interface Ethernet4/3 NX-1(config-if)# switchport NX-1(config-if)# switchport monitor NX-1(config-if)# no shut NX-1(config)# monitor session 1 NX-1(config-monitor)# source interface Ethernet4/1-2 both NX-1(config-monitor)# source interface Ethernet5/1 rx NX-1(config-monitor)# destination interface Ethernet4/3 NX-1(config-monitor)# no shut NX-1(config-monitor)# exit
Example 2-2 displays the status of the monitor session. In this example, the rx, tx, and both fields are populated for interface Eth4/1 and Eth4/2, but the interface Eth5/1 is listed only for the rx direction. There is also an option to filter VLANS under the monitor session using the filter vlan vlan-id command.
Example 2-2 Verifying SPAN Session
NX-1# show monitor session 1
session 1
---------------
type : local
state : up
source intf :
rx : Eth4/1 Eth4/2 Eth5/1
tx : Eth4/1 Eth4/2
both : Eth4/1 Eth4/2
source VLANs :
rx :
tx :
both :
filter VLANs : filter not specified
destination ports : Eth4/3
Legend: f = forwarding enabled, l = learning enabled
The default behavior of a SPAN session is to mirror all traffic to the destination port, but NX-OS also provides the capability to perform a filter on the traffic to be mirrored to the destination port. To filter the relevant traffic, an access control list (ACL) is created, to be referenced in the SPAN session configuration by using the filter access-group acl command. Example 2-3 illustrates the filtering configuration on the SPAN session and verification using the show monitor session command.
Example 2-3 Filtering SPAN Traffic: Configuration and Verification
NX-1(config)# ip access-list TEST-ACL
NX-1(config-acl)# permit ip 100.1.1.0/24 200.1.1.0/24
NX-1(config-)# exit
NX-1(config)# monitor session 1
NX-1(config-monitor)# filter access-group TEST-ACL
NX-1(config-monitor)# exit
NX-1# show monitor session 1
session 1
---------------
type : local
state : up
acl-name : TEST-ACL
source intf :
rx : Eth4/1 Eth4/2 Eth5/1
tx : Eth4/1 Eth4/2
both : Eth4/1 Eth4/2
source VLANs :
rx :
tx :
both :
filter VLANs : filter not specified
destination ports : Eth4/3
Legend: f = forwarding enabled, l = learning enabled
Encapsulated Remote SPAN
Encapsulated Remote SPAN (ERSPAN) is a SPAN feature in which the SPAN traffic is encapsulated to IP-GRE frame format, to support remote monitoring traffic over an IP network. ERSPAN enables monitoring of multiple remote switches across the network—that is, the ERSPAN spans traffic from source ports across multiple switches to the destination switch, where a network analyzer is connected. An ERSPAN session consists of the following components:
ERSPAN ID
ERSPAN source session
GRE-encapsulated traffic
ERSPAN destination session
The ERSPAN ID is used to distinguish among multiple source devices, sending spanned traffic to one single centralized server.
Figure 2-2 shows a network topology with ERSPAN setup. Two Nexus switches are connected by a routed network. The N6k-1 switch is configured as the ERSPAN-source with a local source SPAN port, and the destination port is located in an IP network on the N7k-1 switch. The GRE-encapsulated packets are transmitted across the IP network toward the destination switch, where they are decapsulated and sent to the traffic analyzer.
)
Figure 2-2 ERSPAN Deployment
The source and destination sessions can be configured on different switches separately for the source traffic in ingress, egress, or both directions. The ERSPAN is configured to span traffic on Ethernet ports, VLANs, VSANs, and FEX ports. The destination port remains in monitoring state and does not participate in the spanning tree or any Layer 3 protocols. Example 2-4 illustrates the configuration of both the source ports and destination ports on two different Nexus switches. Note that the ERSPAN-ID should be the same on both switches.
Example 2-4 ERSPAN Configuration
! ERSPAN Source Configuration N6k-1(config)# monitor session 10 type erspan-source N6k-1(config-erspan-src)# erspan-id 20 N6k-1(config-erspan-src)# vrf default N6k-1(config-erspan-src)# destination ip 192.168.1.10 N6k-1(config-erspan-src)# source interface ethernet 1/10 N6k-1(config-erspan-src)# no shut N6k-1(config-erspan-src)# exit N6k-1(config)# monitor erspan origin ip-address 192.168.1.1 global
! ERSPAN Destination Configuration N7k-1(config)# monitor session 10 type erspan-destination N7k-1(config-erspan-dst)# erspan-id 10 N7k-1(config-erspan-dst)# source ip 192.168.1.10 N7k-1(config-erspan-dst)# destination interface e1/3 N7k-1(config-erspan-dst)# no shut
For the ERSPAN source session to come up, the destination IP should be present in the routing table. The ERSPAN session status is verified using the command show monitor session session-id. Example 2-5 demonstrates the verification of both the source and destination ERSPAN sessions.
Example 2-5 ERSPAN Session Verification
N6k-1# show monitor session 10
session 10
---------------
type : erspan-source
state : up
erspan-id : 20
vrf-name : default
destination-ip : 192.168.1.10
ip-ttl : 255
ip-dscp : 0
acl-name : acl-name not specified
origin-ip : 192.168.1.1 (global)
source intf :
rx : Eth1/10
tx : Eth1/10
both : Eth1/10
source VLANs :
rx :
source VSANs :
rx :
N7k-1# show monitor session 10
session 10
---------------
type : erspan-destination
state : up
erspan-id : 10
source-ip : 192.168.1.10
destination ports : Eth1/3
Legend: f = forwarding enabled, l = learning enabled
SPAN on Latency and Drop
Both SPAN and ERSPAN provide the capability to apply filters to SPAN-specific traffic based on protocol and IP addressing. Often users or applications report high latency or experience traffic drops between the source and destination, making it hard to figure out where the drop is happening. In such instances, gaining visibility of traffic that is impacting users is always helpful during troubleshooting and can both minimize the service impact and speed up the troubleshooting process.
NX-OS provides the capability to span the traffic based on the specified latency thresholds or based on drops noticed in the path. These capabilities are available for both SPAN and ERSPAN.
SPAN-on-Latency
The SPAN-on-Latency (SOL) feature works a bit differently than the regular SPAN session. In SOL, the source port is the egress port on which latency is monitored. The destination port is still the port where the network analyzer is connected on the switch. The latency threshold is defined on the interface that is being monitored using the command packets latency threshold threshold-value. When the packets cross or exceed the specified threshold, the SPAN session is triggered and captures the packets. If the threshold value is not specified under the interface, the value is truncated to the nearest multiple of 8.
Example 2-6 illustrates the SOL configuration, in which packets are sniffed only at the egressing interface Eth1/1 and Eth1/2 for flows that have latency more than 1μs (microsecond). The packet latency threshold configuration is per port for 40G interfaces but if there are 4x10G interfaces, they share the same configuration. For this reason, Example 2-6 displays the log message that interfaces Eth1/1 to Eth1/4 are configured with a latency threshold of 1000 ns.
Example 2-6 SPAN-on-Latency Configuration
N6k-1(config)# monitor session 20 type span-on-latency N6k-1(config-span-on-latency)# source interface ethernet 1/1-2 N6k-1(config-span-on-latency)# destination interface ethernet 1/3 N6k-1(config-span-on-latency)# no shut N6k-1(config-span-on-latency)# exit N6k-1(config)# interface eth1/1-2 N6k-1(config-if-range)# packet latency threshold 1000 Interfaces Eth1/1, Eth1/2, Eth1/3 and Eth1/4 are configured with latency threshold 1000
The SOL-ERSPAN is configured by specifying the type as span-on-latency-erspan in the monitor session command.
The few limitations with the SOL or SOL-ERSPAN are as follows:
Only the Ethernet source is supported. Port-channel is not supported as the source port.
The source cannot be part of any other session.
The direction of SPAN is not allowed with SOL.
ACL filtering is not supported with SOL.
SPAN-on-Drop
SPAN-on-Drop is a new feature that enables the spanning of packets that were dropped because of unavailable buffer or queue space upon ingress. This feature provides the capability to span packets that would otherwise be dropped because the copy of the spanned traffic is transferred to a specific destination port. A SPAN-on-Drop session is configured by specifying the type as span-on-drop in the monitor session configuration. Example 2-7 demonstrates the SPAN-on-Drop monitor session configuration. The source interface Eth1/1 specified in the configuration is the interface where congestion is present.
Example 2-7 SPAN-on-Drop Configuration
N6k-1(config)# monitor session 30 type span-on-drop N6k-1(config-span-on-latency)# source interface ethernet 1/1 N6k-1(config-span-on-latency)# destination interface ethernet 1/3 N6k-1(config-span-on-latency)# no shut N6k-1(config-span-on-latency)# exit
Unlike other SPAN features, SPAN-on-Drop does not have any ternary content addressable memory (TCAM) programming involved. Programming for the source side is in the buffer or queue space. Additionally, only one instance of SPAN-on-Drop can be enabled on the switch; enabling a second instance brings down the session with the syslog message “No hardware resource error.” If the SPAN-on-Drop session is up but no packets are spanned, it is vital to verify that the drop is happening in the unicast flow. This is verified by using the command show platform software qd info interface interface-id and checking that the counter IG_RX_SPAN_ON_DROP is incrementing and is nonzero. Example 2-8 shows the output for the counter IG_RX_SPAN_ON_DROP, confirming that no drops are occurring in the unicast flows.
Example 2-8 Verifying Ingress L3 Unicast Flow Drops
N6k-1# show plat software qd info interface ethernet 1/1 | begin BM-INGRESS BM-INGRESS BM-EGRESS ------------------------------------------------------------------------------- IG_RX 364763|TX 390032 SP_RX 1491|TX_MCAST 0 LB_RX 15689|CRC_BAD 0 IG_RX_SPAN_ON_DROP 0|CRC_STOMP 0 IG_RX_MCAST 14657|DQ_ABORT_MM_XOFF_DROP 0 LB_RX_SPAN 15689|MTU_VIO 0 IG_FRAME_DROP 0| SP_FRAME_DROP 0| LB_FRAME_DROP 0| IG_FRAME_QS_EARLY_DROP 0| ERR_IG_MTU_VIO 0| ERR_SP_MTU_VIO 0| ERR_LB_MTU_VIO 0|
SPAN-on-Drop ERSPAN is an extension of the SPAN-on-Drop feature in which the dropped frames are spanned and sent to a remote IP where the network analyzer is attached.