Scenario 6-2: Configuring CEF-based Layer 3 Switching on the Catalyst 6000/6500 Operating in Hybrid Mode
The next-generation of Cisco Catalyst Layer 3 switches are all based upon Cisco Express Forwarding (CEF). CEF is also the next-generation route caching mechanism for interrupt context switching on Cisco routers so understanding how CEF works and how to configure it is important for network engineers. CEF offers significant improvements over MLS, the most notable being that the first packet in a flow does not need to process-switched by the control plane routing component, as is the case with MLS. With CEF, all packets (even the first) associated with a flow are Layer 3 switched in hardware. This is an important consideration in environments where many new flows are being established continuously (e.g., an Internet service provider environment), because large amounts of new flows in an MLS configuration reduces performance.
In this scenario you learn how to configure Layer 3 switching using CEF on a hybrid mode Catalyst 6000/6500 switch, which refers to a configuration where the Supervisor engine runs CatOS and the MSFC runs a separate Cisco IOS operating system. In later scenarios, you learn how to upgrade from a hybrid mode system to a native mode system and then configure Layer 3 switching using native mode.
Figure 6-13 shows topology used for this scenario.
In Figure 6-13, Switch-A is a Catalyst 6509 switch a Supervisor 2 engine installed, which includes a PFC-2 and MSFC-2. Switch-A is running hybrid mode, with CatOS running on the Supervisor 2 engine and Cisco IOS running on the MSFC-2. The goal of this scenario is simply to enable inter-VLAN routing between Host-X and Host-Y, using CEF-based Layer 3 switching.
)
Figure
6-13 Topology for Scenario 6-2
Introduction to Cisco Express Forwarding
Cisco Express Forwarding (CEF) allows the appropriate information required for the data plane operations of Layer 3 routing (e.g., MAC address rewrites on an Ethernet network and determining the egress port out which a routed frame should be sent) to be stored in a compact data structure optimized for fast lookup.
NOTE
CEF is not only used by Cisco Catalyst Layer 3 switches; it is also used by Cisco routers (in fact, CEF was originally developed on high-end Cisco routers). CEF represents the recommended "route caching" mechanism that should be configured on all Cisco Layer 3 devices if possible.
You have learnt that MLS uses a flow-based caching mechanism, where packets must first be process-switched by the MLS-RP to generate flow entries in the cache. In environments where thousands of new flows are being created per second, this can cause the MLS-RP to become a bottleneck. CEF was developed to eliminate the performance penalty of the first-packet process-switched lookup, allowing the route cache used by the hardware-based L3 routing engine to contain all the necessary information to L3 switch in hardware before any packets associated with a flow are received. To achieve this, CEF creates two data structures in the route cache:
Forwarding Information Base (FIB)—The FIB is generated directly from the route table and contains the next-hop IP address information for each destination (IP route) in the network.
Adjacency table—The adjacency table defines each next-hop IP address in terms of MAC address and egress interface. MAC address information is collected via the ARP cache.
Figure 6-14 illustrates how the route table, FIB, adjacency table and ARP cache work together.
)
Figure
6-14 CEF Components
In Figure 6-14, the route table and ARP cache are both control plane entities, meaning that both are generated and maintained via the control plane processor. From these tables, the FIB and adjacency tables are built, which are data structures optimized for fast lookup by the data plane processor In Figure 6-14, the Layer 3 switching engine uses the FIB and adjacency table to determine the MAC address of the next hop device for a packet, providing the Layer 3 switching engine the required information for MAC address to rewritten in hardware rather than software. Notice how the FIB is built in the route table; the destination address, subnet mask (or prefix), and next hop gateway are extracted from the route table (the remaining information in the route table is not relevant to the Layer 3 switching process). The adjacency table then builds the MAC address details that are used to rewrite the destination MAC address of each Layer 3 switched frame, using the ARP cache (populated via the control plane using ARP requests).
NOTE
The MAC address of the router must also be known to the Layer 3 switching engine, which is used to rewrite the source MAC address of the Layer 3 switched frame. This MAC address is always the same value and hence does not need to be included in the FIB/adjacency tables.
You might notice that all of the information contained in the FIB and adjacency table is the same information contained within the route and ARP tables. The FIB and adjacency exist purely for organizing the specific information required for Layer 3 switching into a structure that is optimized for fast lookups by the data plane.
Distributed and Accelerated CEF
By default, all CEF-based Cisco Catalyst switches use a central Layer 3 switching engine to provide CEF-based Layer 3 switching, where a single processor makes all Layer 3 switching decisions for traffic received on all ports in the switch. Even though the Layer 3 switching engines used in Cisco Catalyst switches provide high performance, in some networks, having a single Layer 3 switching engine to all the Layer 3 switching does not provide sufficient performance. Distributed CEF (dCEF) and Accelerated CEF (aCEF) are technologies that implement multiple Layer 3 switching engines so that simultaneous Layer 3 switching operations can occur in parallel, boosting overall system performance.
dCEF can be used in Cisco Catalyst 6500 switches and refers to the use of multiple CEF tables distributed across multiple line cards installed in the chassis. Each line card that supports dCEF has its own dedicated hardware-based L3 routing engine and CEF route cache, allowing for multiple L3 data plane operations to be performed simultaneously within a single chassis-based system. The main route processor of the switch is responsible for generating a central master FIB and adjacency table and distributing these tables out to each dCEF line card. Figure 6-15 demonstrates the dCEF architecture.
)
Figure
6-15 L3 Switching Using dCEF
aCEF is a new feature supported in conjunction with the new Supervisor 720 engine which works in a similar fashion to MLS. In MLS, line cards send the initial packet of a flow to the Supervisor engine, where the packet is switched in hardware using the master CEF table. The forwarding decision made is then stored in a local scaled-down CEF table on the line card where the flow enters the switch, with the local line card making any subsequent forwarding decisions for packets associated with a flow in the local CEF table.
The Cisco CEF Implementation
The use of CEF-based L3 switching is supported on the following Cisco Catalyst switches:
Catalyst 3550/3750 series with L3 enhanced image
Catalyst 4000/4500 series with Supervisor 3/4
Catalyst 6000/6500 series with Supervisor 2, PFC-2, and MSFC-2
Catalyst 6000/6500 series with Supervisor 720 (includes PFC-3 and MSFC-3)
NOTE
CEF is also supported on all Cisco routers from IOS 12.0 upwards. It is important to understand that the CEF implementation on low-end routers (i.e., Cisco 3700 series and lower) still uses the main processor for data plane operations, whereas L3 switches use a dedicated hardware chip (ASIC) for data plane operations (e.g., PFC-2) instead of the main processor. Using CEF still provides a performance advantage over older route caching technologies, such as fast switching. On higher end routers (e.g., 7500 series, 10000 series, and 12000 series), data plane operations are performed in specialized hardware-based ASICs.
Scenario Prerequisites
To successfully commence the configuration tasks required to complete this scenario, Table 6-5 describes the prerequisite configurations required on each device in the scenario topology. Any configurations not listed can be assumed as being the default configuration.
Table 6-5 Scenario 6-2 Requirements
|
Device |
Required Configuration |
|
|
|
Parameter |
Value |
|
Switch-A |
Hostname |
Switch-A |
|
|
sc0 IP Address (VLAN) |
192.168.1.10/24 (VLAN 1) |
|
|
Enable/Telnet Password |
cisco |
|
|
VTP Mode |
Transparent |
|
|
VLANs (Name) |
VLAN 2 (VLAN02) |
|
|
VLAN Assignments |
VLAN 2: port 2/2 |
|
Router-A |
Hostname |
Router-A |
|
|
Enable/Telnet Password |
cisco |
|
|
IP Address (Interface) |
192.168.1.2/24 (Ethernet0/0) |
|
Host-X |
Operating System |
Windows 2000 Professional or Windows XP |
|
|
IP Address |
192.168.1.100/24 |
|
|
Default Gateway |
192.168.1.1 |
|
Host-Y |
Operating System |
Windows 2000 Professional or Windows XP |
|
|
IP Address |
192.168.2.100/24 |
|
|
Default Gateway |
192.168.2.1 |
Example 6-16 and Example 6-17 shows the configuration required on Switch-A and Router-A before you can begin this scenario.
Example 6-83 Scenario 6-2 Prerequisite Configuration for Switch-A
Console> (enable) set system name Switch-A System name set. Switch-A> (enable) set password Enter old password: ø Enter new password: ***** Reenter new password: ***** Switch-A> (enable) set enablepass Enter old password: ø Enter new password: ***** Retype new password: ***** Switch-A> (enable) set interface sc0 192.168.1.10 255.255.255.0 Interface sc0 IP address and netmask set. Switch-A> (enable) set vtp mode transparent VTP domain modified Switch-A> (enable) set vlan 2 name VLAN2 Vlan 2 configuration successful Switch-A> (enable) set vlan 2 2/2 VLAN 2 modified. VLAN 1 modified. VLAN Mod/Ports ---- ----------------------- 2 2/2
Example 6-84 Scenario 6-2 Prerequisite Configuration for Router-A
Router# configure terminal Router(config)# hostname Router-A Router-A(config)# enable secret cisco Router-A(config)# line vty 0 4 Router-A(config-line)# password cisco Router-A(config-line)# exit Router-A(config)# interface Ethernet0/0 Router-A(config-if)# no shutdown Router-A(config-if)# ip address 192.168.1.2 255.255.255.0
After the prerequisite configuration is implemented, you should attach each device as indicated in Figure 6-13 and verify PING connectivity between devices in the network (where possible) before proceeding.
Configuration Tasks
Configuring CEF-based L3 switching on the Catalyst 6000/6500 operating in hybrid mode is very simple. On the Supervisor 2 engine (running CatOS), all that is required is for a PFC-2 and MSFC-2 daughtercard to be installed. No CatOS software configuration is required; however, you can tune and monitor the PFC-2 from CatOS. All configuration is controlled via the Cisco IOS running on the MSFC-2, where the IP route table is generated from the various routing protocols configured on the MSFC which then automatically populates the CEF table on the PFC-2. In summary, the following is required to configure CEF-based L3 switching:
Configuring the MSFC
Configuring the PFC (optional)
Verifying CEF operation
Configuring the MSFC
To configure the MSFC on a hybrid mode Catalyst 6000/6500, the following tasks are required:
Establishing management connectivity to the MSFC
Configuring the MSFC-2
Establishing Management Connectivity to the MSFC-2
On a hybrid mode Catalyst 6000/6500 switch, configuration of the MSFC requires a management connection to be established to the MSFC operating system (Cisco IOS), which is separate from the CatOS running on the Supervisor engine. On the Catalyst 6000/6500, the MSFC is accessible via module 15, which is an internal designation for the internal interface between the MSFC daughtercard and Supervisor engine. You can access the MSFC management interface using the session 15 command (where 15 represents the slot number of the MSFC), or alternatively, you can also use the switch console command. Example 6-18 demonstrates gaining access to the MSFC from CatOS.
Example 6-85 Gaining Management Access to the MSFC
Switch-A> (enable) switch console Trying Router-15... Connected to Router-15. Type ^C^C^C to switch back... Router>
TIP
The session command can be used only if the Supervisor detects the MSFC. If the MSFC has problems booting (for example, the IOS image is corrupted) and boots to ROMMON mode, the Supervisor does not detect the MSFC, and you cannot use the session command to connect to the MSFC console. In this situation, use the switch console command, because this accesses a console connection that is permanently wired to an internal console port on the MSFC.
Configuring the MSFC-2
After establishing initial management connectivity to the MSFC-2, you can next configure the MSFC-2 for routing. In Example 6-18, the MSFC should have a blank configuration because it has not yet been configured and, therefore, requires not only configuration relevant to Layer 3 switching but also any relevant base configuration. Example 6-19 shows the base configuration required on the MSFC-2.
Example 6-86 Base Configuration for MSFC-2 on Switch-A
Router> enable Router# configure terminal Router(config)# hostname Switch-A-MSFC Switch-A-MSFC(config)# enable secret cisco Switch-A-MSFC(config)# line vty 0 4 Switch-A-MSFC(config-line)# password cisco
In Example 6-19, the MSFC host name, Telnet password, and enable password is configured.
After the base configuration is complete, Layer 3 interfaces need to be created so that IP routing can take place. In this scenario, two Layer 3 switched virtual interfaces (SVIs) are required, which each attach to VLAN 1 and VLAN 2, respectively. Example 6-20 demonstrates configuring the required SVIs on the MSFC.
Example 6-87 Configuring SVIs on the MSFC
Switch-A-MSFC# configure terminal Switch-A-MSFC(config)# interface VLAN 1 Switch-A-MSFC(config-if)# no shutdown Switch-A-MSFC(config-if)# ip address 192.168.1.1 255.255.255.0 Switch-A-MSFC(config-if)# exit Switch-A-MSFC(config)# interface VLAN 2 Switch-A-MSFC(config-if)# ip address 192.168.2.1 255.255.255.0
At this stage, you should be able to ping the sc0 interface (192.168.1.10) on the Supervisor engine of Switch-A because the VLAN 1 interface in Example 6-20 has been configured with an IP address. You should also be able to ping Router-A (192.168.2.2), Host-X, and Host-Y from Switch-A assuming these devices are connected and configured as per the earlier configuration prerequisites section.
Once IP connectivity has been verified, you can then configuring Layer 3 routing on the MSFC. In this scenario, the MSFC is part of an OSPF routing domain and must be configured to exchange routes with Router-A. Example 6-21 demonstrates configuring OSPF on the MSFC.
Example 6-88 Configuring IP Routing on the MSFC
Switch-A-MSFC# configure terminal
Switch-A-MSFC(config)# router ospf 1
Switch-A-MSFC(config-router)# network 192.168.1.0 0.0.0.255 area 0
Switch-A-MSFC(config-router)# network 192.168.2.0 0.0.0.255 area 0
Switch-A-MSFC(config-router)# end
Switch-A-MSFC# show ip ospf neighbors
Neighbor ID Pri State Dead Time Address Interface
192.168.1.2 1 FULL/BDR 00:00:38 192.168.1.2 Vlan1
Switch-A-MSFC# show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is not set
O IA 10.0.0.0/8 [110/2] via 192.168.1.2, 00:00:32, Vlan1
C 192.168.1.0/24 is directly connected, Vlan1
C 192.168.2.0/24 is directly connected, Vlan2
In Example 6-21, both VLAN interfaces are configured as part of OSPF area 0. After the configuration, the show ip ospf neighbors command is executed, which verifies an OSPF adjacency has formed with Router-A. The show ip route command verifies that Switch-A-MSFC has learned the 10.0.0.0/8 network from Router-A.
Configuring the PFC (Optional)
No configuration is required of the PFC using the Supervisor engine to enabled CEF operation. CEF-based Layer 3 switching is solely enabled by configuring the MSFC.
One area where you can configure the Supervisor engine for CEF-based Layer 3 switching is in configuring NetFlow statistics. NetFlow is a protocol that is used in many service provider and enterprise networks which provides statistics to a NetFlow data collector about each flow or connection that passes through a routing device. This information is often for billing purposes in a service provider, making NetFlow a crucial feature of the network. NetFlow information can also be used for traffic analysis, which is common in enterprise networks. Traditionally, routers have been used for NetFlow; however, with the advent of Layer 3 switching, NetFlow has also moved to the switch. On a Layer 3 switch, the flow-based architecture of MLS (discussed in Scenario 6-1) fits well with NetFlow because the information about flow entries stored in cache can be directly exported to NetFlow. In a CEF-based Layer 3 switching architecture, however, the route caching mechanism is not flow-based, instead being based on routing topology and Layer 2 adjacency information rather than flows.
Cisco Catalyst 6000/6500 switches that perform CEF-based Layer 3 switching also collect flow information, purely for the purposes of supporting NetFlow statistics collection. On the Supervisor engine, you configure NetFlow statistics collection much like you configure MLS. The various commands that affect how information is stored in the MLS flow cache are the same commands used to determine how information is stored for NetFlow statistics on a CEF-based switch (all NetFlow information is stored in the NetFlow table). These commands include the following:
set mls agingtime—Used to control how long an entry remains in the NetFlow table.
set mls flow—Used to control the flow mask that defines the granularity of the NetFlow table.
set mls exclude protocol—Used to exclude TCP and/or UDP flows from the NetFlow table.
NOTE
You can use the show mls statistics command to view information about the various flows stored in the Netflow table.
Verifying CEF Operation
At this stage, the configuration of the network is complete. Host-X and Host-Y should be able to communicate with each other and also should be able to ping the loopback 0 interface on Router-A (10.0.0.1). This verifies that inter-VLAN routing is working; however, just because inter-VLAN routing is working doesn't necessarily mean that Layer 3 switching using CEF is working. Packets possibly could be routed in software via the MSFC, which of course in a production environment on a busy network degrades performance. If you are configuring CEF for Layer 3 switching, you must understand how to verify not only inter-VLAN routing, but also how to verify CEF operation.
To verify CEF operation on a hybrid mode Catalyst 6000/6500, you use various show mls commands, which display information relating to CEF operation. The show mls cef mac command can be used to view the MAC address that is used by the MSFC, which allows the PFC on the Supervisor engine to identify a candidate frame for Layer 3 switching. Example 6-22 demonstrates the use of this command.
Example 6-89 Displaying the MSFC MAC Address on CatOS
Switch-A> (enable) show mls cef mac Module 15: Physical MAC-Address 00-04-dd-41-89-0a Module 15 is the designated MSFC for installing CEF entries
In Example 6-22, you can see that Switch-A knows that the MAC address of the MSFC is 00-04-dd-41-89-0a, meaning any frames received with a destination MAC address of this address require Layer 3 switching.
As discussed in the introduction of this chapter, CEF uses two tables to represent control plane routing operation in a format that can be quickly looked up by the L3 engine on the PFC-2. The first table is the Forwarding Information Base (FIB). To view the FIB, you can use the show mls entry cef ip command, as demonstrated on Switch-A in Example 6-23.
Example 6-90 Displaying the FIB on CatOS
Switch-A> (enable) show mls entry cef ip Mod FIB-Type Destination-IP Destination-Mask NextHop-IP Weight --- --------- --------------- ---------------- --------------- ------ 15 receive 255.255.255.255 255.255.255.255 15 receive 127.0.0.12 255.255.255.255 15 receive 127.0.0.0 255.255.255.255 15 receive 127.255.255.255 255.255.255.255 15 resolved 127.0.0.11 255.255.255.255 127.0.0.11 1 15 receive 192.168.2.1 255.255.255.255 15 receive 192.168.2.0 255.255.255.255 15 receive 192.168.2.255 255.255.255.255 15 resolved 192.168.2.100 255.255.255.255 192.168.2.100 1 15 receive 192.168.1.1 255.255.255.255 15 receive 192.168.1.0 255.255.255.255 15 receive 192.168.1.255 255.255.255.255 15 resolved 192.168.1.100 255.255.255.255 192.168.1.100 1 15 receive 224.0.0.0 255.255.255.0 15 connected 192.168.2.0 255.255.255.0 15 connected 192.168.1.0 255.255.255.0 15 resolved 10.0.0.0 255.0.0.0 192.168.1.2 1 15 connected 127.0.0.0 255.0.0.0 15 drop 224.0.0.0 240.0.0.0 15 wildcard 0.0.0.0 0.0.0.0
The FIB table is generated by the MSFC from the MSFC route table and written to the CEF FIB table in the route cache on the PFC-2. In Example 6-23, the Mod column indicates the module number of the MSFC (module 15) that generated each FIB entry. The FIB-Type column describes the type of FIB entry. The following describes each of the FIB types shown in Example 6-23:
resolved—Indicates an entry that represents the route to a host on the local subnet or remote destination subnet derived from the routing table on the MSFC. Every resolved FIB entry includes a next hop IP address.
receive—Indicates that any packet that matches the indicated destination IP address or IP subnet should be sent to the MSFC for processing. For example, these entries are used for any management traffic to a local IP address on the MSFC.
connected—Indicates an entry that represents a locally connected IP subnet.
drop—Indicates that any packet that matches the indicated destination IP address or IP subnet should be dropped. In Example 6-23, because multicast routing is not enabled on the MSFC, all multicast traffic (indicated by a destination IP address in the range of 224.0.0.0 ν 239.255.255.255) is dropped.
wildcard—Indicates the entry that matches any packets that do not match other FIB entries. The actions for this entry is to drop the traffic, as it is deemed unroutable.
The important entries in Example 6-23 are shaded. For example, the last shaded entry specifies a destination prefix of 10.0.0.0/8 and indicates that the next hop for this destination is 192.168.1.2 (Router-A).
The next table related to CEF is the adjacency table, which can be displayed by executing the show mls entry adjacency command, as shown in Example 6-24.
Example 6-91 Displaying the CEF Adjacency Table on CatOS
Switch-A> (enable) show mls entry cef adjacency Mod: 15 Destination-IP: 192.168.2.100 Destination-Mask: 255.255.255.255 FIB-Type: resolved AdjType NextHop-IP NextHop-Mac Vlan Encp Tx-Packets Tx-Octets -------- --------------- ----------------- ---- ---- ------------ ------------- connect 192.168.2.100 00-06-53-fe-84-20 2 ARPA 4 256 ******************************************************************************* Mod: 15 Destination-IP: 127.0.0.11 Destination-Mask: 255.255.255.255 FIB-Type: resolved AdjType NextHop-IP NextHop-Mac Vlan Encp Tx-Packets Tx-Octets -------- --------------- ----------------- ---- ---- ------------ ------------- connect 127.0.0.11 00-00-11-00-00-00 0 ARPA 0 0 ******************************************************************************* Mod: 15 Destination-IP: 192.168.1.100 Destination-Mask: 255.255.255.255 FIB-Type: resolved AdjType NextHop-IP NextHop-Mac Vlan Encp Tx-Packets Tx-Octets -------- --------------- ----------------- ---- ---- ------------ ------------- connect 192.168.1.100 00-10-a4-e0-1e-d3 1 ARPA 4 256 ******************************************************************************* Mod: 15 Destination-IP: 10.0.0.0 Destination-Mask: 255.255.255.0 FIB-Type: resolved AdjType NextHop-IP NextHop-Mac Vlan Encp Tx-Packets Tx-Octets -------- --------------- ----------------- ---- ---- ------------ ------------- connect 192.168.1.2 00-d0-05-15-64-0a 2 ARPA 5 420
In Example 6-24, notice that asterisked lines are included to show each of the separate adjacency entries. The output shows each of the next hop devices listed in the FIB table (see Example 6-23) and indicates the next hop interface and the number of IP packets and bytes transmitted. For example, the last entry in Example 6-24 is for the FIB prefix 10.0.0.0/8. In the adjacency information, you can see that the next hop address is 192.168.1.2 (Router-A). This information is generated from the routing table on the MSFC. The NextHop-Mac address column represents the destination MAC address of the next hop router, which is required as the address used to rewrite the destination MAC address for the relevant prefix. The MAC address information is built from the ARP cache. Finally, the Tx-Packets and Tx-Octets columns indicate how many packets and bytes have been L3 switched that match the adjacency entry.
TIP
A much tidier method of displaying the CEF adjacencies is to use the show mls entry cef long command. This command is not demonstrated here as the output is displayed in a table format that is very wide.