Cisco Catalyst 6000/6500 Switch Architecture
The Catalyst 6000/6500 is the flagship of the Cisco Catalyst switching family and represents one of the most popular switches used for enterprise networks and service providers. If you are tasked with the procuring a Catalyst 6000/6500 switch, it is important to understand the various Supervisor modules available and the technologies that are used with each to perform both Layer 2 and Layer 3 switching.
The following topics are now discussed:
Supervisor architectures
Catalyst 6000/6500 operating systems
Supervisor Architectures
Several architectural options are available when designing a Catalyst 6000/6500 switch, each of which varies in terms of Layer 3 capabilities. Layer 3 capabilities are added to the Catalyst 6000/6500 switch by two key components:
Policy feature card (PFC)—The PFC provides the necessary ASICs to perform hardware-based Layer 3 switching, quality of service (QoS) classification, and access control list (ACL) filtering. The PFC requires a route processor to populate the route cache or optimized route table structure used by the L3 switching ASIC. If no route processor is present, the PFC can perform only Layer 3/4 QoS classification and ACL filtering and cannot perform L3 switching.
Multilayer switching feature card (MSFC)—The MSFC is essentially a Cisco IOS router based upon the high-performance 7200 series router. This provides the route processor functions required by the PFC to implement L3 switching. The MSFC provides the necessary routing information in the PFC route cache so that the PFC can L3 switch packets.
When you purchase the Catalyst 6000/6500, you have a choice as to the generation of Supervisor that you wish to purchase, as well as the L3 components (i.e., the PFC and/or MSFC) that you require depending on your Layer 3 requirements. These options include the following:
Supervisor 1A
Supervisor 2
Supervisor 720—Next-generation Supervisor engine that includes an integrated PFC-3, MSFC-3, and 720-Gbps crossbar switching matrix.
Each of the various configuration options is now examined.
Supervisor 1A with no PFC
The simplest configuration option available for the Catalyst 6000 is just the Supervisor 1 module with no policy feature card (PFC) or MSFC. In this configuration, the switch is essentially a Layer 2 switch and possesses no Layer 3 switching or classification capabilities. A Supervisor 1A can provide a Layer 2 switch up to 15 million packets per second (Mpps).
Supervisor 1A with PFC-1
The next option available for the Catalyst 6000/6500 is the Supervisor 1 module with a policy feature card (PFC-1) installed. The PFC-1 enables Layer 3 and 4 classification for QoS classification and security ACL filtering; however, L3 switching is not supported unless an MSFC is added to provide route processor functions. The Supervisor 1A with PFC-1 is capable of processing frames through the QoS and ACL engines without degrading Layer 2 switching performance, at speeds of up to 15 Mpps. Figure 6-3 demonstrates the architecture of the Supervisor 1A with PFC-1.
)
Figure
6-3 Supervisor 1A with PFC-1 Architecture
In Figure 6-3, the Supervisor 1A contains the basic Layer 2 engine that references the local bridge table for determining the egress port for switching decisions. The PFC contains a Layer 3 engine, flow cache, ACL engine, and ACL table. In this configuration, the PFC is not used for L3 switching, because no route processor (provided by an MSFC) is installed that provides the required next hop information. However, the PFC can be used for Layer 3/4 QoS classification and ACL filtering; the ACL engine is responsible for providing these functions. The ACL table is stored in ternary content addressable memory (TCAM), which stores ACL information in a format that can be referenced very quickly by the ACL engine. When a packet arrives that requires ACL filtering, while the L2 engine determines the forwarding decision to be made based upon the information contained within the L2 bridge table, at the same time, the ACL engine determines whether or not the packet is permitted or denied. Because the L2 lookup and ACL lookup occur in parallel, applying ACLs or QoS classification to traffic does not affect the forwarding rate of the switch (15 Mpps).
Supervisor 1A with PFC-1 and MSFC-1/MSFC-2
The last Supervisor 1A option and only L3 switching option for the Catalyst 6000/6500 using the Supervisor 1A is the Supervisor 1A module with PFC-1 and MSFC-1 or MSFC-2 installed. The MSFC-1 is now end of sale, so you can only purchase the MSFC-2 if you want to add Layer 3 switching capabilities to existing Supervisor 1A configurations.
NOTE
The MSFC-1 and MSFC-2 differ only in performance. The MSFC-1 has an R5000 200-MHz processor, supports up to 128MB memory, and can route packets at up to 170 Kpps in software. The MSFC-2 has an R7000 300-MHz processor, supports up to 512 MB memory, and can route packets at up to 650 Kpps in software. The Layer 3 switching performance in hardware is still 15 Mpps, regardless of the MSFC used.
In this architecture, the L3 engine onboard the PFC-1 can perform L3 switching, because a route processor is now present in the form of the MSFC. Figure 6-4 shows the architecture of the Supervisor 1A with PFC-1 and MSFC.
)
Figure
6-4 Supervisor 1A with PFC-1 and MSFC
In Figure 6-4, the addition of the MSFC allows for the L3 engine to L3 switch inter-VLAN traffic. All other features of the PFC, such as QoS classification and ACL filtering are also supported. The PFC-1 and MSFC-1/MSFC-2 use multilayer switching (MLS) to perform L3 switching; this means that a flow cache exists on the PFC which is used to L3 switch packet flows through the switch. The first packet within a flow must always be routed by the MSFC, which references the routing table to determine the next hop information for a packet. Once the MSFC has made a routing decision and forwarded the frame back to the L3 engine, the L3 engine reads the routed frame information and writes this information into the flow cache. Subsequent packets received and that match flow cache entries can now be L3 switched by the L3 engine, rather than the MSFC. A limitation of the MLS L3 switching mechanism is the initial route lookup performed in software by the MSFC. The first packet in an IP flow must be passed to the MSFC route processor for routing. In an environment that has many connections being established at the same time, this can cause performance problems for the MSFC. This problem in particular applies to service provider environments, which typically must handle conditions where many short term connections (e.g., downloading a web page might open several HTTP connections that are terminated immediately once the page is downloaded) are being established at once. The Supervisor 1 with PFC-1 and MSFC can L3 switch packets at 15 Mpps.
Supervisor 2 with PFC-2
The first configuration available for the Catalyst 6000/6500 with a Supervisor 2 module is the Supervisor 2 with a policy feature card 2 (PFC-2) installed (the Supervisor 2 is integrated with PFC-2; you can't purchase either separately). The PFC-2 is similar in function to the PFC-1, enabling Layer 3 classification for QoS classification and security ACL filtering; however, it is twice as fast as the PFC-1 and supports more ACLs that can be stored in hardware for QoS and Security. The Supervisor 2 with PFC-2 is capable of switching packets and performing Layer 3/4 QoS classification and ACL filtering at up to 30 Mpps; however, this requires switch fabric enabled modules and a switch fabric module to be installed. Because no MSFC is present in this configuration, L3 switching is not possible. Figure 6-5 demonstrates the architecture of the Supervisor 2 with PFC-2.
Comparing the architecture of the Supervisor 1A with PFC-1, notice that the PFC-2 is actually an integrated part of the Supervisor 2 module. The most notable difference is that the Layer 2 and ACL engine are now combined into a single L2/L4 engine, which boosts the performance capabilities of L2 switching combined with Layer 3/4 QoS classification and ACL filtering up to 30 Mpps. The L3 engine is not used for L3 switching, because an MSFC-2 (route processor) is required to generate information contained in the CEF table.
)
Figure
6-5 Supervisor 2 with PFC-2 Architecture
Supervisor 2 with PFC-2 and MSFC-2
To enable Layer 3 switching on a Supervisor 2 with PFC-2, the only option is to add an MSFC-2 (the MSFC-1 is not supported on the Supervisor 2). In this architecture, the L3 engine onboard the PFC-2 can perform L3 switching, because a route processor is now present in the form of the MSFC-2. Figure 6-6 shows the architecture of the Supervisor 2 with PFC-2 and MSFC-2.
In Figure 6-6, the addition of the MSFC allows for the L3 engine to L3 switch inter-VLAN traffic. All other features of the PFC, such as QoS classification and ACL filtering, are also supported. The PFC-2 and MSFC-2 use CEF to perform L3 switching; the MSFC-2 is responsible for generating the appropriate CEF tables (the FIB table and adjacency table, discussed in Scenario 6-2) upon PFC initialization. This means that as soon as packets need to be L3 switched, the L3 engine has the necessary information to L3 switch the packet, without having to send the first packet associated with a flow to the MSFC (as is the case with MLS). This architecture eliminates the issue that MLS has for supporting an environment that has thousands of connections being established every second. The Supervisor 2 with PFC-2 and MSFC-2 can L3 switch packets at 30 million packets per second.
)
Figure
6-6 Supervisor 2 with PFC-2 and MSFC-2
The Switch Fabric Module
The switch fabric module (SFM) is a module that includes a switching backplane that increases the forwarding rate of the Catalyst 6500 backplane from 32 Gbps to 256 Gbps.
NOTE
The SFM is available only for the Catalyst 6500 with Supervisor 2 and must be installed in Slot 5. A redundant SFM is available and must be installed in Slot 6 if used.
The SFM provides a crossbar switching matrix for the switching backplane, which allows multiple frames to be switched between different line cards at the same time. For example, a frame can be switched across the matrix from line card #2 to line card #4 at exactly the same time as another frame is being switched from line card #3 to line card #8. This is not possible on the traditional shared 32-Gbps backplane of the Catalyst 6000/6500; thus the crossbar matrix can support much higher packet forwarding rates. The SFM provides 16 * 8-Gbps full-duplex connections into the switching matrix. Figure 6-7 shows the SFM and how it provides connections to the other switch modules in the switch chassis.
)
Figure
6-7 The Switch Fabric Module
Each switch module has two available 8-Gbps connections to the SFM. Depending on the type of line cards installed, a line card might take advantage of zero, one, or both of the 8-Gbps connections. Three types of switch modules (line cards) relate to the SFM:
Non fabric-enabled card—These cards are not compatible with the SFM and connect only to the 32-Gbps backplane.
Fabric-enabled card—These cards are compatible with both the SFM and 32-Gbps backplane. A single 8-Gbps connection is provided to the SFM, as well as a single connection to the traditional 32-Gbps backplane.
Fabric-only card—These cards connect solely to the SFM via dual 8-Gbps connections. These cards do not connect directly to the traditional 32-Gbps backplane.
It is important to understand that all of the cards listed above can communicate with each other. The fabric-only card can communicate with non fabric-enabled cards because the SFM has a connection to the traditional Catalyst 6000/6500 32-Gbps backplane.
The Distributed Feature Card (DFC)
The Distributed Feature Card (DFC) allows fabric-enabled line cards to make L3 forwarding decisions locally without requiring the L3 switching engine located on the Supervisor PFC. The DFC consists of the same components as the PFC located on the Supervisor module, however it does not contain the MSFC routing engine. Figure 6-8 shows the DFC architecture.
NOTE
Only fabric-enabled line cards support the DFC. If you are using the DFC, you must install a switch fabric module card.
)
Figure
6-8 The Distributed Feature Card
In Figure 6-8, you can see a fabric-enabled line card that has a DFC installed. The DFC looks exactly like a PFC and performs the same functions as the PFC, except for frames received on local ports. The key to the DFC is the use of distributed CEF (dCEF). A master CEF table resides on the Supervisor 2 PFC-2, which is generated by the MSFC routing table. The master CEF table is downloaded (mirrored) to each DFC, which enables the L3 engine on each DFC to make routing decisions locally. If a route table change occurs, the CEF tables on the PFC and each DFC are updated immediately. If frames are received on a DFC-enabled line card that require routing, the L3 engine on the DFC inspects the destination IP address of the IP packet contained within the frame and looks up the CEF table to determine the next-hop MAC address and egress port. If the egress port is local, the L3 engine rewrites the destination MAC address and forwards the frame out the appropriate local egress port. If the egress port is located on another module, the L3 engine rewrites the destination MAC address and forwards the frame onto the SFM matrix, prepending a tag that identifies the egress port the frame should be switched out of. The tagged frame is forwarded to the appropriate switch module, with the local switching engine forwarding the frame out the appropriate egress port. This forwarding of frames across the SFM matrix does not require any intervention by the main Supervisor 2 PFC L3 engine. Given that a DFC can L3 switch up to 30 Mpps, if a Catalyst 6509 has a single Supervisor 2 with PFC-2 and MSFC-2, a SFM, and seven fabric-enabled line cards each with a DFC installed, the total system capacity theoretically is 210 Mpps (7 * 30 Mpps).
Supervisor 720
A recent new addition to the Catalyst 6500 family is the Supervisor 720 engine, which is the third-generation supervisor engine that integrates the following components into a single module:
PFC-3
MSFC-3
Crossbar Switching Fabric that provides 720 Gbps of backplane bandwidth
The Supervisor 720 significantly increases the number of slots available for data modules. For example, in a non-redundant Catalyst 6509 configuration, the Supervisor 720 takes up only a single slot, leaving eight slots for data modules. In comparison, a Supervisor 2 with SFM installed takes up two slots, leaving only seven slots for data modules. In a redundant configuration, the Supervisor 720 engines take up only two slots while the Supervisor 2 engines and redundant SFMs take up four slots.
NOTE
For the Catalyst 6506, 6509, and 6513, the primary Supervisor 720 must be installed in Slot 5, while the redundant Supervisor 720 must be installed in Slot 6. You must also install a minimum of 2500W power supplies to power the Supervisor 720 on all Catalyst 6500 switches.
The Supervisor 720 also provides a large number of feature enhancements, which include the following:
Hardware-based MPLS forwarding
Hardware-based IPv6 Layer 3 switching
Support for hardware assisted NAT and generic routing encapsulation (GRE)
Backplane bandwidth increases to 2 * 20 Gbps, up from 2 * 8 Gbps with the SFM
Maximum throughput of 400 Mpps, almost twice that of the Supervisor 2 with SFM
Catalyst 6000/6500 Operating Systems
On the Catalyst 6000/6500, it is important to understand that the switch can operate in one of three different modes, depending on the hardware installed and operating systems used to manage the switch. These modes include the following:
CatOS—In this mode, the switch only operates a single operating system: CatOS. No MSFC is installed, because this uses its own operating system.
Hybrid mode—Hybrid mode refers to the configuration where an MSFC is installed that is running Cisco IOS, whilst the switch is running CatOS. This means two separate management interfaces are required—one for the switch and one for the MSFC.
Native mode—In this configuration, a single Cisco IOS operating system is used to manage both the switch and the MSFC. This allows for a single management interface to manage both the switching and routing components of the switch (native mode requires an MSFC).