Foundation Topics
Describing VDCs on the Cisco Nexus 7000 Series Switch
The Virtual Device Context (VDC) is used to virtualize the Nexus 7000 switch, meaning presenting the Nexus 7000 switch as multiple logical switches. Within a VDC, you can have a unique set of VLANs and VRFs allowing the virtualization of the control plane. Each VDC will have its own administrative domain allowing the management plane as well to be virtualized. You can assign a line card to a VDC or a set of ports to the VDC.
Without creating any VDC on the Nexus 7000, the control plane runs a single VDC; within this VDC are multiple processes and Layer 2 and Layer 3 services running, as shown in Figure 3-1. This VDC is always active and cannot be deleted.
)
Figure 3-1 Single VDC Operation Mode Structure
Virtualization using VLANS and VRFs within this VDC can still be used. When enabling multiple VDCs, these processes are replicated for each VDC. Within each VDC you can have duplicate VLAN names and VRF names, meaning you can have VRF production in VDC1 and VRF production in VDC2.
When enabling Role-Based Access Control (RBAC) for each VDC, each VDC administrator will interface with the process for his or her own VDC. Figure 3-2 shows what the structure looks like when creating multiple VDCs. You can have fault isolation, separate administration per VDC, separate data traffic, and hardware partitioning.
)
Figure 3-2 Multiple VDC Operation Mode Structure
The following are typical use cases for the Cisco Nexus 7000 VDC, which can be used in the design of the data center:
- Creation of separate production, development, and test environments
- Creation of intranet, extranet, and DMZ
- Creation of separate organizations on the same physical switch
- Creation of separate application environments
- Creation of separate departments for the same organization due to different administration domains
VDC separation is industry certified; it has NSS labs for Payment Card Industry (PCI) compliant environments, Federal Information Processing Standards (FIPS 140-2) certification, and Common Criteria Evaluation and Validation Scheme (CCEVS) 10349 certification with EAL4 conformance.
VDC Deployment Scenarios
There are multiple deployment scenarios using the VDCs, which enable the reduction of the physical footprint in the data center, which in turn saves space, power, and cooling.
Horizontal Consolidation Scenarios
VDCs can be used to consolidate multiple physical devices that share the same function role. In horizontal consolidation, you can consolidate core functions and aggregation functions. For example, using a pair of Nexus 7000 switches you can build two redundant data center cores, which can be useful in facilitating migration scenarios. This can happen by creating two VDCs, as shown in Figure 3-3. VDC1 will accommodate Core A and VDC2 will accommodate Core B. You can allocate ports or line cards to the VDCs, and after the migration you can reallocate the old core ports to the new core.
)
Figure 3-3 Horizontal Consolidations for Core Layer
Aggregation layers can also be consolidated, so rather than building a separate aggregation layer for test and development, you can create multiple VDCs called Aggregation 1 and Aggregation 2. Aggregation 1 and Aggregation 2 are completely isolated with a separate role-based access and separate configuration file. Figure 3-4 shows multiple aggregation layers created on the same physical devices.
)
Figure 3-4 Horizontal Consolidation for Aggregation Layers
Vertical Consolidation Scenarios
When using VDCs in vertical consolidation, you can consolidate core functions and aggregation functions. For example, using a pair of Nexus 7000 switches, you can build a redundant core and aggregation layer. Although the port count is the same, it can be useful because it reduces the number of physical switches. This can happen by creating two VDCs, as shown in Figure 3-5. VDC1 accommodates Core A and Aggregation A; VDC2 accommodates Core B and Aggregation B. You can allocate ports or line cards to the VDCs.
)
Figure 3-5 Vertical Consolidation for Core and Aggregation Layers
Another scenario is consolidation of core aggregation and access while you maintain the same hierarchy. Figure 3-6 shows that scenario; as you can see, we used a pair of Nexus 7000 switches to build a three-tier architecture.
)
Figure 3-6 Vertical Consolidation for Core, Aggregation, and Access Layers
VDCs for Service Insertion
VDCs can be used for service insertion and policy enforcement. By creating two separate VDCs for two logical areas inside and outside, as shown in Figure 3-7, for traffic to cross VDC-A to go to VDC-B it must pass through the firewall. This design can make service insertion more deterministic and secure. Each VDC has its own administrative domain, and traffic must go out VDC-A to reach VDC-B. The only way is through the firewall.
)
Figure 3-7 Using VDCs for Firewall Insertion
Understanding Different Types of VDCs
As explained earlier in the chapter, the NX-OS on the Nexus 7000 supports Virtual Device Contexts (VDC). VDCs simply partition the physical Nexus device into multiple logical devices with a separate control plane, data plane, and management plane. Connecting two VDCs on the same physical switch must be done through external cables; you cannot connect VDCs together from inside the chassis. The following list describes the different types of VDCs that can be created on the Nexus 7000:
Default VDC: When you first log in to the Nexus 7000, you log in to the default VDC. You must be in the default VDC or the Admin VDC (discussed later in the chapter) to do certain tasks, which can be done only in these two VDC types. The default VDC is a full-function VDC; the admin always uses it to manage the physical device and other VDCs created. The default VDC has certain characteristics and tasks that can only be done in the default VDC, which can be summarized in the following:
- VDC creation/deletion/suspend
- Resource allocation—interfaces, memory
- NX-OS upgrade across all VDCs
- EPLD upgrade—as directed by TAC or to enable new features
- Ethanalyzer captures—control plane traffic
- Feature-set installation for Nexus 2000, FabricPath, and FCoE
- Control Plane Policing (CoPP)
- Port channel load balancing
- Hardware IDS checks control
- ACL capture feature enabled
Nondefault VDC: The nondefault VDC is a fully functional VDC that can be created from the default VDC or the Admin VDC. Changes in the nondefault VDC affect only that VDC. An independent process is started for each protocol per VDC created. There is a separate configuration file per VDC, and each VDC has its own RBAC, SNMP, and so on.
ADMIN VDC: The Admin VDC is used for administration functions. You can enable starting from NX-OS 6.1 for SUP2/2E modules, and starting from the NX-OS 6.2(2) for SUP1 module, you can create the admin VDC in different ways:
- During a fresh switch boot, you will be prompted to select an Admin VDC.
- Enter the system admin-vdc command after boot. In that case, the default VDC becomes the Admin VDC, and all nonglobal configuration in the default VDC will be lost.
Enter the system admin-vdc migrate new vdc name command. All nonglobal configuration on the default VDC will be migrated to the new VDC.
The Admin VDC has certain characteristics that are unique to the Admin VDC.
- Features and feature sets cannot be enabled in an Admin VDC.
- You cannot assign any interface to the Admin VDC; only mgmt0 can be assigned to the Admin VDC.
- When you enable the Admin VDC, it replaces the default VDC. You must be careful or you will lose the nonglobal configuration.
- After the Admin VDC is created, it cannot be deleted or changed back to the default VDC. To change back to the default VDC you must erase the configuration and do a fresh boot script.
Storage VDC: The storage VDC is a nondefault VDC that helps maintain the operation model when the customer has a LAN admin and a SAN admin. The storage VDC relies on the FCoE license and doesn’t need a VDC license. The storage VDC counts toward the total number of VDCs. We can create only one storage VDC on the physical device. After the creation of the storage VDC, we assign interfaces to it. There are two types of interfaces, dedicated FCoE interfaces or shared interfaces, which can carry both Ethernet and FCoE traffic. Figure 3-8 shows the two types of ports.
Figure 3-8 Storage VDC Interface Types
)
With Supervisor 1 you can have up to four VDCs and one Admin VDC, which requires 8 GB of RAM because you must upgrade to NX-OS 6.2. Supervisor engine 2 can have up to four VDCs and one Admin VDC. With Supervisor engine 2E you can have up to eight VDCs and one Admin VDC. Beyond four VDCs on Supervisor 2E we require a license to increment the VDCs with an additional four.
Interface Allocation
The only thing you can assign to a VDC are the physical ports. At the beginning and before creating a VDC, all interfaces belong to the default VDC. After you create a VDC, you start assigning interfaces to that VDC. A physical interface can belong to only one VDC at a given time. When you create a shared interface, the physical interface belongs to one Ethernet VDC and one storage VDC at the same time. When you allocate the physical interface to a VDC, all configurations that existed on it get erased. The physical interface gets configured within the VDC. Logical interfaces, such as tunnel interfaces and switch virtual interfaces, are created within the VDC, and within a VDC, physical and logical interfaces can be assigned to VLANs or VRFs.
)
Figure 3-9 VDC Interface Allocation Example
VDC Administration
The operations allowed for a user in VDC depend on the role assigned to that user. Users can have multiple roles assigned to them, and the NX-OS provides four default user roles:
- Network-admin: The first user account that is created on a Cisco Nexus 7000 Series switch in the default VDC is admin. The network-admin role is automatically assigned to this user. The network-admin role gives a user complete read and write access to the device, and it is available only in the default VDC. This role includes the ability to create, delete, or change nondefault VDCs.
- Network-operator: The second default role that exists on Cisco Nexus 7000 Series switches is the network-operator role. This role grants the user read-only rights in the default VDC. The network-operator role includes the right to issue the switchto command, which can be used to access a nondefault VDC from the default VDC. By default, no users are assigned to this role. The role must be assigned specifically to a user by a user who has network-admin rights.
- VDC-admin: When a new VDC is created, the first user account on that VDC is admin. The VDC-admin role is automatically assigned to this admin user on a nondefault VDC. This role gives a user complete control over the specific nondefault VDC. However, this user does not have any rights in any other VDC and cannot access other VDCs through the switchto command.
- VDC-operator: The VDC-operator role has read-only rights for a specific VDC. This role has no rights to any other VDC.
When a network-admin or network-operator user accesses a nondefault VDC by using the switchto command, that user is mapped to a role of the same level in the nondefault VDC. That means a user with the network-admin role is given the VDC-admin role in the nondefault VDC. A user with the network-operator role is given the VDC-operator role in the nondefault VDC.
VDC Requirements
To create VDCs, an advanced license is required. The license is associated with the serial number of a Cisco Nexus 7000 switch chassis. The storage VDC is enabled using the storage license associated with a specific line card. You can try the feature for the 120-day grace period. When the grace period expires, any nondefault VDCs will be removed from the switch configuration.
As stated before, VDCs are created or deleted from the default VDC only. You need network admin privileges to do that. Physical interfaces also are assigned to nondefault VDCs from the default VDC.
Verifying VDCs on the Cisco Nexus 7000 Series Switch
The next few lines show certain commands to display useful information about the VDC. As shown in Example 3-1, the show feature-set command is used to verify and show which feature has been enabled or disabled in a VDC.
Example 3-1 Displaying the Status of a Feature Set on a Switch
N7K-VDC-A# show feature-set Feature Set Name ID State -------------------- -------- -------- fabricpath 2 enabled fex 3 disabled N7K-VDC-A#
The show feature-set command in Example 3-1 shows that the FabricPath feature is enabled and the FEX feature is disabled.
The show vdc command displays different outputs depending on where you are executing it from. If you are executing the command from the default VDC, this command displays information about all VDCs on the physical device.
Example 3-2 Displaying Information About All VDCs (Running the Command from the Default VDC)
N7K-Core# show vdc vdc_id vdc_name state mac ------ -------- ----- ---------- 1 prod active 00:18:ba:d8:3f:fd 2 dev active 00:18:ba:d8:3f:fe 3 MyVDC active 00:18:ba:d8:3f:ff
If you are executing the show vdc command within the nondefault VDC, which in our case is the N7K-Core-Prod, the output displays information about the current VDC.
Example 3-3 Displaying Information About the Current VDC (Running the Command from the Nondefault VDC)
N7K-Core-Prod# show vdc vdc_id vdc_name state mac ------ -------- ----- ---------- 1 prod active 00:18:ba:d8:3f:fd
To display the detailed information about all VDCs, execute show vdc detail from the default VDC, as shown in Example 3-4.
Example 3-4 Displaying Detailed Information About All VDCs
N7K-Core# show vdc detail vdc id: 1 vdc name: N7K-Core vdc state: active vdc mac address: 00:22:55:79:a4:c1 vdc ha policy: RELOAD vdc dual-sup ha policy: SWITCHOVER vdc boot Order: 1 vdc create time: Thu May 14 08:14:39 2012 vdc restart count: 0 vdc id: 2 vdc name: prod vdc state: active vdc mac address: 00:22:55:79:a4:c2 vdc ha policy: RESTART vdc dual-sup ha policy: SWITCHOVER vdc boot Order: 1 vdc create time: Thu May 14 08:15:22 2012 vdc restart count: 0 vdc id: 3 vdc name: dev vdc state: active vdc mac address: 00:22:55:79:a4:c3 vdc ha policy: RESTART vdc dual-sup ha policy: SWITCHOVER vdc boot Order: 1 vdc create time: Thu May 14 08:15:29 2012 vdc restart count: 0
You can display the detailed information about the current VDC by executing show vdc {vdc name} detail from the nondefault VDC, as shown in Example 3-5.
Example 3-5 Displaying Detailed Information About the Current VDC When You Are in the Nondefault VDC
N7K-Core-prod# show vdc prod detail vdc id: 2 vdc name: prod vdc state: active vdc mac address: 00:22:55:79:a4:c2 vdc ha policy: RESTART vdc dual-sup ha policy: SWITCHOVER vdc boot Order: 1 vdc create time: Thu May 14 08:15:22 2012 vdc restart count: 0
To verify the interface allocation per VDC, execute show vdc membership from the default VDC, as shown in Example 3-6.
Example 3-6 Displaying Interface Allocation per VDC
N7K-Core# show vdc membership
vdc_id: 1 vdc_name: N7K-Core interfaces:
Ethernet2/1 Ethernet2/2 Ethernet2/3
Ethernet2/4 Ethernet2/5 Ethernet2/6
Ethernet2/7 Ethernet2/8 Ethernet2/9
Ethernet2/10 Ethernet2/11 Ethernet2/12
Ethernet2/13 Ethernet2/14 Ethernet2/15
Ethernet2/16 Ethernet2/17 Ethernet2/18
Ethernet2/19 Ethernet2/20 Ethernet2/21
Ethernet2/22 Ethernet2/23 Ethernet2/24
Ethernet2/25 Ethernet2/26 Ethernet2/27
Ethernet2/28 Ethernet2/29 Ethernet2/30
Ethernet2/31 Ethernet2/32 Ethernet2/33
Ethernet2/34 Ethernet2/35 Ethernet2/36
Ethernet2/37 Ethernet2/38 Ethernet2/39
Ethernet2/40 Ethernet2/41 Ethernet2/42
Ethernet2/43 Ethernet2/44 Ethernet2/45
Ethernet2/48
vdc_id: 2 vdc_name: prod interfaces:
Ethernet2/47
vdc_id: 3 vdc_name: dev interfaces:
Ethernet2/46
If you execute show VDC membership from the VDC you are currently in, it will show the interface membership only for that VDC.
You can save all the running configurations to the startup configuration for all VDCs using one command. Use the copy running-config startup-config vdc-all command, which is executed from the default VDC.
Example 3-7 Saving All Configurations for All VDCs
N7K-Core# copy running-config startup-config vdc-all [########################################] 100%
To display the running configurations for all VDCs, use the show running-config vdc-all command, executed from the default VDC. You cannot see the configuration for other VDCs except from the default VDC.
You can navigate from the default VDC to the nondefault VDC using the switchto command, as shown in Example 3-8.
Example 3-8 Navigating Between VDCs on the Nexus 7000 Switch
N7K-Core# switchto vdc Prod TAC support: http://www.cisco.com/tac Copyright (c) 2002-2008, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained in this software are owned by other third parties and used and distributed under license. Certain components of this software are licensed under the GNU General Public License (GPL) version 2.0 or the GNU Lesser General Public License (LGPL) Version 2.1. A copy of each such license is available at http://www.opensource.org/licenses/gpl-2.0.php and http://www.opensource.org/licenses/lgpl-2.1.php N7K-Core-Prod#
To switch back to the default VDC, use the switchback command. You cannot execute the switchto command from the nondefault VDC; you must do switchback to the default VDC first before going to another VDC. Use these commands for the initial setup; after that, when you create the user accounts and configure the IP connectivity, you use Secure Shell (SSH) and Telnet to connect to the desired VDC.
Describing Network Interface Virtualization
You can deliver an extensible and scalable fabric solution using the Cisco FEX technology. The FEX technology can be extended all the way to the server hypervisor. This technology enables operational simplicity by having a single point of management and policy enforcement on the access parent switch.
Cisco Nexus 2000 FEX Terminology
The following terminology and acronyms are used with the Nexus 2000 fabric extenders:
- Network Interface (NIF): Port on FEX, connecting to parent switch.
- Host Interface (HIF): Front panel port on FEX, connecting to server.
- Virtual Interface (VIF): Logical construct consisting of a front panel port, VLAN, and several other parameters.
- Logical interface (LIF): Presentation of a front panel port in parent switch.
- Fabric links (Fabric Port Channel or FPC): Links that connect FEX NIF and FEX fabric ports on parent switch.
- Parent switch: Switch (N7K/N5K) where FEX is connected.
- FEX A/A or dual-attached FEX: Scenario in which the fabric links of FEX are connected, actively forwarding traffic to two parent switches.
- Fabric Extender (FEX): NEXUS 2000 is the instantiation of FEX.
- Fabric Port: N7K side of the link connected to FEX.
- Fabric Port Channel: Port channel between N7K and FEX.
- FEX uplink: Network-facing port on the FEX side of the link connected to N7K. The FEX uplink is also called a Network Interface (NIF).
- FEX port: Server-facing port on FEX, also referred to as server port or host port in this presentation. FEX port is also called a Host Interface (HIF).
Figure 3-10 shows the ports with reference to the physical hardware.
)
Figure 3-10 Nexus 2000 Interface Types and Acronyms
Nexus 2000 Series Fabric Extender Connectivity
The Cisco Nexus 2000 Series cannot run as a standalone switch; it needs a parent switch. This switch can be a Nexus 9000, Nexus 7000, Nexus 6000, Nexus 5600, or Nexus 5000 Series. This type of design combines the benefit of the Top-of-Rack (ToR) design with the benefit of the End-of-Row (EoR) design.
Based on the type and the density of the access ports required, dual redundant Nexus 2000 fabric extenders are placed at the top of the rack, The uplink ports from the Nexus 2000 will be connected to the parent switch, as stated before, which will be installed at the EoR. From a cabling point of view, this design is a ToR design. The cabling between the servers and the Nexus 2000 Series fabric extenders is contained within the rack. Only a small number of cables will run between the racks, which will be 10/40 Gbps.
From the logical network deployment point of view, this design is an EoR design. The FEX acts as a remote line card. The server appears as if it is connected directly to the parent switch. From an operation perspective, this design has the advantage of the EoR design; all configuration and maintenance tasks are done from the parent switch, and no operation tasks are required to be performed from the FEXs. Figure 3-10 shows the physical and the logical architecture of the FEX implementation. Refer to Chapter 2, “Cisco Nexus Product Family” in the section “Cisco Nexus 2000 Fabric Extenders Product Family” to see the different models and specifications.
Figure 3-11 shows the physical and logical view for the Nexus 2000 fabric extender deployment.
)
Figure 3-11 Nexus 2000 Deployment Physical and Logical View
VN-Tag Overview
The Cisco Nexus 2000 Series fabric extender acts as a remote line card to a parent switch, as stated earlier. All control and management functions are performed by the parent switch. Forwarding is performed by the parent switch. The physical ports on the Nexus 2000 fabric extender appear as logical ports on the parent switch, and the hosts connected to the Nexus 2000 fabric extender appear as if they are directly connected to the parent switch.
A frame exchanged between the Nexus 2000 fabric extender and the parent switch will have an information tag inserted in it called VN-Tag, which enables advanced functions and policies to be applied to it. The host connected to the Nexus 2000 fabric extender is unaware of any tags.
At the time of writing this book there is no local switching happening on the Nexus 2000 fabric extender, so all traffic must be sent to the parent switch.
VN-Tag is a Network Interface Virtualization (NIV) technology. At the beginning, VN-Tag was being standardized under the IEEE 802.1Qbh working group. However, 802.1Qbh was withdrawn and is no longer an approved IEEE project. The effort was moved to the IEEE 802.1BR in July 2011. Figure 3-12 shows the VN-Tag fields in an Ethernet frame.
)
Figure 3-12 VN-Tag Fields in an Ethernet Frame
- d: Direction bit (0 is host-to-network forwarding;1 is network-to-host)
- p: Pointer bit (set if this is a multicast frame and requires egress replication)
- L: Looped filter (set if sending back to source Cisco Nexus 2000 Series)
- VIF: Virtual interface
Cisco Nexus 2000 FEX Packet Flow
The packet processing on the Nexus 2000 happens in two parts: when the host send a packet to the network and when the network sends a packet to the host. Figure 3-13 shows the two scenarios.
)
Figure 3-13 Cisco Nexus 2000 Packet Flow
When the host sends a packet to the network (diagrams A and B), these events occur:
- The frame arrives from the host.
The Cisco Nexus 2000 Series switch adds a VN-Tag, and the packet is forwarded over a fabric link using a specific VN-Tag. The Cisco Nexus 2000 Series switch adds a unique VN-Tag for each Cisco Nexus 2000 Series host interface. These are the VN-Tag field values:
- The direction bit is set to 0, indicating host-to network forwarding.
- The source virtual interface is set based on the ingress host interface.
- The p (pointer), l (looped), and destination virtual interface are undefined (0).
The packet is received over the fabric link using a specific VN-Tag. The Cisco Nexus switch extracts the VN-Tag, which identifies the logical interface that corresponds to the physical host interface on the Cisco Nexus 2000 Series. The Cisco Nexus switch applies an ingress policy that is based on the physical Cisco Nexus Series switch port and logical interface:
- Access control and forwarding are based on frame fields and virtual (logical) interface policy.
- Physical link-level properties are based on the Cisco Nexus Series switch port.
- The Cisco Nexus switch strips the VN-Tag and sends the packet to the network.
When the network sends a packet to the host (diagram C), these events occur:
The frame is received on the physical or logical interface. The Cisco Nexus switch performs standard lookup and policy processing when the egress port is determined to be a logical interface (Cisco Nexus 2000 Series) port. The Cisco Nexus switch inserts a VN-Tag with these characteristics:
- The direction is set to 1 (network to host).
- The destination virtual interface is set to be the Cisco Nexus 2000 Series port VN-Tag.
- The source virtual interface is set if the packet was sourced from a Cisco Nexus 2000 Series port.
- The l (looped) bit filter is set if sending back to a source Cisco Nexus 2000 Series switch.
- The p bit is set if this frame is a multicast frame and requires egress replication.
- The packet is forwarded over a fabric link using a specific VN-Tag.
- The Cisco Nexus 2000 Series switch strips the VN-Tag, and the frame is forwarded to the host interface.
Cisco Nexus 2000 FEX Port Connectivity
You can connect the Nexus 2000 Series fabric extender to one of the following Ethernet modules that are installed in the Cisco Nexus 7000 Series. There is a best practice for connecting the Nexus 2000 fabric extender to the Nexus 7000, depending on the type and model of card installed in it.
- 12-port 40-Gigabit Ethernet F3-Series QSFP I/O module (N7K-F312FQ-25) for Cisco Nexus 7000 Series switches
- 24-port Cisco Nexus 7700 F3 Series 40-Gigabit Ethernet QSFP I/O module (N77-F324FQ-25)
- 48-port Cisco Nexus 7700 F3 Series 1/10-Gigabit Ethernet SFP+ I/O module (N77-F348XP-23)
- 32-port 10-Gigabit Ethernet SFP+ I/O module (N7K-M132XP-12)
- 32-port 10-Gigabit Ethernet SFP+ I/O module (N7K-M132XP-12L)
- 24-port 10-Gigabit Ethernet I/O M2 Series module XL (N7K-M224XP-23L)
- 48-port 1/10-Gigabit Ethernet SFP+ I/O F2 Series module (N7K-F248XP-25)
- Enhanced 48-port 1/10-Gigabit Ethernet SFP+ I/O module (F2e Series) (N7K-F248XP-25E)
- 48-port 1/10-Gigabit Ethernet SFP+ I/O module (F2e Series) (N77-F248XP-23E)
Figure 3-14 shows the connectivity between the Cisco Nexus 2000 fabric extender and the N7K-M132XP-12 line card.
)
Figure 3-14 Cisco Nexus 2000 Connectivity to N7K-M132XP-12
There are two ways to connect the Nexus 2000 to the parent switch: either use static pinning, which makes the FEX use individual links connected to the parent switch, or use a port channel when connecting the fabric interfaces on the FEX to the parent switch. This is explained in more detail in the next paragraphs.
In static pinning, when the FEX is powered up and connected to the parent switch, its host interfaces are distributed equally among the available fabric interfaces. As a result, the bandwidth that is dedicated to each end host toward the parent switch is never changed by the switch, but instead is always specified by you.
The drawback here is that if the fabric link goes down, all the associated host interfaces go down and will remain down as long as the fabric port is down. You must use the pinning max-links command to create several pinned fabric interface connections so that the parent switch can determine a distribution of host interfaces. The host interfaces are divided by the number of the max-links and distributed accordingly. The default value is max-links 1.
To provide load balancing between the host interfaces and the parent switch, you can configure the FEX to use a port channel fabric interface connection. This connection bundles 10-Gigabit Ethernet fabric interfaces into a single logical channel. A fabric interface that fails in the port channel does not trigger a change to the host interfaces. Traffic is automatically redistributed across the remaining links in the port channel fabric interface. If all links in the fabric port channel go down, all host interfaces on the FEX are set to the down state.
When you have VDCs created on the Nexus 7000, you must follow certain rules. All FEX fabric links belong to the same VDC; all FEX host ports belong to the same VDC; FEX IDs are unique across VDCs (the same FEX ID cannot be used across different VDCs). Figure 3-15 shows the FEX connectivity to the Nexus 7000 when you have multiple VDCs.
)
Figure 3-15 Cisco Nexus 2000 and VDCs
Cisco Nexus 2000 FEX Configuration on the Nexus 7000 Series
How you configure the Cisco Nexus 2000 Series on the Cisco 7000 Series is different from the configuration on Cisco Nexus 5000, 6000, and 5600 Series switches. The difference comes from the VDC-based architecture of the Cisco Nexus 7000 Series switches. Use the following steps to connect the Cisco Nexus 2000 to the Cisco 7000 Series parent switch when connected to one of the supported line cards.
Example 3-9 Installing the Fabric Extender Feature Set in Default VDC
N7K-Agg(config)# N7K-Agg(config)# install feature-set fex
Example 3-10 Enabling the Fabric Extender Feature Set
N7K-Agg(config)# N7K-Agg(config)# feature-set fex
By default, when you install the fabric extender feature set, it is allowed in all VDCs. You can disallow the installed fabric extender feature set in a specific VDC on the device.
Example 3-11 Entering the Chassis Mode for the Fabric Extender
N7K-Agg(config)# fex 101 N7K-Agg(config-fex)# description Rack8-N2k N7K-Agg(config-fex)# type N2232P
Example 3-12 Defining the Number of Uplinks
N7K-Agg(config-fex)# pinning max-links 1
You can use this command if the fabric extender is connected to its parent switch using one or more statically pinned fabric interfaces. There can be only one port channel connection.
Example 3-13 Disallowing Enabling the Fabric Extender Feature Set
N7K-Agg# configure terminal N7K-Agg (config)# vdc 1 N7K-Agg (config-vdc)# no allow feature-set fex N7K-Agg (config-vdc)# end N7K-Agg #
The next step is to associate the fabric extender to a port channel interface on the parent device. In Example 3-14, we create a port channel with four member ports.
Example 3-14 Associate the Fabric Extender to a Port Channel Interface
N7K-Agg# configure terminal N7K-Agg (config)# interface ethernet 1/28 N7K-Agg (config-if)# channel-group 4 N7K-Agg (config-if)# no shutdown N7K-Agg (config-if)# exit N7K-Agg (config)# interface ethernet 1/29 N7K-Agg (config-if)# channel-group 4 N7K-Agg (config-if)# no shutdown N7K-Agg (config-if)# exit N7K-Agg (config)# interface ethernet 1/30 N7K-Agg (config-if)# channel-group 4 N7K-Agg (config-if)# no shutdown N7K-Agg (config-if)# exit N7K-Agg (config)# interface ethernet 1/31 N7K-Agg (config-if)# channel-group 4 N7K-Agg (config-if)# no shutdown N7K-Agg (config-if)# exit N7K-Agg (config)# interface port-channel 4 N7K-Agg (config-if)# switchport N7K-Agg (config-if)# switchport mode fex-fabric N7K-Agg (config-if)# fex associate 101
After the FEX is associated successfully, you must do some verification to make sure everything is configured properly.
Example 3-15 Verifying FEX Association
N7K-Agg# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------ 101 FEX0101 Online N2K-C2248TP-1GE JAF1418AARL
Example 3-16 Display Detailed Information About a Specific FEX
N7K-Agg# show fex 101 detail
FEX: 101 Description: FEX0101 state: Online
FEX version: 5.1(1) [Switch version: 5.1(1)]
FEX Interim version: 5.1(0.159.6)
Switch Interim version: 5.1(1)
Extender Model: N2K-C2248TP-1GE, Extender Serial: JAF1418AARL
Part No: 73-12748-05
Card Id: 99, Mac Addr: 54:75:d0:a9:49:42, Num Macs: 64
Module Sw Gen: 21 [Switch Sw Gen: 21]
pinning-mode: static Max-links: 1
Fabric port for control traffic: Po101
Fabric interface state:
Po101 - Interface Up. State: Active
Eth2/1 - Interface Up. State: Active
Eth2/2 - Interface Up. State: Active
Eth4/1 - Interface Up. State: Active
Eth4/2 - Interface Up. State: Active
Fex Port State Fabric Port Primary Fabric
Eth101/1/1 Up Po101 Po101
Eth101/1/2 Up Po101 Po101
Eth101/1/3 Down Po101 Po101
Eth101/1/4 Down Po101 Po101
Example 3-17 Display Which FEX Interfaces Are Pinned to Which Fabric Interfaces
N7K-Agg# show interface port-channel 101 fex-intf Fabric FEX Interface Interfaces --------------------------------------------------- Po101 Eth101/1/2 Eth101/1/1
Example 3-18 Display the Host Interfaces That Are Pinned to a Port Channel Fabric Interface
N7K-Agg# show interface port-channel 4 fex-intf
Fabric FEX
Interface Interfaces
---------------------------------------------------
Po4 Eth101/1/48 Eth101/1/47 Eth101/1/46 Eth101/1/45
Eth101/1/44 Eth101/1/43 Eth101/1/42 Eth101/1/41
Eth101/1/40 Eth101/1/39 Eth101/1/38 Eth101/1/37
Eth101/1/36 Eth101/1/35 Eth101/1/34 Eth101/1/33
Eth101/1/32 Eth101/1/31 Eth101/1/30 Eth101/1/29
Eth101/1/28 Eth101/1/27 Eth101/1/26 Eth101/1/25
Eth101/1/24 Eth101/1/23 Eth101/1/22 Eth101/1/21
Eth101/1/20 Eth101/1/19 Eth101/1/18 Eth101/1/17
Eth101/1/16 Eth101/1/15 Eth101/1/14 Eth101/1/13
Eth101/1/12 Eth101/1/11 Eth101/1/10 Eth101/1/9
Eth101/1/8 Eth101/1/7 Eth101/1/6 Eth101/1/5
Eth101/1/4 Eth101/1/3 Eth101/1/2 Eth101/1/1
Summary
- VDCs on the Nexus 7000 switches enable true device virtualization; each VDC will have its own management domain, allowing the management plane also to be virtualized.
- To send traffic between VDCs you must use external physical cables that lead to greater security of user data.
- Cisco Nexus 2000 cannot operate in a standalone mode; it needs a parent switch to connect to.
- The FEX technology can be extended all the way to the server hypervisor. This technology enables operational simplicity by having a single point of management and policy enforcement on the access parent switch