Implementing EtherChannel in a Switched Network
In networks where resources may be located far from where users might need them, some links between switches or between switches and servers become heavily solicited. The speed of these links can be increased, but only to a certain point. EtherChannel is a technology that allows you to circumvent the bandwidth issue by creating logical links that are made up of several physical links.
This section examines the benefits of EtherChannel and the various technologies available to implement it and also the types of EtherChannel protocol. In addition, it explains how to configure Layer 2 EtherChannels and how to load balance traffic between physical links inside a given EtherChannel bundle. EtherChannels can also operate in a Layer 3 mode, but this is discussed later in Chapter 5. The following topics are discussed in detail in the following subsections:
- The need for EtherChannel technology
- Port aggregation negotiation protocols
- Configuration steps for bundling interfaces into a Layer 2 EtherChannel
- Configuring EtherChannel
- Changing EtherChannel load-balancing behavior
- How EtherChannel load-balancing works
- The role of EtherChannel Guard
The Need for EtherChannel
Any-to-any communications of intranet applications, such as video to the desktop, interactive messaging, Voice over IP (VoIP), and collaborative whiteboard use, are increasing the need for scalable bandwidth within the core and at the edge of campus networks. At the same time, mission-critical applications call for resilient network designs. With the wide deployment of faster switched Ethernet links in the campus, users need to either aggregate their existing resources or upgrade the speed in their uplinks and core to scale performance across the network backbone.
In Figure 3-23, traffic coming from several VLANs at 100 Mbps aggregate on the access switches at the bottom and need to be sent to distribution switches in the middle. Obviously, bandwidth larger than 100 Mbps must be available on the link between two switches to accommodate the traffic load coming from all the VLANs. A first solution is to use a faster port speed, such as 1 or 10 Gbps. As the speed increases on the VLANs links, this solution finds its limitation where the fastest possible port is no longer fast enough to aggregate the traffic coming from all VLANs. A second solution is to multiply the numbers of physical links between both switches to increase the overall speed of the switch-to-switch communication. A downside of this method is that there must be a strict consistency in each physical link configuration. A second issue is that spanning tree may block one of the links, as shown in Figure 3-23.
Figure 3-23 Network Without EtherChannel
EtherChannel is a technology that was originally developed by Cisco as a LAN switch-to-switch technique of grouping several Fast or Gigabit Ethernet ports into one logical channel. This technology has many benefits:
- It relies on the existing switch ports. There is no need to upgrade the switch-to-switch link to a faster and more expensive connection.
- Most of the configuration tasks can be done on the EtherChannel interface instead of on each individual port, thus ensuring configuration consistency throughout the switch-to-switch links.
- Load balancing is possible between the links that are part of the same EtherChannel. Depending on the hardware platform, you can implement one or several methods, such as source-MAC to destination-MAC or source-IP to destination-IP load balancing across the physical links.
Keep in mind that the logic of EtherChannel is to increase the speed between switches, as illustrated in Figure 3-24. This concept was extended as the EtherChannel technology became more popular, and some hardware nonswitch devices support link aggregation into an EtherChannel link. In any case, EtherChannel creates a one-to-one relationship. You can create an EtherChannel link between two switches or between an EtherChannel-enabled server and a switch, but you cannot send traffic to two different switches through the same EtherChannel link. One EtherChannel link always connects the same two devices only. The individual EtherChannel group member port configuration must be consistent on both devices. EtherChannel technology only bundles ports of the same type. On a Layer 2 switch, EtherChannel is used to aggregate access ports or trunks. For example, if the physical ports of one side are configured as trunks, the physical ports of the other side must also be configured as trunks. Each EtherChannel has a logical port channel interface. A configuration that is applied to the port channel interface affects all physical interfaces that are assigned to that interface. (Such commands can be STP commands or commands to configure a Layer 2 EtherChannel as a trunk or an access port.)
Figure 3-24 Network with EtherChannel
Keep in mind that EtherChannel creates an aggregation that is seen as one logical link. When several EtherChannel bundles exist between two switches, spanning tree may block one of the bundles to prevent redundant links. When spanning tree blocks one of the redundant links, it blocks one EtherChannel, thus blocking all the ports belonging to this EtherChannel link. Where there is only one EtherChannel link, all physical links in the EtherChannel are active because spanning tree sees only one (logical) link. If one link in EtherChannel goes down, the bandwidth of the EtherChannel will be automatically updated, and thus the STP cost will change as well.
EtherChannel Mode Interactions
EtherChannel can be established using one of the following three mechanisms, as shown in Figure 3-25:
- LACP: IEEE’s negotiation protocol
- PAgP: Cisco’s negotiation protocol
Static persistence: No negotiation protocol
Figure 3-25 EtherChannel Modes Interactions
)
LACP
Link Aggregation Control Protocol (LACP) is part of an IEEE specification (802.3ad) that allows several physical ports to be bundled together to form a single logical channel. LACP allows a switch to negotiate an automatic bundle by sending LACP packets to the peer. Because LACP is an IEEE standard, you can use it to facilitate EtherChannels in mixed-switch environments. LACP checks for configuration consistency and manages link additions and failures between two switches. It ensures that when EtherChannel is created, all ports have the same type of configuration speed, duplex setting, and VLAN information. Any port modification after the creation of the channel will also change all the other channel ports.
LACP packets are exchanged between switches over EtherChannel-capable ports. Port capabilities are learned and compared with local switch capabilities. LACP assigns roles to EtherChannel’s ports. The switch with the lowest system priority is allowed to make decisions about what ports actively participate in EtherChannel. Ports become active according to their port priority. A lower number means higher priority. Commonly up to 16 links can be assigned to an EtherChannel, but only 8 can be active at a time. Nonactive links are placed into a standby state and are enabled if one of the active links goes down.
The maximum number of active links in an EtherChannel varies between switches.
These are the LACP modes of operation:
- Active: Enable LACP
- Passive: Enable LACP only if an LACP device is detected
The following are some additional parameters that you can use when configuring LACP:
- System priority: Each switch running LACP must have a system priority. The system priority can be specified automatically or through the CLI. The switch uses the MAC address and the system priority to form the system ID.
- Port priority: Each port in the switch must have a port priority. The port priority can be specified automatically or through the CLI. The port priority and the port number form the port identifier. The switch uses the port priority to decide which ports to put in standby mode when a hardware limitation prevents all compatible ports from aggregating.
- Administrative key: Each port in the switch must have an administrative key value, which can be specified automatically or through the CLI. The administrative key defines the capability of a port to aggregate with other ports, determined by these factors: the port’s physical characteristics, such as data rate, duplex capability, and point-to-point or shared medium.
All the preceding options of LACP are optional to configure. Usually, defaults are the best to use. To configure any of these options, refer to your configuration guide.
PAgP
Port Aggregation Protocol (PAgP) provides the same negotiation benefits as LACP. PAgP is a Cisco proprietary protocol, and it will work only on Cisco devices. PAgP packets are exchanged between switches over EtherChannel-capable ports. Neighbors are identified and capabilities are learned and compared with local switch capabilities. Ports that have the same capabilities are bundled together into an EtherChannel. PAgP forms an EtherChannel only on ports that are configured for identical VLANs or trunking. PAgP will automatically modify parameters of the EtherChannel if one of the ports in the bundle is modified. For example, if configured speed, duplex, or VLAN of a port in a bundle is changed, PAgP reconfigures that parameter for all ports in the bundle. PAgP and LACP are not compatible.
These are the following two PAgP modes of operation:
- Desirable: Enable PAgP
- Auto: Enable PAgP only if a PAgP device is detected
Layer 2 EtherChannel Configuration Guidelines
Before implementing EtherChannel in a network, plan the following steps necessary to make it successful:
- The first step is to identify the ports that you will use for the EtherChannel on both switches. This task helps identify any issues with previous configurations on the ports and ensures that the proper connections are available.
- Each interface should have the appropriate protocol identified (PAgP or LACP), have a channel group number to associate all the given interfaces with a port group, and be configured whether negotiation should occur.
- After the connections are established, make sure that both sides of the EtherChannel have formed and are providing aggregated bandwidth.
Follow these guidelines and restrictions when configuring EtherChannel interfaces:
- EtherChannel support: All Ethernet interfaces on all modules support EtherChannel, with no requirement that interfaces be physically contiguous or on the same module.
- Speed and duplex: Configure all interfaces in an EtherChannel to operate at the same speed and in the same duplex mode. Also, if one interface in the bundle is shut down, it is treated as a link failure, and traffic will traverse other links in the bundle.
- VLAN match: All interfaces in the EtherChannel bundle must be assigned to the same VLAN or be configured as a trunk.
- Range of VLANs: An EtherChannel supports the same allowed range of VLANs on all the interfaces in a trunking Layer 2 EtherChannel.
If the allowed range of VLANs is not the same, the interfaces do not form an EtherChannel, even when set to auto or desirable mode. For Layer 2 EtherChannels, either assign all interfaces in the EtherChannel to the same VLAN or configure them as trunks.
- STP path cost: Interfaces with different STP port path costs can form an EtherChannel as long as they are compatibly configured. Setting different STP port path costs does not, by itself, make interfaces incompatible for the formation of an EtherChannel.
- Port channel versus interface configuration: After you configure an EtherChannel, any configuration that you apply to the port channel interface affects the EtherChannel. Any configuration that you apply to the physical interfaces affects only the specific interface that you configured.
EtherChannel Load-Balancing Options
EtherChannel load balances traffic across links in the bundle. However, traffic is not necessarily distributed equally among all the links.
Frames are forwarded over an EtherChannel link that is based on results of a hashing algorithm. Options that switch can use to calculate this hash depends on the platform.
Table 3-6 shows the comment set of options for EtherChannel load balancing.
Table 3-6 EtherChannel Load-Balancing Options
Hash Input Code |
Hash Input Decision |
Switch Model |
dst-ip |
Destination IP address |
All models |
dst-mac |
Destination MAC address |
All models |
src-dst-ip |
Source and destination IP address |
All models |
src-dst-mac |
Source and destination MAC address |
All models |
src-ip |
Source IP address |
All models |
src-mac |
Source MAC address |
All models |
src-port |
Source port number |
4500, 6500 |
dst-port |
Destination port number |
4500, 6500 |
src-dst-port |
Source and destination port number |
4500, 6500 |
To verify load-balancing options available on the device, use the port-channel load-balance ? global configuration command.
The hash algorithm calculates a binary pattern that selects a link within the EtherChannel bundle to forward the frame.
If only one address or port number is hashed, a switch looks at one or more low-order bits of the hash value. The switch then uses those bits as index values to decide over which links in the bundle to send the frames.
If two or more addresses or port numbers are hashed, a switch performs an XOR operation.
A four-link bundle uses a hash of the last 2 bits. A bundle of eight links uses a hash of the last 3 bits.
Table 3-7 shows results of an XOR on a two-link bundle, using the source and destination addresses.
Table 3-7 XOR for Two-Link EtherChannels
Example IP Addresses |
IPs in Binary |
XOR Result |
Forward Frame over Link with Index |
Source: 192.168.1.2 |
Source: ...xxxxx0 |
...xxxxx0 |
0 |
Destination: 192.168.1.4 |
Destination: ...xxxxx0 |
||
Source: 172.16.1.20 |
Source: ...xxxxx0 |
...xxxxx1 |
1 |
Destination: 172.16.1.21 |
Destination: ...xxxxx1 |
||
Source: 192.168.1.1 |
Source: ...xxxxx1 |
...xxxxx1 |
1 |
Destination: 192.168.1.2 |
Destination: ...xxxxx0 |
||
Source: 10.1.1.101 |
Source: ...xxxxx1 |
...xxxxx0 |
0 |
Destination: 10.1.1.103 |
Destination: ...xxxxx1 |
A conversation between two devices is sent through the same EtherChannel link because the two endpoint addresses stay the same. Only when a device talks to several other devices does traffic get distributed evenly over the links in the bundle.
When one pair of hosts has a much greater volume of traffic than the other pair, one link will be much more utilized than others. To fix the imbalance, consider using some other load-balancing mechanisms, such as source and destination port number, that will redistribute traffic much differently.
If most of the traffic is IP, it makes sense to load balance according to IP addresses or port numbers. For non-IP traffic, the hash uses MAC addresses to calculate the path.
To achieve the optimal traffic distribution, always bundle an even number of links. For example, if you use four links, the algorithm will take the last 2 bits. These 2 bits mean four indexes: 00, 01, 10, and 11. Each link in the bundle will get assigned one of these indexes. If you bundle only three links, the algorithm still needs to use 2 bits to make decisions. One of the three links in the bundle will be used more than the other two. With four links, the algorithm strives to load balance traffic in a 1:1:1:1 ratio. A three-link algorithm strives to load balance traffic in a 2:1:1 ratio.
Configuring EtherChannel in a Switched Network
This section shows you how to configure the Layer 2 EtherChannel and explains its load-balancing behavior. Configure a port channel between SW1 and SW2 shown in Figure 3-26.
)
Figure 3-26 EtherChannel Configuration Topology
Table 3-8 shows device information.
Table 3-8 Device Information
Device |
IP Address |
Interface |
Neighbor |
Interface on the Neighbor |
PC 1 |
172.16.1.101/24 |
Ethernet 0/0 |
Switch 1 |
Ethernet 0/1 |
PC 2 |
172.16.1.102/24 |
Ethernet 0/0 |
Switch 1 |
Ethernet 0/2 |
PC 3 |
172.16.1.203/24 |
Ethernet 0/0 |
Switch 2 |
Ethernet 0/1 |
Switch 1 |
No IP address |
Ethernet 1/1 |
Switch 2 |
Ethernet 0/2 |
Switch 1 |
No IP address |
Ethernet 1/2 |
Switch 2 |
Ethernet 0/3 |
EtherChannel Configuration and Load Balancing
Complete the following steps to configure EtherChannel on Switch 1. Switch 2 has EtherChannel preconfigured.
Step 1. On Switch 1, configure the two ports that connect to Switch 2 to use channel group 1 and LACP active mode:
Switch1# configure terminal Switch1(config)# interface range Ethernet 1/1-2 Switch1(config-if-range)# channel-group 1 mode active Creating a port-channel interface Port-channel 1
Now the two interfaces are bundled into channel group 1. Because you chose the active keyword, LACP will work as the negotiation protocol. Because Switch 2 has its ports bundled and activated for LACP passive mode, EtherChannel should come right up.
%LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
Notice that by assigning the two ports to a port channel, the switch has created a port channel 1 interface.
Issue the show ip interface brief command. Port channel 1 will be listed as just another interface at the very bottom of the list.
Step 2. Enter interface configuration mode for the newly created port channel interface and configure it for trunk mode using dot1Q:
Switch1(config)# interface port-channel 1 Switch1(config-if)# switchport trunk encapsulation dot1q Switch1(config-if)# switchport mode trunk
The configuration applied to the port channel will also reflect on physical interfaces that are bundled into that port channel. You can investigate the running configuration and see that EtherChannel 1/1 and EtherChannel 1/2 both have had the trunking configuration applied.
Step 3. On Switch 1, enter the show etherchannel summary command:
Switch1# show etherchannel summary Flags: D - down P - bundled in port-channel I - stand-alone s - suspended H - Hot-standby (LACP only) R - Layer3 S - Layer2 U - in use f - failed to allocate aggregator M - not in use, minimum links not met u - unsuitable for bundling w - waiting to be aggregated d - default port Number of channel-groups in use: 1 Number of aggregators: 1 Group Port-channel Protocol Ports ------+-------------+-----------+-------------------------------------- 1 Po1(SU) LACP Et1/1(P) Et1/2(P)Group 1 port channel is a Layer 2 EtherChannel that is in use (SU flag). The negotiation protocol in use is LACP, and the ports bundled (notice the P flag) are Ethernet 1/1 and Ethernet 1/2.
If a port comes up but cannot join the port channel, it is denoted with an I flag (for “independent”).
Step 4. Enter the show etherchannel load-balance command to verify which information EtherChannel uses to load balance traffic:
Switch1# show etherchannel load-balance EtherChannel Load-Balancing Configuration: src-dst-ip EtherChannel Load-Balancing Addresses Used Per-Protocol: Non-IP: Source XOR Destination MAC address IPv4: Source XOR Destination IP address IPv6: Source XOR Destination IP addressNotice that the default configuration for load balancing is src-dst-ip. This means the source and destination IP address are used for hash input.
Step 5. For testing how much traffic goes over each link, as shown in Figure 3-27, clear interface counters on Switch 1 using the clear counters command:
Figure 3-27 EtherChannel Load-Balancing Configuration Option
Switch1# clear counters Clear "show interface" counters on all interfaces [confirm] [Enter]
By clearing the counters, you are setting up to test how much traffic goes over each link.
Step 6. Perform an extended ping from PC 1 to PC 3:
PC1# ping Protocol [ip]: [Enter] Target IP address: 172.16.1.203 Repeat count [5]: 10000 Datagram size [100]: 1500 Timeout in seconds [2]: [Enter] Extended commands [n]: [Enter] Sweep range of sizes [n]: [Enter] Type escape sequence to abort. Sending 10000, 1500-byte ICMP Echos to 172.16.1.203, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! <... output omitted ...>
In the next step, you check over which interface all the traffic went.
Step 7. Verify counters on Switch 1 for both interfaces:
Switch1# show interface ethernet 1/1 | i packets output 10094 packets output, 15146494 bytes, 0 underruns Switch1# show interface ethernet 1/2 | i packets output 13 packets output, 1664 bytes, 0 underrunsNotice that most of the traffic went over the Ethernet 1/1 interface.
But what about if you ping from PC 2 to PC 3? Will traffic go over the other interface in EtherChannel bundle?
Step 8. Clear interface counters on Switch 1 using the clear counters command:
Switch1# clear counters Clear "show interface" counters on all interfaces [confirm] [Enter]
Step 9. Perform an extended ping from PC 2 to PC 3:
PC2# ping Protocol [ip]: [Enter] Target IP address: 172.16.1.203 Repeat count [5]: 10000 Datagram size [100]: 1500 Timeout in seconds [2]: [Enter] Extended commands [n]: [Enter] Sweep range of sizes [n]: [Enter] Type escape sequence to abort. Sending 10000, 1500-byte ICMP Echos to 172.16.1.203, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! <... output omitted ...>
Step 10. Verify counters on Switch 1 for both interfaces:
Switch1# show interface ethernet 1/1 | i packets output 29 packets output, 2201 bytes, 0 underruns Switch1# show interface ethernet 1/2 | i packets output 10003 packets output, 15140537 bytes, 0 underrunsSo, with the ping from PC 1 to PC 3, traffic went over Ethernet 1/1. With the ping from PC 2 to PC 3, traffic went over Ethernet 1/2. This is for the default load-balancing method that takes destination and source IP address for calculating the hash.
Step 11. Change the load-balancing behavior on Switch 1 from src-dst-ip to dst-ip:
Switch1(config)# port-channel load-balance dst-ip
How will traffic get distributed over the two links now?
Step 12. Verify that the load-balancing behavior has changed:
Switch1# show etherchannel load-balance EtherChannel Load-Balancing Configuration: dst-ip EtherChannel Load-Balancing Addresses Used Per-Protocol: Non-IP: Source XOR Destination MAC address IPv4: Source XOR Destination IP address IPv6: Source XOR Destination IP addressStep 13. Clear the interface counters on Switch 1 by using the clear counters command:
Switch1# clear counters Clear "show interface" counters on all interfaces [confirm] [Enter]
Step 14. Perform an extended ping from PC 1 to PC 3:
PC1# ping Protocol [ip]: [Enter] Target IP address: 172.16.1.203 Repeat count [5]: 10000 Datagram size [100]: 1500 Timeout in seconds [2]: [Enter] Extended commands [n]: [Enter] Sweep range of sizes [n]: [Enter] Type escape sequence to abort. Sending 10000, 1500-byte ICMP Echos to 172.16.1.203, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! <... output omitted ...>
Step 15. Verify the counters on Switch 1 for both interfaces:
Switch1# show interface ethernet 1/1 | i packets output 32 packets output, 2108 bytes, 0 underruns Switch1# show interface ethernet 1/2 | i packets output 10002 packets output, 15140188 bytes, 0 underrunsThe majority of the traffic went over the Ethernet 1/2 port.
Step 16. Clear the interface counters on Switch 1 by using the clear counters command:
Switch1# clear counters Clear "show interface" counters on all interfaces [confirm] [Enter]
Step 17. Perform an extended ping from PC 2 to PC 3:
PC2# ping Protocol [ip]: [Enter] Target IP address: 172.16.1.203 Repeat count [5]: 10000 Datagram size [100]: 1500 Timeout in seconds [2]: [Enter] Extended commands [n]: [Enter] Sweep range of sizes [n]: [Enter] Type escape sequence to abort. Sending 10000, 1500-byte ICMP Echos to 172.16.1.203, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! <... output omitted ...>
Step 18. Verify counters on Switch 1 for both interfaces:
Switch1# show interface ethernet 1/1 | i packets output 31 packets output, 2329 bytes, 0 underruns Switch1# show interface ethernet 1/2 | i packets output 10004 packets output, 15140597 bytes, 0 underruns
)
){kind=link}
){kind=link}
Now that the load balancing is based on destination IP, the behavior has changed. Because the only input information for calculation of the hash is destination IP address, it does not matter whether you ping PC 3 from PC 1 or PC 2. In both cases, the hash function will be the same, and traffic will go over the same link (in this example, Ethernet ½).
EtherChannel Guard
The EtherChannel Guard feature is used to detect EtherChannel misconfigurations between the switch and a connected device.
EtherChannel misconfiguration occurs when the channel parameters do not match on both sides of the EtherChannel, resulting in the following message:
%PM-SP-4-ERR_DISABLE: channel-misconfig error detected on Po3, putting E1/3 in err-disable state
The EtherChannel Guard feature can be enabled by using the spanning-tree etherchannel guard misconfig global configuration command.
However, EtherChannel Guard is enabled by default. To verify whether it is configured, use the show spanning-tree summary command, as demonstrated in Example 3-13.
Example 3-13 Show VTP Status and Show VLAN outputs from SW1 and SW3
Switch1# show spanning-tree summary Switch is in pvst mode Root bridge for: VLAN0001 Extended system ID is enabled Portfast Default is disabled PortFast BPDU Guard Default is disabled Portfast BPDU Filter Default is disabled Loopguard Default is disabled EtherChannel misconfig guard is enabled <...output omitted...>