Fundamental Technologies
The two technologies that will play a big role in enabling I/O consolidation are PCI-Express and 10 Gigabit Ethernet (10GE).
PCI-Express
Peripheral Component Interconnect (PCI) is an old standard to interconnect peripheral devices to computer that has been around for many years [1].
PCI-Express (PCI-E or PCIe) [2] is a computer expansion card interface format designed to replace PCI, PCI-X, and AGP. It removes one of the limitations that have plagued all these I/O consolidation attempts (i.e., the lack of I/O bandwidth in the server buses), and it is compatible with current operating systems.
PCIe uses point-to-point full duplex serial links called lanes. Each lane contains two pairs of wires: one to send and one to receive. Multiple lanes can be deployed in parallel: 1x means a single lane; 4x means 4 lanes.
In PCIe 1.1, the lanes run at 2.5 Gbps (2 Gbit/s at the datalink), and 16 lanes can be deployed in parallel. This supports speeds from 2 Gbit/s (1x) to 32 Gbit/s (16x). Due to protocol overhead 8x is required to support a 10GE interface.
PCIe 2.0 (i.e., PCIe Gen 2) doubled the bandwidth per lane from 2 Gbit/s to 4 Gbit/s and extended the maximum number of lanes to 32x. It is shipping now.
PCIe 3.0 will approximatively double the bandwidth again: "The final PCIe 3.0 specifications, including form factor specification updates, may be available by late 2009, and could be seen in products starting in 2010 and beyond." [3].
10 Gigabit Ethernet
10GE is a practical interconnection technology since 2008. The standard has reached the maturity status and cheap cabling solutions are available. Fiber continues to be used for longer distances, but copper is deployed in the Data Center for its lower cost.
Switches and CNAs have standardized their connectivity using the Small Form-factor Pluggable (SFP) transceiver. SFPs are used to interface a network device motherboard (i.e., switches, routers, or CNAs) to a fiber optic or copper cable. SFP is a popular industry format supported by several component vendors. It has expanded to become SFP+, which supports data rates up to 10 Gbit/s [9]. Applications of SFP+ include 8GFC and 10GE.
The key benefits of SFP+ are:
- A comparable panel density as SFP
- A lower module power than XENPAK, X2, and XFP
- A Nominal 1W power consumption (optional 1.5W high power module)
- Backward compatibility with SFP optical modules
The IEEE standard for twisted pair cabling (10GBASE-T) is not yet a practical interconnection technology, because it requires an enormous number of transistors, especially when the distance grows toward 100 meters (328 feet). This translates to significant power requirements and into additional delay (see Figure 1-4). Imagine trying to cool a switch linecard that has 48 10GBASE-T ports on the front-panel, each consuming 4 watts!
)
Figure 1-4 Evolution of Ethernet Physical Media
A more practical solution in the Data Center, at the rack level, is to use SFP+ with copper Twinax cable (defined in SFF-8431, see [9]). The cable is flexible, approximately 6 mm (1/4 of an inch) in diameter, and it uses the SFP+ themselves as the connectors. Cost is limited; power consumption and delay are negligible. It is limited to 10 meters (33 feet) that are sufficient to connect a few racks of servers to a common top of the rack switch.
These cables are available from Cisco, Amphenol, Molex, Panduit, and others.
Figure 1-5 illustrates the advantages of using Twinax cable inside a rack or few racks.
)
Figure 1-5 Twinax Copper Cable
The cost of the transmission media is only one of the factors that need to be addressed to manufacture 10GE ports that are cost competitive. Other factors are the size of the switch buffers, and Layer 2 versus Layer 3/4 functionality.