Cisco WAAS Architecture, Hardware, and Sizing

Performance and Scalability Metrics

Design of a Cisco WAAS solution involves many factors, but the cornerstone of the solution design is based on the performance and scalability metrics required for the solution as a whole and for each individual location where WAAS is deployed. Every component in an end-to-end system has a series of static and dynamic system limits. For instance, a typical application server might be limited in terms of the number of connections it can support, disk I/O throughput, network throughput, CPU speed, or number of transactions per second. Likewise, each Cisco WAAS device has static and dynamic system limits that dictate how and when a particular WAAS device is selected for a location within an end-to-end design. This section examines the performance and scalability metrics of the Cisco WAAS hardware family, and provides a definition of what each item is and how it is relevant to a localized (per location) design and an end-to-end system design.

The static and dynamic limits referred to are used as a means of identifying which device is best suited to provide services to a particular location in the network. The device might be deployed as an edge device, where it connects to potentially many peer devices in one or more data center locations, or as a core device, where it serves as an aggregation point for many connected edges. WAAS devices can also be deployed as devices to optimize links between data center locations, where devices on each side are realistically core devices. A fundamental understanding of the performance and scalability metrics is paramount in ensuring a sound design. Although WAAS devices have no concept of "core" or "edge," the deployment position within the network has an effect on the type of workload handled by a device and should be considered—primarily as it relates to TCP connection count and peer fan-out (how many peers can connect to a device for the purposes of optimization). This section examines each of the performance and scalability system limits, both static and dynamic, that should be considered. These include device memory, disk capacity, the number of optimized TCP connections, WAN bandwidth and LAN throughput, the number of peers and fan-out, and the number of devices managed.

Device Memory

The amount of memory installed in a device dictates the level of performance and scalability the device can provide. As the memory capacity increases, the ability of a WAAS device to handle a larger number of connections, a larger addressable index space for compression, or a longer history of compression data also increases. Having larger amounts of memory also enables the WAAS device to run additional services, such as application acceleration, disk encryption, or virtualization, and positions the device to accept additional features that might be introduced in future software releases.

The NME-WAE family members have fixed memory capacity and cannot be upgraded. Thus, the system limits for the NME-WAE family are static. From the WAE appliance family, the 7341 and 7371 have fixed memory configurations. However, the WAE-512, WAE-612, and WAE-674 have configurable memory options, in that:

• The WAE-512 can be configured with 1 GB or 2 GB of memory.
• The WAE-612 can be configured with 2 GB or 4 GB of memory.
• The WAE-674 can be configured with 4 GB or 8 GB of memory.

For devices that support flexible memory configuration (such as the WAE-512, WAE-612, and WAE-674), higher levels of WAN bandwidth can be realized, along with an increase in the number of optimized TCP connections that can be handled concurrently by that device. For virtualization-capable platforms, a larger number of VBs can be supported. The WAVE appliance family models 274 and 474, like the network modules, are fixed configuration and do not support a memory upgrade, whereas the 574 model—like the WAE 512, 612, and 674—does support memory configuration (either 3 GB or 6 GB).

The amount of installed memory directly impacts what license is supported on each of the device models. The Transport license can be configured on any WAAS hardware model. WAAS hardware models that have 1 GB of memory or more (all do except the NME-WAE-302) can be configured with the Enterprise license, which allows the WAAS device to operate all of the Enterprise license features.

Previous versions of Cisco WAAS (version 4.0.x and version 4.1.x when using legacy mode compatibility) had distinct core and edge CIFS acceleration services. With legacy mode, a device with 1 GB of RAM can support only edge services for CIFS, whereas a device with 2 GB of RAM or more can support edge or core services, or both together. As of Cisco WAAS version 4.1.1, this deployment mode is no longer required unless interoperability with version 4.0.x is required. Generally speaking, most customers upgrade the entire network in a short and well-defined period of time and can take advantage of the simplified deployment model provided in 4.1.x, which does not have such restrictions.

Disk Capacity

Optimization services in the Cisco WAAS hardware family leverage both memory and disk. From a disk perspective, the larger the amount of available capacity, the larger the amount of optimization history that can be leveraged by the WAAS device during run-time operation. For instance, an NME-WAE-502 has 120 GB of physical disk capacity, of which 35 GB is available for use by DRE for compression history. With 35 GB of compression history, one can estimate the length of the compression history given WAN conditions, expected network utilization, and assumed redundancy levels.

Table 2-1 shows how the length of the compression history can be calculated for a particular WAAS device, along with an example. This example assumes a T1 WAN that is 75 percent utilized during business hours (75 percent utilization over 8 hours per day) and 50 percent utilized during nonbusiness hours (16 hours per day), and assumes that data traversing the network is 75 percent redundant (highly compressible by DRE). This table also assumes an NME-WAE-502 with 35 GB of allocated capacity for DRE compression history.

Table 2-1. Calculating Compression History

It is generally recommended that, at minimum, five days of compression history be available in a WAAS device to better ensure that substantial performance improvements are possible. In the example in Table 2-1, the NME-WAE-502 contains enough storage capacity to provide an effective compression history of two weeks. In most cases, users tend to access data that is newer more frequently, whereas older data is accessed less frequently. Because of this, having five days worth of compression history could even be considered overkill.

The disk capacity available to a WAAS device is split among five major components:

• DRE compression history: This capacity is used for storing DRE chunk data and signatures.
• CIFS cache: This capacity is preallocated on all devices using the Enterprise license.
• Print services: This capacity is preallocated for print spool capacity. Print services require that the Enterprise license be configured and that CIFS edge services be configured, which implies that legacy mode is being used. In cases where print services are configured, the 1 GB of disk capacity is allocated. Given that 1 GB is a fraction of the total storage capacity of a device, it is not accounted for in Table 2-2.

  Table 2-2. Disk Capacity Allocation per Platform

  Platform	Total Usable Capacity	DRE	CIFS	VBs
  NME-WAE-302	80 GB	30 GB	0 GB	0 GB
  NME-WAE-502	120 GB	35 GB	49 GB	0 GB
  NME-WAE-522	160 GB	67 GB	67 GB	0 GB
  WAVE-274	250 GB	40 GB	120 GB	35 GB
  WAVE-474	250 GB	60 GB	120 GB	35 GB
  WAE-512-1GB	250 GB RAID-1	60 GB	120 GB	0 GB
  WAE-512-2GB	250 GB RAID-1	80 GB	100 GB	0 GB
  WAVE-574-3GB	500 GB RAID-1	80 GB	120 GB	60 GB
  WAVE-574-6GB	500 GB RAID-1	120 GB	120 GB	180 GB
  WAE-612-2GB	300 GB RAID-1	100 GB	120 GB	0 GB
  WAE-612-4GB	300 GB RAID-1	120 GB	120 GB	0 GB
  WAE-674-4GB	600 GB RAID-5	120 GB	120 GB	120 GB
  WAE-674-8GB	600 GB RAID-5	150 GB (with VB) 320 GB (without VB)	120 GB	200 GB (with VB) 0 GB (without VB)
  WAE-7326	900 GB RAID-1	320 GB	230 GB	0 GB
  WAE-7341	900 GB RAID-5	500 GB	230 GB	0 GB
  WAE-7371	1500 GB RAID-5	1 TB	230 GB	0 GB

• Platform services: This capacity is preallocated for operating system image storage, log files, and swap space.
• Virtual Blades: This capacity is preallocated for any guest operating systems and applications that are installed to run in a WAAS VB.

Table 2-2 shows the storage allocation for each WAAS device for each of these components.

Number of Optimized TCP Connections

Each WAAS device has a static number of TCP connections that can be optimized concurrently. Each TCP connection is allocated memory and other resources within the system, and if the concurrently optimized TCP connection static limit is met, additional connections are handled in a pass-through fashion. Adaptive buffering (memory allocation) is used to ensure that more active connections are allocated additional memory, and less active connections are only allocated the memory they require.

The TCP connection limit of each WAAS device can be roughly correlated to the number of users supported by a given WAAS device model, but note that the number of TCP connections open on a particular node can vary based on user productivity, application behavior, time of day, and other factors. It is commonly assumed that a user will have 5 to 15 connections open at any given time, with roughly 6 to 10 of those connections requiring optimization. If necessary, policies can be adjusted on the WAAS Central Manager to pass through certain applications that might realize only a small amount of benefit from WAAS. This type of change could potentially help increase the number of users that can be supported by a particular WAAS device.

Table 2-3 shows the optimized TCP connection capacity per device model.

Table 2-3. Optimized TCP Connection Capacity per Platform

The number of connections a typical user has in a location can be determined by using tools that exist in the operating system of the user's workstation. Although the estimate of six to ten optimized TCP connections is accurate for the broad majority of customers, those that wish to more accurately determine exactly how many connections a typical user has open at any given time can do so.

Microsoft provides two methods for determining the number of connections that are open on a given computer. The first is through the Command Prompt program netstat. By opening a Command Prompt window (click Start > Run, then type cmd and click Ok) and typing the command netstat, you can see a list of the open connections from the computer to all of the other endpoints to which that computer is connected. Notice the connections that are in the state of ESTABLISHED. These connections are currently open and in use and have not yet been closed. In many cases, the protocol associated with the connection is listed next to the foreign address, but some might not be. From here, you can identify the servers to which the user is connected and determine which should and should not be optimized. Figure 2-7 shows an example of the output of this command.

Figure 2-7 Determining the Number of TCP Connections In Use Using netstat

Another tool provided by Microsoft that (along with many other things) provides visibility into the number of TCP connections in use on a particular computer is Performance Monitor. Performance Monitor can be accessed by clicking Start > Run and typing perfmon, followed by clicking Ok. From within the Performance Monitor window, click the + sign, select the TCP performance object, and then add the Connections Established counter. Doing so shows you the number of connections established over time, and this data can even be exported for offline use. Figure 2-8 illustrates an example output from Performance Monitor showing the number of established TCP connections.

Figure 2-8 Determining the Number of TCP Connections in Use Using Performance Monitor

Linux, UNIX, and Macintosh provide similar tools to understand the number of connections that are open on a given computer. The netstat command is available on virtually any Linux distribution and is available in most UNIX platforms and versions of Apple's Macintosh OS/X operating system.

For the data center, the sum of all remote office TCP connections should be considered one of the key benchmarks by which the data center sizing should be done. Note that the largest Cisco WAAS device supports up to 50,000 optimized TCP connections—which is approximately 5,000 users (assuming ten TCP connections per user). For organizations that need to support a larger number of users or want to deploy the data center devices in a high-availability manner, multiple devices can be used. The type of network interception used (discussed in Chapter 4) determines the aggregate number of optimized TCP connections that can be supported by a group of Cisco WAAS devices deployed at a common place within the data center. Recommended practice dictates that sites that require high availability be designed with N+1 availability in consideration relative to the number of maximum optimized TCP connections—that is, if 100,000 optimized TCP connections must be supported, the location should have a minimum of two WAE-7371 devices to support the workload, a third WAE-7371 device to handle failure of one of the devices, and use an interception mechanism such as WCCP or ACE that supports load-balancing of workload across the entire set of three devices. Other considerations apply, as discussed in Chapter 4.

WAN Bandwidth and LAN Throughput

WAAS devices are not restricted in software or hardware in terms of the amount of WAN bandwidth or LAN throughput supported. However, recommendations are in place to specify which WAAS device should be considered for a specific WAN environment. WAN bandwidth is defined as the amount of WAN capacity that the WAAS device can fully use when employing the full suite of optimization capabilities (this includes DRE, PLZ, TFO, and the other application acceleration capabilities). LAN throughput is defined as the maximum amount of application layer throughput (throughput as perceived by the users and servers) that can be achieved with the particular WAAS hardware model and an equivalent or more-powerful peer deployed at the opposite end of the network.

For some deployment scenarios, it is desired to use the Cisco WAAS devices only for TCP optimization. Cisco WAAS TFO provides a powerful suite of optimizations to better allow communicating nodes to "fill the pipe" (that is, fully leverage the available WAN bandwidth capacity) when the application protocol is not restricting throughput due to application-induced latency. Each Cisco WAAS device has a TFO-only throughput capacity that can be considered when WAAS devices are deployed strictly for TCP optimization only. This is recommended only for situations where compression, redundancy elimination, and application acceleration are not required, and the application throughput has been validated to be hindered only by the performance of the TCP implementation in use. This is common in some data center to data center applications—such as data replication or data protection—where the traffic that is sent is previously compressed, redundancy eliminated, or encrypted. TFO attempts to fully utilize the available bandwidth capacity, but might be hindered by congestion in the network (not enough available bandwidth) or performance impedance caused by application protocol chatter.

Table 2-4 shows the WAN bandwidth supported by each WAAS device model and the maximum LAN-side throughput and TFO-only throughput capacity. Note that other factors can influence these values and throughput levels can be achieved only when the link capacity available supports such a throughput level. For instance, a LAN throughput maximum of 150 Mbps is not possible on a Fast Ethernet connection; rather, a Gigabit Ethernet connection is required. Similarly for throughput speeds more than 1 Gbps, multiple 1-Gbps interfaces must be used.

Table 2-4. WAN Bandwidth and LAN Throughput Capacity per WAAS Device

The amount of bandwidth required per site is the sum of available WAN capacity that can be used at that site and not the sum of all WAN bandwidth for every connected peer. For instance, if a branch office has four bundled T1 links (totaling 6 Mbps of aggregate WAN throughput) but only two are used at any given time (high availability configuration), a device that supports 3 Mbps or more is sufficient to support the location.

Similarly, if a data center has four DS-3 links (totaling 180 Mbps of aggregate WAN throughput) but uses only three at a time (N+1 configuration), a device that supports 135 Mbps of WAN bandwidth or more is sufficient to support that location. The WAN throughput figures mentioned in the preceding table are (as discussed previously) not limited in hardware or software. In some cases, the WAN throughput that a device achieves might be higher than the values specified here. Those interested in using a smaller device to support a larger WAN link (for instance, qualifying a WAVE-274 for an 8-Mbps ADSL connection) are encouraged to test the system under those conditions and validate the performance prior to making a decision to use that specific platform.

Number of Peers and Fan-Out

Each Cisco WAAS device has a static system limit in terms of the number of concurrent peers it can actively communicate with at any one given time. When designing for a particular location where the number of peers exceeds the maximum capacity of an individual device, multiple devices can be deployed, assuming an interception mechanism that uses load balancing is employed (such as WCCPv2 or ACE; these are discussed in Chapter 4). In cases where load balancing is used, TCP connections are distributed according to the interception configuration, thereby allowing for near-linear scalability increases in connection count, peer count, and WAN bandwidth, as devices are added to the pool. Load-balancing interception techniques are recommended when multiple devices are used in a location, and in general, an N+1 design is recommended.

Peer relationships are established between Cisco WAAS devices during the automatic discovery process on the first connection optimized between the two devices. These peer relationships time out after ten minutes of inactivity (that is, no active connections are established and optimized between two peers for ten minutes). Each WAAS device supports a finite number of active peers, and when the peer relationship is timed out, that frees up peering capacity that can be reused by another peer. Data stored in the DRE compression history remains intact even if a peer becomes disconnected due to inactivity, unless the DRE compression history becomes full. In cases where the DRE compression history becomes full, an eviction process is initiated to remove the oldest set of data in the DRE compression history to make room for new data.

Table 2-5 shows the maximum number of concurrent peers supported per WAAS platform. If peers are connected beyond the allocated limit, the WAE permits the connections to be established and gracefully degrades performance as needed. Connections associated with peers in excess of the maximum fan-out ratio are able to use the existing compression history but are not able to add new chunks of data to it. The end result is lower effective compression ratios for the connections using peers that are in excess of the specified fanout ratio.

Table 2-5. Maximum Supported Peers per WAAS Device

The number of peers supported by a device is typically the last factor that should be considered when sizing a solution for a particular location. The primary reason being that the WAN capacity or number of connections supported at the maximum concurrent peers specification is generally an order of magnitude higher than what the device can support. For instance, although a WAE-7371 can support up to 2800 peers, even if those peers were the NME-302 (each supporting 250 optimized TCP connections), it is not able to handle the 700,000 possible optimized TCP connections that all 2,800 NME-302s were attempting to optimize with it. It is best to size a location first based on WAN bandwidth capacity and TCP connections, and in most cases, only a simple validation that the number of peers supported is actually required.

Number of Devices Managed

Each Cisco WAAS deployment must have at least one Cisco WAAS device deployed as a Central Manager. The Central Manager is responsible for system-wide policy definition, synchronization of configuration, device monitoring, alarming, and reporting. The Central Manager can be deployed only on appliances and can be deployed in an active/standby fashion. When a certain WAAS device is configured as a Central Manager, it is able to, based on the hardware platform selected for the Central Manager, manage a maximum number of WAAS devices within the topology. Only WAAS appliances can be configured as Central Manager devices, and in high-availability configurations, each Central Manager WAE should be of the same hardware configuration. Although hardware disparity between Central Manager WAEs works, it is not a recommended practice given the difference in the number of devices that can be managed among the WAE hardware models. It should be noted that standby Central Managers (such a configuration is examined in Chapter 7) receive information in a synchronized manner identical to how accelerator WAAS devices do. Table 2-6 shows the maximum number of managed nodes that can be supported by each WAAS appliance when configured as a Central Manager.

Table 2-6. Central Manager Scalability

Use of multiple WAAS devices configured as Central Manager devices do not increase the overall scalability in terms of the number of devices that can be managed. To manage a number of devices greater than the capacities mentioned in the preceding table, multiple autonomous Central Managers are needed. For instance, in an environment with 3000 devices, two separate instances of Central Manager are required, and each instance can be comprised of a single device or multiple devices deployed in a high availability primary/standby configuration.

Replication Acceleration

The WAE-7341 and WAE-7371 devices support a deployment mode called Replication Accelerator, which requires Cisco WAAS version 4.0.19, or a version newer than that from the 4.0 train. This mode of acceleration is used for data center to data center deployments where replication and backup acceleration is required, and when configured, adjusts the behavior of the WAAS device to allocate larger blocks of memory to a smaller number of connections, and minimizes the processing latency of DRE by using only memory for deduplication. Although only memory is used for DRE, the DRE data is persistent in that it is written to disk, but the disk is used only to reload the previous compression history. This enables WAAS to provide high levels of throughput necessary to accelerate replication and backup traffic between data centers.

The network typically found in these cases is high-bandwidth and relatively low latency (above 10–20 ms), where a significant amount of data needs to be moved from one location to another location in a short period of time. The performance and scalability metrics of replication accelerator mode are different than the performance and scalability metrics that would normally be considered for these devices when not deployed in replication accelerator mode and are documented in Table 2-7.

Table 2-7. Replication Accelerator Performance and Scalability Metrics

Although all WAAS devices in a given network can be managed by a common Central Manager, WAAS devices configured in replication accelerator mode can only peer with other WAAS devices that are configured as replicator accelerator devices. Should intermediary application accelerator devices exist in the network path between two replication accelerator devices (this is generally rare, as replication accelerator devices are deployed between backend networks as opposed to the enterprise WAN), the application accelerator devices are not able to peer with replication accelerator devices.

Replication accelerator devices are commonly deployed on backend data center to data center networks and not the enterprise WAN due to the high bandwidth requirements. WAAS devices configured as replication accelerators are commonly found deployed as follows:

• Directly attached to one or more storage array IP/Ethernet interfaces: Such a deployment model dedicates the devices to optimize replication for that particular array and that particular interface.
• Directly attached to one or more storage fabric switch or director IP/Ethernet interfaces: Including the Cisco MDS 9000 family, such a deployment model enables the devices to optimize replication or backup traffic traversing fabrics in distant sites over IP.
• Directly behind the data center interconnect device: Such a deployment model enables optimization of any traffic between data centers. In this deployment model, replication accelerator should be carefully considered against the standard application accelerator mode which may be more applicable in cases where a large body of non-replication and nonbackup traffic exists.

Virtual Blades

The Cisco WAVE appliance family and the WAE-674 provide branch office virtualization capabilities that enable consolidation of remote branch office servers onto the WAAS device as a shared platform. Sizing for VBs should be done in conjunction with sizing for WAN optimization and application acceleration because the available disk capacity to support VBs and the number of VBs supported varies per platform based on the hardware configuration as shown in Table 2-8.

Table 2-8. VB Capacity

To accurately size a virtualization solution for a branch office, it is necessary to understand the minimum and recommended memory requirements to support the operating system and applications you plan to install on top of that operating system. Many vendors support installation of their server operating system onto systems with only 512 MB of memory, which increases the maximum number of VBs that can be installed on a WAAS device; however, many have requirements for larger amounts of memory.

Additionally, consider the disk capacity requirements necessary for each VB, and reconcile that amount with the total VB storage capacity of the platform selected for that given location. Even the smallest virtualization-capable WAAS device (the WAVE-274) supports 35 GB of disk capacity for VBs—meaning that with two VBs, configured, you have approximately 17.5 GB of disk space for each. Storage capacity allocation is flexible in that you can allocate as much space as is available from the pool to any particular VB. However, you should ensure that you size the system for the location with enough capacity to support the current application and operating system requirements as well as future requirements. More information on configuration and deployment of VBs can be found in Chapter 10, "Branch Office Virtualization."