Introduction to the IWAN Domain
An IWAN domain is a collection of sites that share the same set of policies and are managed by the same logical PfR domain controller. Each site runs PfR and gets its path control configuration and policies from the logical IWAN domain controller through the IWAN peering service. At each site, an MC is the local decision maker and controls the BRs responsible for performance measurement and path enforcement. The IWAN domain can be an entire enterprise WAN, a particular region, and so forth.
The key point for PfRv3 is that provisioning is fully centralized at a logical domain controller, whereas path control decisions and enforcement are fully distributed within the sites that are in the domain, making the solution very scalable.
Figure 7-5 shows a typical IWAN domain with central and branch sites. R10, R20, R31, R41, and R51 are all MCs for their corresponding sites and retrieve their configuration from the logical domain controller. R11, R12, R21, R22, R31, R41, R51, and R52 are all BRs that report back to their local MC. Notice that R31, R41, and R51 operate as both the MC and the BR for their sites.
)
Figure 7-5 IWAN Domain Concepts
IWAN Sites
An IWAN domain includes a mandatory hub site, optional transit sites, as well as branch sites. Each site has a unique identifier called a site ID that is derived from the loopback address of the local MC.
Central and headquarters sites play a significant role in PfR and are called IWAN Points of Presence (POPs). Each site has a unique identifier called a POP ID. These sites house DMVPN hub routers and therefore provide the following traffic flows (streams):
Traditional DMVPN spoke-to-hub connectivity.
Spoke-to-hub-to-spoke connectivity until DMVPN spoke-to-spoke tunnels establish.
Connectivity through NHS chaining until DMVPN spoke-to-spoke tunnels establish.
Transit connectivity to another site via a data center interconnect (DCI) or shared data center network segment. In essence, these sites act as transit sites for the traffic crossing them. Imagine in Figure 7-5 that R31 goes through R21 to reach a network that resides in Site 1. R21 does not terminate the traffic at the local site; it provides transit connectivity to Site 1 via the DCI.
Data centers may or may not be colocated with the hub site. To elaborate further, some hub sites contain data centers whereas other hub sites do not contain data centers (such as outsourced colocation cages).
Hub site
The logical domain controller functions reside on this site’s MC.
Only one hub site exists per IWAN domain because of the uniqueness of the logical domain controller’s presence. The MC for this site is known as the Hub MC, thereby making this site the hub site.
MCs from all other sites (transit or branch) connect to the Hub MC for PfR configuration and policies.
A POP ID of 0 is automatically assigned to a hub site.
A hub site may contain all other properties of a transit site as defined below.
Transit sites
Transit sites are located in an enterprise central site, headquarters, or carrier-neutral facilities.
They provide transit connectivity to access servers in the data centers or for spoke-to-spoke traffic.
A data center may or may not be colocated with the transit site. A data center can be reached via a transit site.
A POP ID is configured for each transit site. This POP ID has to be unique in the domain.
The local MC (known as a Transit MC) peers with the Hub MC (domain controller) to get its policies and to monitor configuration and timers.
Branch sites
These are always DMVPN spokes and are stub sites where traffic transit is not allowed.
The local Branch MC peers with the logical domain controller (Hub MC) to get its policies and monitoring guidelines.
Figure 7-6 shows the IWAN sites in a domain with two central sites (one is defined as the hub site and the other as a transit site). R10, R11, and R12 belong to the hub site, and R20, R21, and R22 belong to a transit site. R31, R41, R51, and R52 belong to a branch site. The dotted lines represent the site’s local MC peering with the Hub MC.
)
Figure 7-6 IWAN Domain Hub and Transit Sites
Device Components and Roles
The PfR architecture consists of two major Cisco IOS components, a master controller (MC) and a border router (BR). The MC is a policy decision point where policies are defined and applied to various traffic classes (TCs) that traverse the BR systems. The MC can be configured to learn and control TCs on the network:
Border routers (BRs) are in the data-forwarding path. BRs collect data from their Performance Monitor cache and smart probe results, provide a degree of aggregation of this information, and influence the packet forwarding path as directed by the site local MC to manage router traffic.
The master controller (MC) is the policy decision maker. At a large site, such as a data center or campus, the MC is a dedicated (physical or logical) router. For smaller branch locations, the MC is typically colocated (configured) on the same platform as the BR. As a general rule, large locations manage more network prefixes and applications than a branch deployment, thus consuming more CPU and memory resources for the MC function. Therefore, it is a good design practice to dedicate a chassis for the MC at large sites.
Each site in the PfR domain must include a local MC and at least one BR.
The branch typically manages fewer active network prefixes and applications. Because of the costs associated with dedicating a chassis at each branch, the network manager can colocate the local MC and BR on the same router platform. CPU and memory utilization should be monitored on platforms operating as MCs, and if utilization is high, the network manager should consider an MC platform with a higher-capacity CPU and memory. The local MC communicates with BRs and the Hub MC over an authenticated TCP socket but has no requirement for populating its own IP routing table with anything more than a route to reach the Hub MC and local BRs.
PfR is an intelligent path selection technology and requires
At least two external interfaces under the control of PfR
At least one internal interface under the control of PfR
At least one configured BR
If only one BR is configured, both external interfaces are attached to the single BR.
If more than one BR is configured, two or more external interfaces are configured across these BRs.
The BR, therefore, owns external links, or exit points; they may be logical (tunnel interfaces) or physical links (serial, Ethernet, and so on). With the IWAN prescriptive design, external interfaces are always logical DMVPN tunnels.
A device can fill five different roles in an IWAN domain:
Hub MC: This is the MC at the hub site. It acts as MC for the site, makes optimization decisions for that site, and provides the path control policies for all the other MCs. The Hub MC contains the logical PfR domain controller role.
Transit MC: This is the MC at a transit site that makes optimization decision for those sites. There is no policy configuration on Transit MCs because they receive their policies from the Hub MC.
Branch MC: The Branch MC is the MC for branch sites that makes optimization decisions for those sites. There is no policy configuration on Branch MCs because they receive their policies from the Hub MC.
Transit BR: The Transit BR is the BR at a hub or transit site. The WAN interface terminates in the BRs. PfR is enabled on these interfaces. At the time of this writing, only one WAN interface is supported on a Transit BR. This limitation is overcome by using multiple BR devices.
Branch BR: The Branch BR resides at the branch site and forwards traffic based on the decisions of the Branch MC. The only PfR configuration is the identification of the Branch MC and setting its role as a BR. The WAN interface that terminates on the device is detected automatically.
The PfR Hub MC is currently supported only on the IOS and IOS XE operating systems.
IWAN Peering
PfR uses an IWAN peering service between the MCs and BRs which is based on a publish/subscribe architecture. The current IWAN peering service uses Cisco SAF to distribute information between sites, including but not limited to
Learned site prefix
PfR policies
Performance Monitor information
The IWAN peering service provides an environment for service advertisement and discovery in a network. It is made up of two primary elements: client and forwarder.
An IWAN peering service client is a producer (advertises to the network), a consumer of services (requests a service from the network), or both.
An IWAN peering service SAF forwarder receives services advertised by clients, distributes the services reliably through the network, and makes services available to clients.
An IWAN peering service client needs to send a register message to a forwarder before it is able to advertise (publish) or request (subscribe to) services.
The IWAN peering service also adopts a logical unicast topology to implement the peering system. Each instance that joins the IWAN peering service serves as both a client and a forwarder:
The Hub MC listens for unicast packets for advertisements or publications from Transit MCs, Branch MCs, and local BRs.
The Transit MC peers with the Hub MC and listens to its local BRs.
The Branch MC peers with the Hub MC and listens to its local BRs.
BRs always peer with their local MC.
Figure 7-7 illustrates the IWAN peering service with the policies advertised from the Hub MC, the advertisement of monitors, and the exchange of site prefixes.
)
Figure 7-7 IWAN SAF Peering Service
SAF is automatically configured when PfR is enabled on a site. SAF dynamically discovers and establishes a peering as defined previously. The Hub MC advertises all policies and monitoring configuration to all the sites. Every site is responsible for advertising its own site prefix information to other sites in the domain.
Each instance must use an interface with an IP address that is reachable (routed) through the network to join in the IWAN peering system. PfRv3 requires that this address be a loopback address. It is critical that all these loopback addresses be reachable across the IWAN domain.
Parent Route Lookups
PfR uses the concept of a parent route lookup which refers to locating all the paths that a packet can take to a specific network destination regardless of the best-path calculation. The parent route lookup is performed so that PfR can monitor all paths and thereby prevent network traffic from being blackholed because the BRs have only summary routes in their routing table. PfR has direct API accessibility into EIGRP and BGP and can identify all the paths available for a prefix regardless of whether alternative paths were installed into the RIB.
PfR requires a parent route for every WAN path (primary, secondary, and so on) for PfR to work effectively. PfR searches the following locations in the order listed to locate all the paths for a destination:
NHRP cache (when spoke-to-spoke direct tunnels are established)
BGP table (where applicable)
EIGRP topology table (where applicable)
Static routes (where applicable)
RIB. Only one path is selected by default. In order for multiple paths to be selected, the same routing protocol must find both paths to be equal. This is known as equal-cost multipathing (ECMP).
The following logic is used for parent route lookups:
The parent route lookup is done during channel creation (see the following section, “Intelligent Path Control Principles,” for more information).
For PfR Internet-bound traffic, the parent route lookup is done every time traffic is controlled.
In a typical IWAN design, BGP or EIGRP is configured to make sure MPLS is the preferred path and the Internet the backup path. Therefore, for any destination prefix, MPLS is the only available path in the RIB. But PfR looks into the BGP or EIGRP table and knows if the Internet is also a possible path and can use it for traffic forwarding in a loop-free manner.