Designing Scalable BGP Designs
Border Gateway Protocol (BGP) is commonly used in sites with multiple connections to the Internet. BGP is also frequently present in medium-to large networks to provide a controlled interconnection between multiple routing domains running OSPF or EIGRP. Large-scale internal BGP networks are also becoming more prevalent as large enterprises implement internal Multiprotocol Label Switching (MPLS) VPNs for security segmentation, business unit or brand isolation, and similar purposes.
This section discusses designing advanced routing solutions using BGP. It describes how to identify scaling issues in internal BGP designs and how to use techniques to alleviate these issues.
Scaling BGP Designs
This section discusses aspects of scaling in basic internal BGP (IBGP) design. This is illustrated in Figure 3-20.
)
Figure 3-20 IBGP Full-Mesh Requirement
BGP can provide a controlled interconnection between multiple routing domains running OSPF or EIGRP and support internal MPLS VPNs. IBGP requires a full mesh of BGP peers.
The full mesh of IBGP routers is needed because IBGP routers do not re-advertise routes learned via IBGP to other IBGP peers. This behavior is part of BGP protocol behavior that is used to prevent information from circulating between IBGP speaking routers in a routing information loop or cycle. External BGP (EBGP) relies on the autonomous system path to prevent loops. However, there is no way to tell whether a route advertised through several IBGP speakers is a loop. Because IBGP peers are in the same autonomous system, they do not add anything to the autonomous system path, and they do not re-advertise routes learned via IBGP.
Full-Mesh IBGP Scalability
Because IBGP requires a full mesh of peers, scaling the full mesh is a concern. In general, for N peers in an IBGP full mesh, each would have N - 1 peers. There are N (N - 1) / 2 peering relationships. This means that each peer would need the CPU, memory, and bandwidth to handle updates and peer status for all the other routers. This is not a hierarchical design, and it would not be cost-effective to scale for large networks.
There are two IBGP alternatives to scale IBGP:
- Route reflectors
- Confederations
The following sections explore the basic design and behavior of route reflectors and confederations and demonstrate how they can be used in a routing design.
Scaling IBGP with Route Reflectors
A BGP route reflector is an IBGP speaker that reflects or repeats routes learned from IBGP peers to some of its other IBGP peers. This is shown in Figure 3-21.
)
Figure 3-21 BGP Route Reflectors
To prevent loops, a route reflector adds an originator ID and a cluster list to routes that it reflects between IBGP speakers. These attributes act similarly to the autonomous system path attribute to prevent routing information loops.
All configuration of the route reflector is done on the route reflector itself. The configuration identifies which IBGP peers are route reflector clients.
Implementing route reflectors is fairly simple and can be done incrementally. Each client router needs to be configured as a client on the route reflector or on multiple route reflectors. Unnecessary peers can then be removed from the configuration on the client router. Often, route reflector clients peer only with the route reflectors. In a service provider network, route reflector clients might also be provider edge (PE) devices, which also peer with customers using EBGP.
To avoid a single point of failure, redundant route reflectors are typically used.
BGP Route Reflector Definitions
A route reflector client (shown in Figure 3-22) is an IBGP router that receives and sends routes to most other IBGP speakers via the route reflector. The route reflector client needs no special configuration, other than removing peering with some or all neighbors other than the route reflector.
)
Figure 3-22 BGP Route Reflector Definitions
A cluster is a route reflector together with its clients. The route reflector relieves the route reflector client routers of needing to be interconnected via an IBGP full mesh.
Route reflector clusters may overlap.
A nonclient router (shown in Figure 3-23) is any route reflector IBGP peer that is not a route reflector client of that route reflector.
)
Figure 3-23 Additional BGP Route Reflector Definitions
Route reflectors are typically nonclients with regard to the other route reflectors in the network.
Route reflectors must still be fully IBGP meshed with nonclients. Therefore, route reflectors reduce meshing within clusters, but all mesh links outside the cluster must be maintained on the route reflector. The route reflector clients get information from IBGP speakers outside the cluster via the route reflector.
If a route reflector receives a route from a nonclient, it reflects it to route reflector clients but not to other nonclients. The route reflector receives the routes if it has a direct peering relationship to the original nonclient. The route reflector also sends the route to EBGP peers, which is standard behavior. IBGP routes get repeated to all EBGP peers.
Route Reflector Basics
This section briefly looks at how route advertisement works with route reflectors. This is illustrated in Figure 3-24.
)
Figure 3-24 Route Reflector Basics
If a route reflector receives a route from an EBGP peer, it passes that route to all route reflector clients and nonclients, just as in normal IBGP peering behavior.
If the route reflector receives a route from a route reflector client, it reflects the route to the other clients within the cluster and nonclients. It also reflects the route to EBGP peers. Here's another way to think of this: The route reflector takes over the communication for the route reflector clients, passing along all the messages they would normally transmit directly via a peering session.
Scaling IBGP with Confederations
BGP confederations are another way of scaling IBGP. Their behavior is defined in RFC 5065. Confederations insert information using the autonomous system path into BGP routes to prevent loops within an autonomous system. The basic idea with confederations is to divide a normal BGP autonomous system into multiple sub-autonomous systems. The outer or containing autonomous system is called the confederation autonomous system. This is all that is visible to the outside world.
Each of the inner autonomous systems is a smaller sub-autonomous system that uses a different autonomous system number, typically chosen from the private autonomous system number range of 64,512 through 65,534.
BGP Confederation Definitions
This section defines terms used with confederations (see Figure 3-25).
)
Figure 3-25 Confederation Definitions
Peers within the same sub-autonomous system are confederation internal peers.
IBGP peers that are in a different sub-autonomous system are confederation external peers.
As IBGP information is passed around within a confederation autonomous system, the sub-autonomous system numbers are put into a confederation sequence, which works like an autonomous system path.
Confederation Basics
Route advertisement with confederations works similarly to that of route reflectors in the following ways:
- A route learned from an EBGP peer is advertised to all confederation external and internal peers.
- A route learned from a confederation internal peer is advertised to all confederation external peers, and to EBGP peers.
- A route learned from a confederation external peer is advertised to all confederation internal peers, and to EBGP peers.
Another way to understand this is that IBGP between sub-autonomous systems acts like EBGP. Private autonomous system numbers are used internally within the confederation autonomous system and removed from updates sent outside the confederation.
Confederations Reduce Meshing
Like route reflectors, confederations are used to reduce the amount of IBGP meshing needed. Without route reflectors or confederation, IBGP requires a full mesh of peering relationships, as illustrated in Figure 3-26.
)
Figure 3-26 IBGP Full-Mesh Peering
However, confederations can reduce meshing requirements, as shown in Figure 3-27.
)
Figure 3-27 Confederations Reduce the Number of IBGP Peers
Routers in different sub-autonomous systems do not peer with each other, except at sub-autonomous system borders. It is generally recommended to use two or three links between sub-autonomous system borders. More links just consume CPU and memory in the border routers.
When you use sub-autonomous systems for confederations, the meshing is restricted to within the sub-autonomous systems, with some additional peering between sub-autonomous system border routers.
Route reflectors can be used within confederations to further reduce network complexity. Historically, service providers have not done this, but they are now starting to. Using route reflectors alleviates the need to fully mesh within a sub-autonomous system.
Deploying Confederations
In Figure 3-28, router B could be configured to set the BGP next hop to itself for advertisement to routers C and D. This is not normally done by IBGP routers. This would impose the constraint that routers C and D would need to have routes to the new next hop, router B.
)
Figure 3-28 Deploying Confederations
Using this configuration breaks the confederation up from a next-hop perspective from both the IGP and BGP point of view. This scenario allows for more flexibility and scaling in very large networks. This deployment might make sense for large organizations that support separate entities such as government organizations that have distinct branches or divisions.
Using confederation sub-autonomous systems has other advantages. The IBGP policies can differ internally within and between the sub-autonomous systems. In particular, multi-exit discriminator (MED) acceptance or stripping, local preference settings, route dampening, and so on can vary between sub-autonomous systems. In addition, policy controls can be used on peerings between sub-autonomous systems.
This highlights some advantages of confederations. Confederations can ease the transition in an acquisition or merger. The new network can be treated as another sub-autonomous system and keep its IGP. It can also keep its EBGP policies with its customers.
A disadvantage of confederations is that there is no graceful way to migrate from full mesh to using confederations. The migration may well require downtime.
Table 3-1 compares how confederations and route reflectors provide various IBGP scaling features.
Table 3-1. Comparing Confederations to Route Reflectors
|
Confederation |
Route Reflector |
|
|
Loop prevention |
Autonomous system confederation set |
Originator or set cluster ID |
|
Break up a single autonomous system |
Sub-autonomous systems |
Clusters |
|
Redundancy |
Multiple connections between sub-autonomous systems |
Client connects to several reflectors |
|
External connections |
Anywhere in the network |
Anywhere in the network |
|
Multilevel hierarchy |
Reflectors within sub-autonomous systems |
Hierarchical clusters |
|
Policy control |
Along outside borders and between sub-autonomous systems |
Along outside borders |
|
Scalability |
Medium; still requires full IBGP within each sub-autonomous system |
Very high |
|
Migration |
Very difficult (impossible in some situations) |
Moderately easy (impossible in some situations) |
In general, route reflectors are simpler to migrate to and relatively simple to use, whereas confederations are more flexible as to IGP and policy.