Shooting Trouble with IP

Routing Protocols

Routing protocols have their own unique characteristics, and various Layer 2 encapsulation types have a big impact on them. Understanding how they work will certainly assist you in troubleshooting them now and later. Most of the statistics listed for each can be found with IOS commands such as show ip protocols, show ip route, and show ip routing-protocol ?. Logging, debug, and protocol analyzers all with various levels of impact on the working environment are certainly valuable tools to see more detail when troubleshooting, too. However, I want you to reserve them for later chapters. After all, many of these protocols are not just IP-specific.

First I discuss the following Interior Gateway Protocols (IGPs): RIP, Interior Gateway Routing Protocol (IGRP)/Enhanced Interior Gateway Routing Protocol (EIGRP), OSPF, and IS-IS. Then I review Border Gateway Protocol (BGP), which is an Exterior Gateway Protocol (EGP).

RIP

Routing Information Protocol (RIP) was originally designed for Xerox PARC Universal Protocol (PUP), and in many ways is still for "pups." It was called GWINFO in the Xerox Network Systems (XNS) protocol suite in 1981, and defined in RFC 1058 in 1988. It is easy to configure, and it works very well in small networks. In larger networks, however, it can be less effective; as I say to myself, "It can RIP you apart." There are alternatives to RIP for larger environments.

Everyone knows RIP because it has been widely adopted by PC, UNIX, and router makers alike. RIP has disadvantages in that it operates over UDP port 520 and the maximum hop count is 15. RIPv2 assists with the broadcast nature in that it operates via multicast over 224.0.0.9. Both RIPv1 and RIPv2 are distance vector routing protocols, which are often referred to as routing by rumor protocols.

Examine the following list of RIP characteristics and refer back to the examples throughout the chapter, for you have already experimented with RIPv1 and RIPv2. I hope your practical exercises, with RIPv1 not supporting discontiguous subnets and RIPv2 supporting them, will stay with you for a long time.

The following are RIP characteristics:

• Open protocol, widely used, stable.

• Good for small networks in that it is very easy to configure.

• There are RIP-like distance vector routing protocols for Novell and AppleTalk.

• Distance vector routing protocol.

• IGP.

• IP RIP updates are sent every 30 seconds via broadcast (224.0.0.9 for RIPv2).

• UDP port 520

• Administrative distance is 120.

• Single metric is hop count. (The limit is 15 to assist with count-to-infinity.)

• Timers help regulate performance:

  • Update timer—Frequency of routing updates. Every 30 seconds IP RIP sends a complete copy of its routing table, subject to split horizon. (IPX RIP does this every 60 seconds.)

  • Invalid timer—Absence of refreshed content in a routing update. RIP waits 180 seconds to mark a route as invalid and immediately puts it into holddown.

  • Hold-down timers and triggered updates—Assist with stability of routes in the Cisco environment. Holddowns ensure that regular update messages do not inappropriately cause a routing loop. The router doesn't act on nonsuperior new information for a certain period of time. RIP's hold-down time is 180 seconds.

  • Flush timer—RIP waits an additional 240 seconds after holddown before it actually removes the route from the table.

• Other stability features to assist with routing loops include the following:

  • Split horizon—Not useful to send information about a route back in the direction from which it came.

  • Poison reverse—Updates that are sent to invalidate a route and place it in holddown.

• Bellman-Ford algorithm.

• RIPv2 supports VLSM and summarization. (RIPv1 doesn't.) RIPv2 always autosummarizes at the class boundary.

RIP maintains only the best route in its routing table, as you can verify in examples throughout the chapter. In Example 3-2, for example, I configured RIPv1 on r1 and continued to configure the other routers. In Example 3-15, while troubleshooting I looked at the routing tables and IP routing protocols. Then I configured RIPv2 in Example 3-21. Example 3-22 displays the routing tables and output of show ip protocols with RIPv2 configured. Look back at these examples to review such things as the update characteristics, timers, and administrative distance associated with RIP.

Example 3-35 illustrates routes in holddown. First I turn on service time stamps and set the clock so that you can see the actual timing of events. Then I turn on debug ip rip events, remove the cable from router 2 Serial 1, plug it back in, watch the routing updates, and view the appropriate routing tables.

Example 3-35 Debug Output for RIPv2 Packets

r1(config)#service timestamps debug datetime localtime
r1(config)#end
r1#clock set 3:22:00 11 October 2002
r1#copy running-config startup-config
Destination filename [startup-config]? 
Building configuration...
[OK]
r1#debug ip rip events
RIP event debugging is on
Oct 11 03:22:52: RIP: sending v2 update to 224.0.0.9 via Ethernet0 (192.168.1.1)
Oct 11 03:22:52: RIP: Update contains 8 routes
Oct 11 03:22:52: RIP: Update queued
Oct 11 03:22:52: RIP: sending v2 update to 224.0.0.9 via Ethernet1 (192.168.4.1)
Oct 11 03:22:52: RIP: Update sent via Ethernet0
Oct 11 03:22:52: RIP: Update contains 5 routes
Oct 11 03:22:52: RIP: Update queued
Oct 11 03:22:52: RIP: sending v2 update to 224.0.0.9 via Serial0 (10.1.1.1)
Oct 11 03:22:52: RIP: Update sent via Ethernet1
Oct 11 03:22:52: RIP: Update contains 8 routes!
Oct 11 03:22:52: RIP: Update queued
Oct 11 03:22:52: RIP: sending v2 update to 224.0.0.9 via Serial1 (192.168.2.1)
Oct 11 03:22:52: RIP: Update sent via Serial0
Oct 11 03:22:52: RIP: Update contains 4 routes
Oct 11 03:22:52: RIP: Update queued
Oct 11 03:22:52: RIP: Update sent via Serial1
Oct 11 03:22:59: RIP: received v2 update from 192.168.2.2 on Serial1
Oct 11 03:22:59: RIP: Update contains 5 routes
Oct 11 03:23:02: RIP: received v2 update from 192.168.4.2 on Ethernet1
Oct 11 03:23:02: RIP: Update contains 4 routes
r1#!!!now I will unplug the r2s1 cable
r1#
Oct 11 03:23:14: RIP: received v2 update from 192.168.2.2 on Serial1
Oct 11 03:23:14: RIP: Update contains 5 routes
Oct 11 03:23:14: RIP: received v2 update from 192.168.4.2 on Ethernet1
Oct 11 03:23:14: RIP: Update contains 4 routes
Oct 11 03:23:14: RIP: sending v2 update to 224.0.0.9 via Ethernet0 (192.168.1.1)
Oct 11 03:23:14: RIP: Update contains 8 routes
Oct 11 03:23:14: RIP: Update queued
...
r1#show ip route
...
C  192.168.4.0/24 is directly connected, Ethernet1
R  192.168.5.0/24 is possibly down, routing via 192.168.4.2, Ethernet1
   10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
R    10.2.2.0/24 [120/1] via 192.168.2.2, 00:00:10, Serial1
R    10.0.0.0/8 [120/2] via 192.168.4.2, 00:00:07, Ethernet1
C    10.1.1.0/24 is directly connected, Serial0
R  192.168.6.0/24 [120/1] via 192.168.4.2, 00:00:07, Ethernet1
          [120/1] via 192.168.2.2, 00:00:11, Serial1
C  192.168.1.0/24 is directly connected, Ethernet0
C  192.168.2.0/24 is directly connected, Serial1
R  192.168.3.0/24 [120/1] via 192.168.2.2, 00:00:11, Serial1

Plug the cable back in, continue to review the results, and turn off all debug activity, as in Example 3-36.

Example 3-36 Plug the Cable Back In and Observe the Results

r1#!!!now I will plug the cable back in
...
r1#show ip route
C  192.168.4.0/24 is directly connected, Ethernet1
R  192.168.5.0/24 is possibly down, routing via 192.168.4.2, Ethernet1
   10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
R    10.2.2.0/24 [120/1] via 192.168.2.2, 00:00:13, Serial1
...
Oct 11 03:26:16: RIP: received v2 update from 192.168.4.2 on Ethernet1
Oct 11 03:26:16: RIP: Update contains 4 routes
Oct 11 03:26:16: RIP: sending v2 update to 224.0.0.9 via Ethernet0 (192.168.1.1)
Oct 11 03:26:16: RIP: Update contains 8 routes
Oct 11 03:26:16: RIP: Update queued
Oct 11 03:26:16: RIP: sending v2 update to 224.0.0.9 via Ethernet1 (192.168.4.1)
Oct 11 03:26:16: RIP: Update sent via Ethernet0
Oct 11 03:26:16: RIP: Update contains 5 routes
Oct 11 03:26:16: RIP: Update queued
...
r1#show ip route
...
C  192.168.4.0/24 is directly connected, Ethernet1
R  192.168.5.0/24 [120/1] via 192.168.4.2, 00:00:04, Ethernet1
   10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
R    10.2.2.0/24 [120/1] via 192.168.2.2, 00:00:07, Serial1
R    10.0.0.0/8 [120/2] via 192.168.4.2, 00:00:04, Ethernet1
C    10.1.1.0/24 is directly connected, Serial0
R  192.168.6.0/24 [120/1] via 192.168.4.2, 00:00:04, Ethernet1
          [120/1] via 192.168.2.2, 00:00:07, Serial1
C  192.168.1.0/24 is directly connected, Ethernet0
C  192.168.2.0/24 is directly connected, Serial1
R  192.168.3.0/24 [120/1] via 192.168.2.2, 00:00:07, Serial1
r1#undebug all

Fix any issues before you continue. If you cannot see the debug output when you telnet in, you may need to turn on terminal monitor (term mon). In this example, I did not show the output of the interface status on r2, but that is pretty important in troubleshooting such issues. Repeat the example as necessary or log it for future reference.

See Figure 3-24 or look at one of your Sniffer traces to analyze the RIP packet format. Up to 25 destinations can be listed in a single packet. Next I discuss the Cisco proprietary routing protocols IGRP and EIGRP.

IGRP/EIGRP

Interior Gateway Routing Protocol (IGRP) was developed in the mid-1980s as a Cisco proprietary protocol to help overcome some of limitations of RIP, such as the single metric of hop count. It has stability features similar to RIP—hold-down timers, split horizon, poison reverse, and triggered updates. The timers are as follows: invalid 270 seconds, holddown 280 seconds, and flush 630 seconds. It also contains mechanisms to influence route selection and unequal load sharing. I use the phrase Big Dogs Really Like Meat to remember the metrics for IGRP and EIGRP:

• Bandwidth

• Delay

• Reliability

• Load

• MTU

IGRP is an IGP, a distance vector routing protocol based on the Bellman-Ford algorithm that broadcasts routing updates every 90 seconds over IP protocol number 9. It is fine for small and medium-size networks, but Cisco enhanced it greatly and added VLSM support to its replacement, EIGRP.

Cisco developed Enhanced IGRP (EIGRP) in the early 1990s to overcome limitations of RIP and its own IGRP. Cisco says IGRP is going to be removed from IOS. EIGRP is suitable for large networks today and supports multiple routed protocols. It consumes significantly less bandwidth because of its partial, bounded updates, and can be one of the fastest converging routing protocols there is.

The following are EIGRP characteristics:

• Cisco proprietary protocol.

• Good for small to large networks.

• Very easy to configure. Uses autonomous system (AS) number.

• Supports multiple Layer 3 routed protocol stacks, such as IP, Novell IPX, and AppleTalk.

• Advanced distance vector routing protocol. Often called a hybrid due to its incremental updates and rapid convergence capabilities.

• IGP.

• Multicast triggered updates over 224.0.0.10, not periodic.

• IP protocol number 88.

• Internal administrative distance is 90; external is 170.

• Metrics are bandwidth, delay, reliability, load, and MTU.

• Supports equal- and unequal-cost load sharing.

• Other stability features to assist with routing loops:

  • Split horizon—Not useful to send information about a route back in the direction from which it came

  • Poison reverse—Updates that are sent to remove a route and place it in holddown

• Uses Diffusing Update algorithm (DUAL) to select loop-free paths and give it very fast convergence.

• Supports VLSM and manual summarization (classless).

• Automatic classful boundary summarization.

• Manual summarization on update sent out each interface.

• Automatic redistribution with IGRP if same AS number.

• Route tagging for policy-based routing.

EIGRP gets its reliability from the Reliable Transport Protocol (RTP). It maintains not only a routing table, but also a neighbor and topology table. EIGRP maintains alternate routes referred to as successors (routing table) and feasible successors (topology table) to quickly converge. The following packet types are used for neighbor communications: hellos (multicast) and acks (unicast), update (multicast or unicast), query (multicast), reply (unicast), and request (multicast or unicast). Packets are held in a queue for retransmission, and there are separate neighbor tables (and entirely separate processes) for each protocol.

Cisco has some wonderful white papers on EIGRP that you should download and review to assist you with troubleshooting. Other popular references include the book EIGRP Network Design Solutions (Cisco Press). Just remember when troubleshooting EIGRP, active ain't good, but passive is. Active means that you are actively looking for something that you don't have. Pay particular attention to summarization when you are experiencing stuck-in-active situations. Besides the basic IOS commands in your repertoire, you should add show ip eigrp ? and debug ip eigrp ? for your EIGRP IP troubleshooting assistance. Next I very briefly review OSPF, IS-IS, and BGP. You will continue to configure, analyze, and troubleshoot the convergence of these protocols over various Layer 2 technologies throughout the rest of this book.

OSPF

Open Shortest Path First (OSPF) overcomes the disadvantages of RIP and is not proprietary in nature, but its openness supports only the IP routed protocol. The protocol is a link-state IGP based on the Dijkstra algorithm developed by the Internet Engineering Task Force (IETF) to support large heterogeneous networks. Lots of research was completed from 1987 until the current OSPFv2 specification in 1991. Link-state advertisements are sent to all, which causes an initial flood on the router; after that, however, OSPF is very efficient in operation. It uses three different databases (tables) for the neighbors, link states, and routes.

The following are OSPF characteristics:

• Open protocol.

• Good for small to large networks.

• Not as easy to design and configure as other protocols.

• Supports only the IP Layer 3 routed protocol stack.

• Link-state routing protocol (doesn't just send to neighbors like distance vector).

• IGP.

• Multicast link-state advertisement (LSA) updates over 224.0.0.5 and 224.0.0.6.

• IP protocol number 89.

• Administrative distance is 110.

• Metric is a cumulative cost (inversely proportional to bandwidth).

• Supports only equal-cost load sharing, but some implementations can take advantage of type of service (TOS) requests.

• Requires a routing hierarchy in that every area must touch the backbone area (otherwise temporary fixes such as virtual links are used). Various router types, LSA types, area types, and states, depending on your design and Layer 2 topology.

• Uses Dijkstra algorithm to select loop-free paths and give it fast convergence. This uses LSAs and is based on the Shortest Path First (SPF) algorithm, where the protocol got its name.

• Supports VLSM and summarization (classless).

• Supports manual summarization only; this is not automatic like EIGRP. Must be performed on an ABR (area range) or ASBR (summary address) only.

• Route tagging for policy-based routing.

Besides the basic IOS commands in your repertoire, you should add show ip ospf ? and debug ip ospf ? for your OSPF troubleshooting assistance.

OSPF references include the books OSPF Network Design Solutions (Cisco Press) and OSPF: Anatomy of an Internet Routing Protocol (Addison-Wesley). John Moy is the author of the latter, and if you truly want the RFC detail, this is the book to read. Next take a look at IS-IS.

IS-IS

The ISO was working on Intermediate System-to-Intermediate System (IS-IS) about the same time the IAB was working on OSPF. The late 1980s and early 1990s, a time in our history when everyone thought the OSI suite would overtake TCP/IP, was when Integrated IS-IS was proposed. Although originally designed for OSI routing, IS-IS was developed by ISO to support CLNS/CLNP. Integrated IS-IS, which supports IP, was a later development. The purpose was to provide a single routing protocol that could route Connectionless Network Protocol (CLNS) and IP. IS-IS is in use by ISPs today. OSPF and IS-IS have many common features.

The following are IS-IS characteristics:

• Open protocol.

• Good for medium to very large networks.

• ISO link-state routing protocol similar to OSPF.

• IGP.

• IS-IS Layer 2 PDUs rather than IP packets.

• Uses Layer 2 multicast.

• Administrative distance is 115.

• Very limited metric dynamic range (0–63).

• Equal-cost load sharing.

• Two-level hierarchical topology.

• Uses Dijkstra/SPF algorithm.

• Supports VLSM and summarization.

• Manual summarization.

• Route tagging for policy-based routing.

Besides the basic IOS commands in your repertoire, you should add show is-is ?, show clns ?, and debug is-is ? for your IS-IS troubleshooting assistance.

Jeff Doyle's Routing TCP/IP, Volume I does a good job of explaining many topics, including IS-IS. Volume II is great for BGP, as is Sam Halabi's Internet Routing Architectures.

BGP

Border Gateway Protocol (BGP) is an EGP that pretty much replaces the legacy EGP protocol itself. BGP performs routing between autonomous systems and is the standard routing protocol on the Internet. This is referred to as External BGP (EBGP), whereas when BGP is used to route within an AS it is referred to as Interior (IBGP). BGP is not a routing protocol for the fainthearted. It requires all manual configurations for a very good reason; you are not only affecting you, but you are also affecting me. For troubleshooting BGP, if routes are not in the BGP table, there is no way they will be in the routing table. Always make sure your neighbors are talking to you. One of the most useful commands in troubleshooting BGP is show ip bgp summary.

The following are BGP characteristics:

• Open protocol.

• Good for very large internetworks.

• Not as easy to design and configure as other protocols. Everything is manual, including neighbors (peers).

• Advanced distance vector or path vector routing protocol.

• EGP.

• TCP port 179.

• Internal administrative distance is 200; external is 20.

• Metrics include many attributes such as MED, Origin, AS-Path, Next-hop, and Community.

• Does not demand a particular routing hierarchy; roll your own.

• Automatic and manual summarization features.

• Route tagging for policy-based routing.

Besides the basic IOS commands in your repertoire, you should add show ip bgp ? and debug ip bgp ? for your BGP troubleshooting assistance. Remember that BGP is an EGP and that what you do affects others. ISPs often turn on dampening to account for those who really don't know what they are doing with BGP configuration or to compensate for link flaps. Too many changes within a certain period of time may mean that you don't get to communicate at all.

Individual routing protocols are books in themselves. However, I have quickly summarized the common routing protocols for you. The routing table is a good place to start troubleshooting, but if routes are missing, ultimately they may depend on Physical Layer or Data Link Layer issues, neighbor relationships, and topology or link-state tables. Cisco does a great job at categorizing more specific IP routing issues for you (see Figure 3-33).

Figure 3-33 IP Routing Top Issues

Now it is time for the Trouble Tickets. The plan here is to give you several things to do, let you make mistakes and fix some things on your own, and to introduce other problems that you should have some experience with as a support person. Shooting trouble with IP can easily be a multiple-volume set of books in itself. Because it is a big part of troubleshooting today, I also integrate more IP-related issues into the other chapters.