The Problem of Persistent Loops Due to Protocol Conflicts
If n routers are running OSPF and are connected in a full mesh over ATM PVCs, a single physical ATM link failure might result in ATM-layer rerouting of a large number of PVCs. If this takes too long, or if the ATM network cannot reroute PVCs, a large number of PVCs effectively fails.
The number of PVCs involved might be of the same order magnitude as n, and even n2 in some cases. In any case, it is likely to be seen by O(n) routers, where O(n) means "a number proportional to n." So, a single ATM link failure causes each of O(n) routers to send a link-state advertisement (LSA) of size (at least) O(n) to n 1 neighbors. Thus, a single event in the ATM network results in O(n3) to O(n4) traffic.
When a router receives an LSA, it must immediately recalculate its routing table because it must not forward packets based on old routing information. The processor load caused by a storm of routing updates might cause the routers to drop or not send keepalive packets. This appears to the neighboring routers as further link failures. These lead to further LSAs being sent, which perpetuates the problem.
The net result is that a full-mesh network can become persistently unstable after a single network event.
This critical failure occurs because the routers do not see the state of the ATM links and switches directly. IS-IS has somewhat better performance than OSPF in full-mesh conditions because IS-IS has more sophisticated flooding capabilities. (These capabilities, especially the ability to pace and block flooding on some interfaces, are also becoming available on OSPF.) However, this does not address the underlying problem.
The solution is to enable IP routing to directly see the state of ATM links, which is what ATM MPLS does.
MPLS addresses the fundamental problem underlying the instability of the full-mesh network: the basic conflict between routing protocols. PNNI routing at the ATM layer can make decisions that conflict with OSPF or similar routing at the IP layer. These conflicting decisions can lead to persistent loops. The only reliable solution to this problem is to use the same routing protocol at the IP layer and ATM layer. This is exactly what MPLS does in ATM networks.