Enterprise Network Testing: Testing Throughout the Network Lifecycle to Maximize Availability and Performance

• Copyright 2011
• Edition: 1st

• Book
• ISBN-10: 1-58714-127-2
• ISBN-13: 978-1-58714-127-0

Enterprise Network Testing

Testing Throughout the Network Lifecycle to Maximize Availability and Performance

Andy Sholomon, CCIE® No. 15179

Tom Kunath, CCIE No. 1679

The complete guide to using testing to reduce risk and downtime in advanced enterprise networks

Testing has become crucial to meeting enterprise expectations of near-zero network downtime. Enterprise Network Testing is the first comprehensive guide to all facets of enterprise network testing. Cisco enterprise consultants Andy Sholomon and Tom Kunath offer a complete blueprint and best-practice methodologies for testing any new network system, product, solution, or advanced technology.

Sholomon and Kunath begin by explaining why it is important to test and how network professionals can leverage structured system testing to meet specific business goals. Then, drawing on their extensive experience with enterprise clients, they present several detailed case studies. Through real-world examples, you learn how to test architectural “proofs of concept,” specific network features, network readiness for use, migration processes, security, and more.

Enterprise Network Testing contains easy-to-adapt reference test plans for branches, WANs/MANs, data centers, and campuses. The authors also offer specific guidance on testing many key network technologies, including MPLS/VPN, QoS, VoIP, video, IPsec VPNs, advanced routing (OSPF, EIGRP, BGP), and Data Center Fabrics.

§         Understand why, when, and how you should test your network

§         Use testing to discover critical network design flaws

§         Incorporate structured systems testing into enterprise architecture strategy

§         Utilize testing to improve decision-making throughout the network lifecycle

§         Develop an effective testing organization and lab facility

§         Choose and use test services providers

§         Scope, plan, and manage network test assignments

§         nLeverage the best commercial, free, and IOS test tools

§         Successfully execute test plans, including crucial low-level details

§         Minimize the equipment required to test large-scale networks

§         Identify gaps in network readiness

§         Validate and refine device configurations

§         Certify new hardware, operating systems, and software features

§         Test data center performance and scalability

§         Leverage test labs for hands-on technology training

This book is part of the Networking Technology Series from Cisco Press®, which offers networking professionals valuable information for constructing efficient networks, understanding new technologies, and building successful careers.

Online Sample Chapter

Crafting the Test Approach

Sample Pages

Download the sample pages (includes Chapter 4 and Index)

Table of Contents

Part I Introduction to Enterprise Network Testing

Chapter 1 A Business Case for Enterprise Network Testing 3

Why Testing Is Important 3

The Network as a Business Platform 4

The Cost of Network Downtime 5

Network Changes and Downtime 7

Testing in Support of Change Control 7

Testing and the Pursuit of “Five Nines” 9

A Structured Approach to Systems Testing 13

    Step 1: Assessment 13

    Step 2: Test Planning 13

    Step 3: Setup 14

    Step 4: Execution 14

    Step 5: Results 14

Summary 15

Chapter 2 Testing Throughout the Network Lifecycle 17

Enterprise and Network Architecture Primer 17

How the Enterprise Architecture Comes Together 18

Following a Convergence Vision 19

The Cisco Lifecycle Services Approach (PPDIOO) 21

    PPDIOO Phase 1: Prepare 21

    PPDIOO Phase 2: Plan 21

    PPDIOO Phase 3: Design 22

    PPDIOO Phase 4: Implement 22

    PPDIOO Phase 5: Operate 22

    PPDIOO Phase 6: Optimize 22

Testing and the Network Lifecycle 24

    Prepare Phase: Design and Test Activities 24

        Customer Requirements Document 24

        Network Architectural Strategy Development 25

        Business Case Document 25

        Network Testing and Lab Strategy Development 25

        Facilities Readiness Assessments 26

    Plan Phase: Design and Test Activities 27

        Architecture Design Workshops 27

        Current Architectural Assessment 27

        High-Level Design 28

        Proof of Concept Testing 28

        Network Readiness Testing 28

        Network Capacity Planning and Testing 29

    Design Phase: Design and Test Activities 29

        Low-Level Design 29

        Migration Plan 30

        Design Verification Testing 30

        Migration Plan Testing 31

    Implement Phase: Deliverables and Test Activities 31

        Network Implementation Plan 31

        Network Ready for Use Test 32

    Operate Phase: Deliverables and Test Activities 32

        Hands-On Lab Training 32

        Re-creation of Network Problems 32

    Optimize Phase: Deliverables and Test Activities 33

        Predeployment Testing for Minor Design Changes 33

        Software Acceptance Testing 33

Summary 34

Chapter 3 Testing and Lab Strategy Development 35

    Cost Analysis and Resource Planning 36

    Estimating CAPEX Necessary to Create a New Test Lab 36

        Environmental Considerations 36

    Estimated OPEX to Operate a Test Lab 44

        Staffing 44

        Power 44

        Physical Facility 45

        Maintenance Obligations 45

        Other OPEX 46

Test Organization Financing Models 46

    Cost of Business 46

    Project-Based Funding 47

    Departmental Chargeback 47

    Testing as a Business Function 47

    Return on Investment 47

Outsourced Testing 48

Test Lab Facilities Design 49

    Functional Lab Design: Selecting the Hardware and Software 49

    Physical Design 50

        Equipment Cabinet Floor Plan Layout 53

Test Lab Operations 56

    Test Organization Charter 56

    Team Roles and Responsibilities 57

    Management Systems 58

        Equipment Inventory System 58

        Equipment Scheduling/Lab Checkout Tool 58

        Team Website 58

        Other Operational Considerations 59

Summary 59

Chapter 4 Crafting the Test Approach 61

Motivations for Different Types of Testing 62

    Proof of Concept Testing 62

    Network Readiness Testing 63

    Design Verification Testing 63

    Hardware Certification Testing 63

    Network Operating System Testing 64

    Migration Plan Testing 64

    Network Ready for Use Testing 65

Test Scoping 66

    Step 1: Categorize the Type of Test to Be Completed 67

    Step 2: Identify Project Stakeholders 67

    Step 3: Identify Indicators of Test Success 68

        Network Design Verification Test 68

        Network Ready for Use Test 68

    Step 4: Estimate the Resources Required to Complete the Test 69

    Step 5: Identify Risks 70

    Step 6: Identify the Timeline for Completion 70

Test Planning 71

    Design the Functional Prototype Network System 71

    Constructing a High-Level Lab Topology Diagram 72

    Identifying the Test Suites and Test Cases 74

Choosing the Right Test Tools 75

    Stateless Packet Generators (Bit Blasters) 76

        Interfaces 76

        Tool Power/Capacity 76

        Packet/Traffic Manipulation 77

        Results 78

        Automation 78

        When to Use Stateless Packet Generators 78

        Packet Generator Vendors 79

    Stateful Packet Generators (Application Simulators) 79

        Stateful Generation Tool Vendors 80

        Results Reporting 80

        When to Use Stateful Packet Generators 80

    Network Delay and Impairment Tools 81

        Delay 81

        Impairment 81

    Network Modeling and Emulation Tools 82

        Network Modeling Tools 82

        Network Modeling Tool Vendors 82

    Application Simulation Tools 83

    Security Testing Tools 84

    Network Protocol Analysis Tools 86

Writing the Test Plan 86

    Overall Project Scope and Objectives 86

    Test Objectives and Success Criteria 87

    Test Resources Required 88

    Test Schedule 90

    Developing the Detailed Test Cases 91

    Understanding System Test Execution Methodologies 92

        Conformance Testing 92

        Functional and Interoperability Testing 93

        Performance and Scalability Testing 94

    Format for Written Test Case 94

Summary 95

Chapter 5 Executing the Test Plan 97

Building and Operating the Functional Network Prototype System 98

    Equipment Allocation and Connectivity 98

    Test Lab Telemetry 100

The Test Engineer’s Toolkit 103

    Understanding Your Test Tools: Quirks and Limitations 104

    Understanding the Different Types of Test Traffic 105

        RFCs Pertaining to Test Execution 108

    Tools to Execute Complex Testing 110

        Scale Testing: Simulating Large Networks with Limited Devices 110

    High-Availability Testing: How to Measure Convergence Times 121

        Convergence Testing: How to Trigger a Failover 123

    Testing Using Delay, Jitter, and Errors 123

    Using Cisco IOS Test Tools 124

        Chargen Service 124

        Cisco IOS IP Service-Level Agreements 125

    Embedded Event Manager Scripting 129

        EEM Monitored Events 130

        EEM Actions 131

    Using Customized Scripts 132

Test Execution 136

    Before You Begin 136

    Order of Testing: Getting Organized 137

Running the Test Cases 139

Capturing and Saving Results 142

    Organizing the Capture Files 143

    Router Configuration Files 144

    Data Archival 144

Summary 145

Part II Case Studies

Chapter 6 Proof of Concept Testing Case Study 149

Background for the Proof of Concept Testing Case Study 149

    Proposed Data Center Architecture 150

    Compute Infrastructure 151

    Storage Infrastructure 152

    LAN Infrastructure 152

    WAN Infrastructure 153

    Virtualization Software 153

    Risks of Deploying the Proposed Solution 153

Proof of Concept Test Strategy 154

    POC Test Objectives 154

    POC Test Topology 154

    Proof of Concept Test Scope 156

        Network Baseline Test 156

        Application Baseline Test 156

        Network and Application Integrity Test 157

        Failure/Recovery Test 157

        Feature Validation Tests 157

        Automation Validation Test 157

        Performance/Scalability/Capacity Test 157

    Summary of POC Test Cases 158

Summary 162

Chapter 7 Network Readiness Testing Case Study 163

Background for the Network Readiness Testing Case Study 163

    Legacy Network Infrastructure Overview 164

    Cisco Unified Communications Proposed Solution 164

    Risks Associated with Implementing the Proposed Solution 165

Network Readiness Assessment Approach and Findings 166

    Network Readiness Assessment 166

        Hierarchy and Modularity 166

        Utilization and Redundancy 167

        Access Layer Links 168

        IP Routing 169

        QoS 169

    Network Path Analysis 170

        Details of Network Path Analysis Testing 171

        Summary of Recommendations 173

Summary 174

Chapter 8 Design Verification Testing Case Study 175

Background for the Design Verification Testing Case Study 176

High-Level Design for Blue Ridge University MPLS Backbone 177

Low-Level Design for Blue Ridge University MPLS Backbone 178

    Risks of Deploying the Proposed Solution 182

Low-Level Design Verification Test Strategy 182

    Test Objectives 182

    Test Topology 183

    Design Verification Test Scope 184

        Network Baseline Test 184

        Feature/Functionality Tests 184

        Negative/Destructive Tests 185

        Performance/Scalability Tests 185

        Operations/Duty Cycle Tests 185

    Summary of Design Verification Test Cases 185

Summary 190

Chapter 9 Migration Plan Testing Case Study 191

Background for the Migration Plan Testing Case Study 192

Legacy and New Network Design Overview 192

New Backbone Design 194

End-State Network Design 194

High-Level Network Migration Plan 197

Migration Test Plan 198

    Summary of Migration Plan Testing 199

Summary 201

Chapter 10 New Platform and Code Certification Case Study 203

Background for the New Platform and Code Certification Case Study 204

Proposed Top-of-Rack Architecture 205

Hardware for the New Infrastructure 207

Platform and Code Certification Test Plan 210

    New Platform Certification Objectives 210

    New Software Certification Objectives 210

    New Platform and Code Certification Test Topology 211

    New Platform and Code Certification Test Scope 212

        Network and SAN Baseline Tests 212

        Management Functionality Test 212

        Failure/Recovery Test 213

        Feature Validation Test 213

        Performance/Scalability/Capacity Tests 213

    Summary of New Platform and Code Certification Test Cases 213

Summary 217

End Notes 217

Chapter 11 Network Ready for Use Testing Case Study 219

Background for the NRFU Case Study 220

Sports and Entertainment Stadium Network Architecture 221

Network Topology 224

    Physical Network Topology 225

        Core Layer Components 225

        Distribution Layer Components 225

        Access Layer Components 226

    Multicast Architecture 226

        Stadium HD Video 227

        General IP Multicast Topology 228

    Additional Infrastructure Considerations 230

Network Ready for Use Test Strategy 230

    Success Criteria 230

    Test Prerequisites 231

    Test Phases 231

    Test Tools 232

    Summary of NRFU Test Cases 232

Summary 240

Part III Test Plans

Chapter 12 Inter-Organization Secure Data Center Interconnect: Firewall Test Plan 249

Background 249

    Physical and Logical Test Topology 250

    Test Objectives 251

Test Case Summary 251

Detailed Test Cases 252

Chapter 13 Site-to-Site IPsec Virtual Private Networking: DMVPN and GET VPN Test Plans 273

Background 274

    Physical and Logical Test Topology 274

    Test Objectives 279

DMVPN Test Cases Summary 279

Detailed DMVPN Test Cases 280

GET VPN Test Cases Summary 302

Detailed GET VPN Test Cases 302

Chapter 14 Data Center 3.0 Architecture: Nexus Platform Feature and Performance Test Plan 323

Background 324

Physical and Logical Test Topology 325

    Test Objectives 328

    Traffic Flows for All Tests 328

Test Case Summary 328

Detailed Test Cases 329

End Note 356

Chapter 15 IPv6 Functionality Test Plan 357

The IPv6 Specification 357

Considerations for IPv6 Testing 358

    IPv6 Header Format 358

        IPv6 Address Scopes 359

        IPv6 Extension Headers 361

    IPv6 Source Address Selection 362

    ICMPv6 363

        IPv6 Neighbor Discovery 363

        IPv6 Autoconfiguration 364

        IPv6 PMTUD 365

    IPv6 Security 365

Physical and Logical Test Topology 366

    Test Objectives 368

Test Case Summary 368

Detailed Test Cases 368

End Notes 382

Chapter 16 MPLS/VPN: Scalability and Convergence Test Plan 383

Background 384

    Physical and Logical Test Topology 386

    Technical Details of the Test Topology 387

    Emulated Control Plane Scale 388

    Control Plane Scale Methodology 389

    Test Objectives 389

Test Case Summary 390

Detailed Test Cases 391

Chapter 17 WAN and Application Optimization: Performance Routing and Wide Area Application Services Test Plan 433

Background 434

    Physical and Logical Test Topology 434

    Test Traffic 438

    Test Objectives 440

Test Case Summary 440

Detailed Test Cases 441

Chapter 18 Using the Lab for Hands-on Technology Training: Data Center 3.0 Configuration Lab Guide 487

Background 488

    Physical and Logical Lab Topology 489

    Lab Objectives 490

Detailed Hands-on Lab 490

    Step 1: Log In to Your Assigned Pod 490

    Lab 1: Configuring Unified Computing System Ethernet Ports and Named VLANs Using Unified Computing System Manager 490

        Step 1: Launch UCSM from a Web Browser 493

        Step 2: Enable the Server Ports Between the UCS 6100 Fabric Interconnect and the UCS Chassis 493

        Step 3: Enable the Uplink Ports Between the UCS 6100 Fabric Interconnect and the Nexus 7000 Switches 496

        Step 4: Configure Named VLANs on the UCS 498

    Lab 2: Configuring UCS Network and Server-Related Pools 500

        Step 1: Configure an IP Pool for External Blade Management 501

        Step 2: Create a MAC Address Pool for the UCS 503

    Lab 3: Creating Virtual PortChannels on the Nexus 7000 Series Switches 505

        Virtual Device Context Overview 505

        Virtual PortChannel Overview 506

        vPC Terminology 507

        Step 1: Create VLANs on the Nexus 7000s 507

        Step 2: Create a vPC on the Nexus 7000s for Connectivity to Your UCS Chassis 509

        Step 3: Create a 40-Gbps PortChannel on the UCS 6100 Fabric Interconnect for Connectivity to the Nexus 7000 Pair 517

        Step 4: Verify PortChannel and vPC on the Nexus 7000 519

    Lab 4: Creating a VSAN and Enabling Fibre Channel Connectivity Between the UCS 6100 Fabric Interconnect and MDS 9506 521

        Terminology 521

        Step 1: Enable NPIV Mode, Create a VSAN, and Associate the Fibre Channel Ports of the MDS to the New VSAN 523

        Step 2: Create a New VSAN on the UCS 525

        Step 3: Associate Fibre Channel Interfaces with the UCS VSAN 526

    Lab 5: Configuring UCS Service Profiles 526

        Terminology for Service Profiles 528

        Step 1: Create a vNIC Template 529

        Step 2: Create a SAN Pool and vHBA Template 531

        Step 3: Configure Server Boot Policies (SAN and LAN) 534

        Step 4: Create an IPMI Profile 538

        Step 5: Create a Local Disk Configuration Policy 539

        Step 6: Create a Serial over LAN Policy 540

        Step 7: Create a UUID Suffix Pool 540

        Step 8: Create a Server Pool 542

        Step 9: Create a Service Profile Template 543

        Step 10: Create Service Profiles from a Service Profile Template 552

        Step 11: Clone and Manually Associate a Service Profile 554

    Lab 6: Configuring SAN Zoning and Core Switch Connectivity on the MDS 9506 556

        Step 1: Record UCS Service Profile WWPN Assignments 557

        Step 2: Create a Zone for each Service Profile on the MDS 559

        Step 3: Place the Zones in a Zoneset for Your POD/VSAN 901 561

        Step 4: Activate the Zoneset on the MDS 562

        Step 5: Configure MDS Connectivity to the Core SAN 562

    Lab 7: Enabling IP and Routing Features on the Nexus 7000 Series Switches 564

        Step 1: Configure Layer 3 VLAN Interfaces with IPv4 Addressing 565

        Step 2: Configure Hot Standby Router Protocol 567

        Step 3: Configure OSPF Routing on Core and VLAN Interfaces 570

        Step 4: Enable OSPF Routing on the VLAN Interfaces 572

        Step 5: Add a Redundant Path to the Core—Add OSPF Adjacency Between Nexus 7000s Across the PortChannel Trunk 573

    Lab 8: Verifying the Blade Servers Boot VMware ESX 4.0 576

        Step 1: Connect to Server KVM Console and Verify Boot Status 576

        Step 2: Verify ESX Service Console IP Connectivity 578

    Lab 9: Adding the UCS Blade Servers into VMware vCenter 580

9781587141270   TOC   3/16/2011

Submit Errata