Verifications
A verification runs one or multiple commands to retrieve the current state of the device. The main purpose of a verification is to capture and compare the operational state before and after a trigger, or set of triggers, performs an action on a device. The state of the device can be retrieved via the multiple connection types offered by the pyATS library (CLI, YANG, and so on). Verifications typically run in conjunction with triggers to verify the trigger action did what it was supposed to do and to check for unexpected results, such as changes to a feature you didn’t initiate.
Verification Types
Verifications can be broken down into two types: global and local. The difference between the two is related to scoping and when each verification type runs within a script.
Global verifications: Global verifications are used to capture a snapshot of a device before a trigger is executed. Global verifications run immediately after the Common Setup section and before a trigger in a script. If more than one trigger is executed, subsequent snapshots are captured before and after each trigger using the same set of verifications.
Local verifications: Local verifications are independent of global verifications and run as subsections within a trigger. More specifically, a set of snapshots is taken before a trigger action and a subsequent set is taken after to compare to the first one. Local verifications confirm the trigger action did what it was supposed to do (configure/unconfigure, shut/no shut, and so on).
Figure 9-2 shows where and when the different verification types run in a Genie job.
)
Figure 9.2 Verification Execution
Verification Datafile
A verification datafile is used to customize the execution of built-in or custom verifications. Like other datafiles, verification_datafile.yaml must be provided to gRun using the verification_datafile argument. Example 9-7 shows a verification datafile that extends the default verification datafile (via the extends: key) and overrides the default setting of connecting to only the “uut” device (via the devices: key) for the Verify_Interfaces verification. If you wanted to change the list of devices to connect to for another verification, you would need to add that verification in the datafile.
Example 9-7 Verification Datafile – ex0908_verification_datafile.yaml
# Extend default verification datafile to inherit the required keys
# (class, source, etc.) per the verification datafile schema
extends: "%CALLABLE{genie.libs.sdk.genie_yamls.datafile(verification)}"
Verify_Interfaces:
devices: ["cat8k-rt1", "cat8k-rt2"]
In order to run the verifications listed in the datafile, or any other built-in verifications, you’ll need to include them in the verification_uids argument to gRun. Example 9-8 shows how to run the Verify_Interfaces verification with the verification datafile from Example 9-7.
Example 9-8 gRun – Verifications
from genie.harness.main import gRun
def main():
gRun(
verification_uids=["Verify_Interfaces"],
verification_datafile="ex0908_verification_datafile.yaml"
)
Writing a Verification
The process in which a feature is verified during testing may be different for different use cases. If a built-in verification does not suffice, the pyATS library (Genie) allows you to create your own verification.
There are several ways to create your own verification. You can use a Genie Ops feature (model), a parser, or callable. For the Genie Ops feature and parser options, you can use an existing model or parser, or you can create your own. The last option, using a callable, is discouraged, as it isn’t OS-agnostic and does not provide extensibility to use different management interfaces (CLI, YANG, and so on). Example 9-9 shows a custom verification built with the show bgp all parser. Take note of the list of excluded values from the parsed data (found under the exclude: key). The reason is because many of these values are dynamic (such as timers, counters, and so on) and are almost guaranteed to be different between snapshots. Remember, if a parsed value is different between snapshots, the verification will fail.
Example 9-9 Custom Verification – ex0910_verification_datafile.yaml
# Local verification datafile that already extends the default datafile
extends: verification_datafile.yaml
Verify_Bgp:
cmd:
class: show_bgp.ShowBgpAll
pkg: genie.libs.parser
context: cli
source:
class: genie.harness.base.Template
devices: ["cat8k-rt1", "cat8k-rt2"]
iteration:
attempt: 5
interval: 10
exclude:
- if_handle
- keepalives
- last_reset
- reset_reason
- foreign_port
- local_port
- msg_rcvd
- msg_sent
- up_down
- bgp_table_version
- routing_table_version
- tbl_ver
- table_version
- memory_usage
- updates
- mss
- total
- total_bytes
- up_time
- bgp_negotiated_keepalive_timers
- hold_time
- keepalive_interval
- sent
- received
- status_codes
- holdtime
- router_id
- connections_dropped
- connections_established
- advertised
- prefixes
- routes
- state_pfxrcd
To run the custom verification, you follow the same process as running any other verification. Pass the verification datafile with the custom verification to gRun via the verification_datafile argument and add the custom verification name to the list of verifications in the verification_uids argument. Example 9-10 shows the updated gRun call with the custom verification name and datafile. Remember, the custom verification datafile (Example 9-9) extends the original verification datafile (Example 9-7), which essentially inherits all the built-in verifications from the pyATS library (Genie).
Example 9-10 gRun – Custom Verification
from genie.harness.main import gRun
def main():
gRun(
verification_uids=["Verify_Interfaces", "Verify_Bgp"],
verification_datafile="ex0910_verification_datafile.yaml"
)