Pre-deployment validation topic

[gusmb]

Just like the eos_validate_state performs live checks in the network based on the structured data that was generated, would be nice to have an Arista solution for pre-deployment validation based purely on the generated EOS configs (or structured data), without the need for huge infrastructure to replilcate prod state. In a similar way Batfish does, which can compute the data plane based entirely on input configs, and validate the whole topology, would be nice to have the ability to run some data plane tests (reachability, VLAN/VXLAN, BGP neighbors, ISIS, OSPF, etc), based on the generated structured_data. In fact this could be a batfish plugin, that assumes the customer has a batfish server installation and the goal would be to do process batfish queries to run topology tests, basic teachability tests, look for incompatible no sessions or wrong routing policies etc, all based on the intended configuration snapshot generated by AVD.
I'm sure many have the same requirements, but not an easy way to tackle it. Such capability could add a lot of value to the Automation toolchain.

[ClausHolbechArista]

There have been attempts on this before but we don't have an updated/maintained plugin. My main concern would be feature/design coverage. Batfish can only emulate a certain subset of the configs, so it is not a full test.
We have been looking at prevalidation by spawning a virtual network and performing tests on that. The coverage would be better (not 100% either) but there is also more moving parts.

[gusmb]

I don't think the use case here is full testing, but more like essential config plane and data plane tests that would save time by detecting inconsistencies early in the change management process. Examples:

• baseline configs match the expected
• all underlay links are up and compatible
• underlay and overlay bgp Sessions established
• all loopbacks reachable
• no duplicate ip addresses in the topology
• no undefined references (e.g. no undefined route maps referenced in s bgp session)
• MLAG configuration consistent
• all devices have the management IP defined

These are just a few that batfish can easily do and what typically is found only when the devices (or Cloudvision) rejects the configuration. So rather than full testing I would say it is config validation to ensure the devices will accept the config and basic connectivity will be consistently established. Post-deployment checks on a working topology allow for more complete set of tests and detect scenarios where a rollback is required...

Testing with a virtual lab is an issue when dealing with hw platforms differences, interface namings,
features supported, and a large number of configuration files (more than 200) to test...

[ClausHolbechArista]

Sounds like a realistic ambition, and I agree this would be a valuable addition.

[perimore]

Any more thoughts about this? I have seen more interest in being able to pre-validate configuration in AVD, possibly by integrating with Batfish.

How about we build on top of Batfish and define our own tests?

[gusmb]

Meanwhile Batfish has been improving a lot in the recent versions. There are now more things supported on BGP that previously didn't work well, such as dynamic BGP peering, typically used on EVPN underlay.
IMO, this could be functionality outside of AVD, since the tests only need the collection of CLI configs as an input, but it would be indeed very helpful for pre-deployment validation. ANTA and AVD cover very well NRFU and post-deployment validation