The following post is based on collaborative research of Andriy Veselovsky, Vsevolod Pavlenko, and Olesia Tsvigun.
Compliance testing for network solutions – Why bother?
One of the critical aspects of networking equipment is interoperability with other vendors. Actually, interoperability is the final stage of ensuring that your equipment will not break your customer’s network harmony.
It’s logical to assume that if interoperability testing exists, interoperability problems also exist. But if product development is based on common telecom standards, why do we need interop testing at all?
Standards do exist, and they define product behavior. Surely, standards provide almost all necessary information to build your product. However, there is still room for variations in implementation of the communication protocols. Therefore, the first step to avoiding interop issues in your equipment should be compliance to standard tests.
Having spent loads of time doing interoperability testing, trying to find out the root cause of issues emerging in communication between different vendors’ equipment, we know for sure that compliance testing results are vital for knowing “where to dig”, so to speak.
Surely, compliance testing does not give 100% warranty that your product will not have interop issues, but it gives you confidence that you performed protocol implementation correctly.
As a small example, let’s play with LACP on some branded equipment.
About LACP in brief
Link Aggregation Control Protocol (LACP) in accordance with IEEE 802.1AX-2008(formerly IEEE 802.3ad) is used for connecting one or more physical links to a single Link Aggregation Group, as if it were a single link. LACP can be used by Internet providers, Enterprise, Datacenter, and in networking solutions for small and medium business.
What LACP provides:
- Network redundancy via load-balancing
- Increased bandwith, failover, load sharing
- Automatic traffic load-balansing in case of one link failure.
- More info about LACP advantages in brief presentations:
Test environment
The test setup configuration we used looks as follows:
Test Node:
- AMD Athlon(tm) 64 X2 Dual Core with 4GB of RAM
- OS: Ubuntu 12.04.5 LTS 3.11.0-26-generic
Test Framework:
- PyTest 2.6.1
Packet capture tools
- WireShark 1.6.7
- Scapy 2.2.0
- Dpkt 1.8
Traffic generator:
- Scapy 2.2.0
- Dpkt 1.8
- Ostinato 0.6
Device Under Test (DUT):
- Cisco SG-300 Small Business series switch
LACP test suite details
LACP compliance test suite is purposed for conducting conformance testing with IEEE Link Aggregation Control Protocol (LACP) (802.1AX-2008) standard.
Here is a brief description of key features of the test suite:
- Full version of test suite allows to make network equipment compliance test to LACP protocol (802.1AX-2008)
- Test cases were created based on standard and industry experience
- All test cases are, easy to work with, and can be used in different frameworks> or programing languages for automation
- Test cases are universal and can be utilized for compliance testing with standard equipment of various vendors
- 64% of a standard already covered by test suite
- Results report provides an easy way for test results analysis
- Integration with kuTEST-Net framework will provide all kuTEST-Net benefits in the nearest future.
Total test cases for now is 105 and it is near 64% coverage of LACP 802.xxx standard. PLVision is currently working on extending the test suite to achieve 100% standard coverage.
Standard coverage by the test suite
Here is the list of 802.1AX-2008 paragraphs covered by test suite with a couple of high level test cases examples.
Link Aggregation Control (Paragraph 5.3)
- System Identification
- Port Identification
- Link Aggregation Group identification and seleting
- Attaching and detaching link to an Aggregator
- Enabling Collection And Distribution
LACP (Paragraph 5.4)
- LACPDU structure and encoding
Configuration capabilities and restrictions (Paragraph 5.6)
- Use of system and port priorities
- Selection Logic variants
What’s with the failed tests?
The diagram above shows that there are 12% skipped and 15% failed tests. You must be wondering – are these guys trying to tell us there is something wrong with the industry leaders’ equipment? Surely not. Some features are simply not supported or not represented in the DUT due to implementation specific, which actually produced such results:
- Limited implementation of LACP functionality
Device under test is Cisco SG-300 Small Business series switch, and it does not have a separate Aggregator instance – this does not affect LAG functionality, but causes skipped test cases.
- Variations in standard implementation
The standard specifies various ways of switching to Collecting and Distributing states , and the the test cases cover all of the described variants; however, the DUT’s Collecting and Distributing states are enabled only simultaneously. Again, this doesn’t indicate a test fail, but some of the test cases need to be skipped to avoid “false” fails.
- Vendor-specific defaults
One of the reasons for failed test cases is that this Cisco provides by default Enabled states for Synchronization, Collecting, Distributing states, but according to standard it might be assumed that all these states are to be Disabled by default.
- The DUT doesn’t support Marker protocol, Churn detection machine, and LACP passive mode, so we view these features as not supported, but not failed either.
In conclusion
Regardless of the fact that 15% of tests failed, we can’t say that this Cisco is working incorrectly. These fails dot not interfere with basic LACP functionality. The reason is that there are some restrictions, some personal vision, and vendor specific implementation, but in general it definitely will not break your existing network.
Analyzing existing fails and restrictions, it looks like we shouldn’t expect any problems with other device interoperability in case other devices also pass the compliance tests with such good results.
References:
