ETSI GS NFV-TST 010 (TST010) is a published API conformance testing specification for NFV Management and Orchestration (NFV-MANO) APIs. Specifically, it contains conformance tests for the APIs used on the following reference points:

• Os-Ma-Nfvo, defined by ETSI GS NFV-SOL 005 (SOL005)
• Ve-Vnfm, defined by ETSI GS NFV-SOL 002 (SOL002)
• Or-Vnfm, defined by ETSI GS NFV-SOL 003 (SOL003)

The latest released version of TST010 is Version 2.6.1 (available from the ETSI website), which means that it supports the corresponding 2.6.1 versions of the above SOL documents (i.e. SOL02, SOL003 and SOL005). Version 2.4.1 is also available, and it similarly corresponds to the 2.4.1 versions of the SOL documents. This will always be the case going forward as well: the TST010 version will always match the corresponding version of the SOL documents specifying the reference points being tested.

The unique aspect about TST010 that sets it apart from other documents produced by the ETSI NFV TST Working Group (Testing, Experimentation and Open Source) is that it is not only a document: it also includes fully implemented test cases that are freely available to be used by anyone who wants to. In fact, the methodology used to build the TST010 conformance test package was to build the test cases in code first, and then extract the appropriate information from the code to produce the test descriptions in the TST010 document. In this way, we have implemented a single source of truth (the test code), which will always be in sync with the document.

The test tool used to implement the TST010 test cases is the Robot Framework (https://robotframework.org). It is an open source test automation framework that is released under the Apache 2.0 license. The framework is supported by an active community, and it is extensible with libraries and tools. Since it is also widely used by other open source communities, industry adoption is a plus for using Robot for the purposes of TST010. All TST010 test cases are implemented using the Robot Framework.

The implemented test cases are available on the ETSI Forge: https://forge.etsi.org/rep/nfv/api-tests. The test suite is provided under BSD-3-Clause License. The test suite is divided into three sections: SOL002, SOL003 and SOL005. For each of the reference point sections, the test cases are further classified for each API tested by the test cases. For example, the SOL005 section has sub-sections for the Package Management API, the Performance Management API, and so on.

The test suite has been exercised and validated against commercial NFV-MANO products in order to ensure a high level of quality of the implemented test cases. This has happened during the last two ETSI NFV Plugtests^TM. Specifically, the 4th NFV Plugtests (report available on the ETSI Portal) and the ETSI NFV & MEC Plugtests (report available on the ETSI Portal) both helped to validate the TST010 API conformance test suite.

For people who like to browse the test cases in a document rather than in a code repository, this is also available. When obtaining the TST010 document from the ETSI website, make sure to download the zip file that’s also available next to the PDF version of the TST010 document. The zip file contains 3 documents, outlining all of the test descriptions implemented, again on a per SOL document basis: SOL002, SOL003 and SOL005. These documents, which are annexes to the main TST010 document, are automatically generated from the code that implements the test cases themselves (i.e. from the Robot code). The test descriptions contain vital high-level information for each test case, including test ID and title, the objective, any pre-conditions, and post-conditions.

The TST010 document itself contains more general information about the test suite. It shows which test configurations are used throughout the test cases (clause 4.3), as well as information such as the generic test description format, the scope of the tests, and the verification done by the test suite.

TST010 is a constant and evolving activity within the ETSI NFV TST working group. Current and future activities and enhancements include:

• Version 2.7.1 of the SOL002, SOL003 and SOL005 test suites
• Implementation of test cases for versions 3.3.1 of the above SOL documents
• Implementation of ETSI GS NFV-SOL 009 (v3.3.1) conformance test cases (RESTful protocols specification for the management of NFV-MANO)
• Implementation of ETSI GS NFV-SOL 011 (v3.3.1) conformance test cases (RESTful protocols specification for the Or-Or Reference Point)
• Implementation of ETSI GS NFV-SOL 012 (v3.3.1) conformance test cases (RESTful protocols specification for the Policy Management Interface)
• Implementation of reference VNFDs and NSDs for an improved ability to test conformance based on actual packages

Version 2.7.1 of the Test Suites will be validated during the upcoming NFV&MEC API Plugtests 2021, that will be held remotely from 1 to 28 February 2021. Registration is free of charge and open to any organization willing to test its implementation.

Be aware that draft versions of the TST010 document and associated test cases are available from the ETSI open document server. These draft documents will constantly be updated as we develop future versions of the document. The draft Robot test cases are also available by following the links provided in the draft documents.

All figures © ETSI