Skip to main content

Result Analysis

TLS-Anvil stores the test results in multiple json files. In addition the network traffic is captured with tcpdump during the execution. Since analyzing those files by hand is tedious, we created a small web application to get the job done.

The result analyzer application is also shipped as docker container. Since a database is required, docker-compose is the easiest way to start the application. The docker-compose.yml file is part our GitHub Repo. Alternatively you can copy & paste it from here.

Result Analyzer Docker Compose File
version: '3.7'



image: mongo
restart: always
- mongodb_DB:/data/db
- mongodb_conf:/data/configdb
- 27017:27017
- app

image: mongo-express
restart: always
- 8081:8081
- app

restart: always
context: .
REST_API_BASE_URL: http://localhost:5000/api/v1
- 5000:5000
- app

Start the application

First we start the web application.

docker-compose pull
docker-compose up -d

The application should be available at http://localhost:5000.

Importing the results

Next the results need to be imported, i.e. importing the JSON files of TLS-Anvil into a MongoDB that is accessed by the backend of the web application. The uploader tool is also available as Docker image.

cd into any folder where child folders contain test results and run the uploader tool. It searches recursively for TLS-Anvil result reports and imports all of them into the database.

cd results
docker run \
--rm \
-it \
--network host \
-v $(pwd):/upload \

Using the application

  1. Open your web browser at http://localhost:5000.
  2. Click Analyzer in the navbar and select the uploaded report from the dropdown menu on the top left.
    • If the menu is empty, reload the page.
  3. The result for each test template ⓘ is presented in the table.
  4. The table rows are clickable and show the results for each test input ⓘ, i.e. each performed handshake.
    1. Click on a test result icon of a test case to view the recorded PCAP dump for the handshake as well as additional information about the handshake. DerivationContainer, for example, shows the test input ⓘ for the test case, generated by the combinatorial testing algorithm.
    2. Click on the table column head (first row) to see more information about the test template ⓘ, including the failure inducing combinations.

Possible Test Results

Strictly Succeeded (✅)
A strictly succeeded test means that the SUT ⓘ behaved exactly as expected. If multiple test cases ⓘ are performed during the execution of a test template ⓘ, the SUT ⓘ must have behaved correctly across all of them.

Conceptually Succeeded (⚠️✅)
A conceptually succeeded test means that an implementation did not precisely fulfill the RFC requirements or did not do so in all test cases ⓘ but effectively behaved correctly. This usually applies to tests where a fatal alert was expected, but the library either only closed the connection but did not send an alert, or the alert description did not match the RFC's specification.

Partially Failed (⚠️❌)
When multiple handshakes are performed for a test template ⓘ, the partially failed result indicates that not all test inputs ⓘ failed for a specific test template ⓘ.

Fully Failed (❌)
A fully failed result means that the SUT ⓘ did not behave correctly for any test input ⓘ.