Skip to main content

Result Analysis

TLS-Anvil stores the test results in multiple json files. In addition the network traffic is captured during the execution. Since analyzing those files by hand is tedious, we created a small web application to get the job done.

The result analyzer application is also shipped as docker container. Since a database is required, docker-compose is the easiest way to start the application. The docker-compose.yml file is part the Anvil-Web GitHub Repo.

Download docker-compose.yml

Start the application​

First we start the web application.

docker-compose pull
docker-compose up -d

The application should be available at http://localhost:5001.

Importing the results​

Next the results need to be imported, i.e. importing the JSON files of TLS-Anvil into a MongoDB that is accessed by the backend of the web application. The easiest way to do that is to zip your results folder (the folder that contains the report.json file) and upload it in the web-interface. Just go to Tests -> Upload Test and select the zip file.

Using the application​

  1. Open your web browser at http://localhost:5001.
  2. Click Tests in the navbar (if not already selected) and look for the test you just uploaded. Here you can click on Details.
  3. You will see an overview over the testresults. The specific result for each test template ⓘ is presented in the table at the bottom, sorted by RFC.
  4. The table rows are clickable, if you do, a detailed view will be presented to you showing what exactly got tested in that run and the results for each test input ⓘ, i.e. each performed handshake.
    1. Click on a row of a test case to view the recorded PCAP dump for the handshake as well as additional information about the handshake. Parameter Combination, for example, shows the test input ⓘ for the test case, generated by the combinatorial testing algorithm.

Possible Test Results​

Strictly Succeeded (✅)
A strictly succeeded test means that the SUT ⓘ behaved exactly as expected. If multiple test cases ⓘ are performed during the execution of a test template ⓘ, the SUT ⓘ must have behaved correctly across all of them.

Conceptually Succeeded (⚠️✅)
A conceptually succeeded test means that an implementation did not precisely fulfill the RFC requirements or did not do so in all test cases ⓘ but effectively behaved correctly. This usually applies to tests where a fatal alert was expected, but the library either only closed the connection but did not send an alert, or the alert description did not match the RFC's specification.

Partially Failed (⚠️❌)
When multiple handshakes are performed for a test template ⓘ, the partially failed result indicates that not all test inputs ⓘ failed for a specific test template ⓘ.

Fully Failed (❌)
A fully failed result means that the SUT ⓘ did not behave correctly for any test input ⓘ.

Result Annotations​

Additional Information (❗️) In some cases, test templates highlight details that affected the result of a test. Tests with these details are indicated by an exclamation mark. When viewing the individual sessions of a test, the collected details are shown when hovering on a single result.

Contradictory Additional information (⁉️) If a test template added additional details for the result but these details differ between individual test cases of the template, the test result is annotated with ' ⁉️ '. This may hint towards unexpected behavioral differences. In the result view, you can filter for results with specific additional information.