Fortran - Testing - Towards a generic, testable test runner

Maybe we’ve been writing more “generic” code for our custom test program than expected. We have been gradually, and quite safely, extracting code from a simple “check program”. But now that we’re looking at a pretty generic test runner that we’re going to use for all our future unit tests, we inevitably have to address the concern that we’re writing code that isn’t tested itself. Who tests the test framework? Well, the test framework itself has all the ingredients. However, testing the framework requires that we can run its code in isolation: that we can call one of its functions, and assert that it behaves correctly. Unfortunately, that isn’t possible yet. The code still lives in the main block of the tester program, and it doesn’t allow in-memory inspection of the test results, because it only prints them to the terminal, which we can’t unit-test.

Breaking out of the main program

In order to unit-test any code, we should be able to call that code as a procedure (function or subroutine), which we can use inside a test module. Code in a program block can’t be used or called in other modules. Of course, we need some code in the program block, or the program wouldn’t do anything, but we should jump out of it and into a proper procedure as soon as possible. So the test loop that we have in our tester program should be extracted to the test_framework module. Let’s move the code into a new subroutine called run_test_suites. While doing so, we don’t want to transfer the knowledge of which actual test modules we use in our project to the test_framework module too. The framework should remain generic. From the test program we should pass the array of instantiated test suites to the run_test_suites subroutine:

module test_framework
   public :: run_test_suites
   
   ! ...

contains

   subroutine run_test_suites(test_suites)
      type(test_suite_t), dimension(:), intent(in) :: test_suites

      ! ...

      if (error_counter > 0) then
         print *, error_counter, 'tests failed'
         error stop
      end if
   end subroutine run_test_suites

end module test_framework

The test program is very clean now. We can even pass the array of test suites as the result of an expression, instead of assigning it to a local variable first:

program tester
   use test_polyline, only: collect_polyline_tests
   use test_framework, only: run_test_suites, &
                             new_test_suite

   implicit none(type, external)

   call run_test_suites([new_test_suite(collect_polyline_tests)])
end program tester

Making run_test_suites testable

If eventually we want to be able to test the test runner itself, the run_test_suites subroutine isn’t very helpful. We could call it from a test, but we don’t get any information on whether all the tests were executed, whether any of them failed, etc. In other words, no data is exposed to the caller. Also, any output is provided with print, so it ends up directly in the user’s terminal. We can’t inspect the output and verify its correctness. Finally, the subroutine stops the entire program if there was a failing test, and that makes it impossible for us to ever test this code. We’ll encounter these problems in a future post too, when we discuss working with legacy code (this code is also legacy code, or: code without tests). For now, we’ll just quickly work towards the solution.

First we make sure that the results of running the tests are returned by the run_test_suites, which has to become a function in order to that. To preserve the existing behavior we only need to return error_counter:

function run_test_suites(test_suites) result(error_counter)
   type(test_suite_t), dimension(:), intent(in) :: test_suites
   
   integer :: error_counter
   
   ! ...
end function run_test_suites

With the test results available to us, we can now extract a new subroutine that uses the error count to print the test results. This removes any lines from the run_test_suites function that print something:

subroutine handle_test_results(error_counter)
   integer, intent(in) :: error_counter

   if (error_counter > 0) then
      print *, error_counter, 'tests failed'
      error stop
   end if
end subroutine handle_test_results

The tester program now has two calls:

program tester
   use test_polyline, only: collect_polyline_tests
   use test_framework, only: run_test_suites, &
                             new_test_suite, &
                             handle_test_results

   implicit none(type, external)

   integer :: error_counter

   error_counter = run_test_suites([new_test_suite(collect_polyline_tests)])
   call handle_test_results(error_counter)
end program tester

What matters is that we can call run_test_suites separately from handle_test_results. But it would still be nice if the test program needs only one call statement. In particular because we will return more information from run_test_suites, like the total number of tests that were executed, and it would be great “developer experience” if we don’t have to change the code in the tester program anymore. There are several ways in which we can accomplish this, but the simplest is to make a new subroutine, run_test_suites_and_print_results, which combines the two statements that were previously in the test program:

subroutine run_test_suites_and_print_results(test_suites)
   type(test_suite_t), dimension(:), intent(in) :: test_suites

   integer :: error_counter

   error_counter = run_test_suites(test_suites)

   call handle_test_results(error_counter)
end subroutine run_test_suites_and_print_results

Collecting more test results

An added benefit of this new subroutine is that the test program doesn’t have to capture the test results in a variable, or pass it to the handle_test_results procedure, which it would have to remember to do. Essentially we’ve encapsulated the test results, so now we can easily upgrade the data type used. For instance, we can add a counter for the total number of tests, and the number of passed tests. Let’s collect them in a single derived type with a proper name:

type :: test_results_t
   integer :: all_tests
   integer :: passed_tests
   integer :: failed_tests
end type test_results_t

We should first upgrade run_test_suites to return a test_results_t instance. To make the existing code work, we increment the failed_tests data component instead of the error_counter variable:

-function run_test_suites(test_suites) result(error_counter)
+function run_test_suites(test_suites) result(test_results)
   type(test_suite_t), dimension(:), intent(in) :: test_suites

-  integer :: error_counter
+  type(test_results_t) :: test_results

   ! ...

         if (test_failed) then
-           error_counter = error_counter + 1
+           test_results%failed_tests = test_results%failed_tests + 1
         end if
   ! ...
end function run_test_suites

After updating the code in run_test_suites_and_print_results and handle_test_results, everything keeps working as it did before. Now we can populate the other data components of test_results_t as well:

test_failed = unit_tests(unit_test_index)%test_procedure()

test_results%all_tests = test_results%all_tests + 1
if (test_failed) then
   test_results%failed_tests = test_results%failed_tests + 1
else
   test_results%passed_tests = test_results%passed_tests + 1
end if

Finally, we can print the new information:

subroutine handle_test_results(test_results)
   type(test_results_t), intent(in) :: test_results

   print *, test_results%all_tests, ' test(s) executed, ', &
      test_results%passed_tests, ' test(s) passed'

   if (test_results%failed_tests > 0) then
      print *, test_results%failed_tests, 'test(s) failed'
      error stop
   end if
end subroutine handle_test_results

An example of the enhanced output:

2  test(s) executed,            1  test(s) passed
1 test(s) failed

The need may arise to influence or configure the exact output format here. We’d also like to remove the burden from tests to print their own descriptions to the terminal, like we do in our polyline tests:

function test_polyline_with_3_points() result(test_failed)
   logical :: test_failed

   real(kind=real64), dimension(3, 2) :: three_points

   print *, 'Calculate length of polyline with 3 points'

In the next post we’ll tackle this problem and redesign the output handling of our new test framework.

• Fortran
• Unit Testing