Fortran - Testing - Showing progress and printing results

Matthias Noback

In the previous post we’ve successfully split the running of the tests and collecting the test results from handling those results by printing the counters and error stop-ping the test program. We still ask each test procedure to print a small description. We do this to assist the programmer when there is some kind of crash during the test run: they need to be able to find out which test caused it.

We already concluded that we shouldn’t have any normal output during the test run: the test runner itself should be fully in control of any output, or the terminal screen will become a mess. An additional benefit of centralized output is that if there is only one print statement left which is managed by the framework, we can change the output formatting for all tests at once. The easiest way to accomplish this is to define a few subroutines that should be called at the right moments during the test run:

subroutine print_test_name(test_name)
   character(len=*), intent(in) :: test_name

   write (stdout, '(A,A)', advance='no') test_name, '... '
end subroutine print_test_name

subroutine print_test_result_passed()
   write (stdout, '(A)') 'PASS'
end subroutine print_test_result_passed

subroutine print_test_result_failed()
   write (stdout, '(A)') 'FAIL'
end subroutine print_test_result_failed

subroutine print_test_results_summary(test_results)
   type(test_results_t), intent(in) :: test_results

   print *, test_results%all_tests, ' test(s) executed, ', &
      test_results%passed_tests, ' test(s) passed'

   if (test_results%failed_tests > 0) then
      print *, test_results%failed_tests, 'test(s) failed'
   end if
end subroutine print_test_results_summary

Note the use of advanced='no': this ensures the name of the test isn’t followed by a newline as it normally is. This means we can now print the name of the test and the test result (PASS or FAIL) on the same line.

We then call these subroutines before running each test and when handling the result:

function run_test_suites(test_suites) result(test_results)
   ! ...
   
   do test_suite_index = 1, size(test_suites)
+     call print_test_suite_name('The name of the test suite')
      unit_tests = test_suites(test_suite_index)%collect()

      do unit_test_index = 1, size(unit_tests)
         ! ...

+        call print_test_name('The name of the test')
         test_failed = unit_tests(unit_test_index)%test_procedure()

         test_results%all_tests = test_results%all_tests + 1
         if (test_failed) then
            test_results%failed_tests = test_results%failed_tests + 1
+           call print_test_result_failed()
         else
            test_results%passed_tests = test_results%passed_tests + 1
+           call print_test_result_passed()
         end if
      end do
   end do
end function run_test_suites

After removing the old print statements from the test procedures, we get the following output:

[The name of the test suite]
The name of the test... PASS
The name of the test...  Actual:    6.41421356237309     
 Expected:    4.41421356237309     
 Epsilon:   1.000000000000000E-010
 Reals are not equal
FAIL
           2  test(s) executed,            1  test(s) passed
           1 test(s) failed

Of course, what’s missing is the right test names: unit tests and test suites don’t declare their own name yet, so the test framework has nothing to show, except the dummy text “The name of the test”. We can change that by adding string name components to unit_test_t and test_suite_t:

type :: unit_test_t
+  character(len=:), allocatable :: name
   procedure(test_procedure_interface), pointer, nopass :: test_procedure
end type unit_test_t

type :: test_suite_t
+  character(len=:), allocatable :: name
   procedure(collect_unit_tests_interface), pointer, nopass :: collect
end type test_suite_t

When instantiating these types, test modules have to provide the string description to the constructor:

! For unit tests:
unit_tests = [ &
             unit_test_t('Calculate length of polyline with 2 points', &
                         test_polyline_with_2_points), &
             unit_test_t('Calculate length of polyline with 3 points', &
                         test_polyline_with_3_points) &
             ]

! For test suites:
new_test_suite('Polyline tests', collect_polyline_tests) &

Finally, we modify the arguments for the various print_* calls:

-call print_test_suite_name('The name of the test suite')
+call print_test_suite_name(test_suites(test_suite_index)%name)

-call print_test_name('The name of the test')
+call print_test_name(unit_tests(unit_test_index)%name)

And then the output becomes:

[Polyline tests]
Calculate length of polyline with 2 points... PASS
Calculate length of polyline with 3 points...  Actual:    6.41421356237309     
 Expected:    4.41421356237309     
 Epsilon:   1.000000000000000E-010
 Reals are not equal
FAIL
           2  test(s) executed,            1  test(s) passed
           1 test(s) failed

Much better. One issue is that assert_equals is still print-ing directly, in the middle of the test run. It would be better to do that afterward. First we show that the test failed, then show the reason why. We’ll leave that for later.

The most useful improvement we can do now is to make output completely optional, in the case we want to test the framework without causing any output to be written directly to the terminal. As an additional benefit: by completing this task we’ll make it easy for others to implement their own way of printing test output. If they like colors, they can add them, if they just want the summary that’s fine too.

Making an abstract progress printer

In essence, we need a way to replace the implementation of all the print_ subroutines we created. A common solution to such a problem is to introduce a derived type that contains all these procedures as type-bound procedures. We then define an abstract base type with deferred procedures, which allows alternative implementations for these procedures by its subtypes. We’ve seen this approach before when we created abstract logger services.

Here, we start by extracting the interfaces for the various print_ procedures. There’s a very simple recipe for this:

  • Copy the existing subroutines into a new abstract interface block.
  • Add a suffix “interface” to the procedure name. Remove any statements and declarations, except those of the dummy arguments and return types.
  • Add implicit none(type, external) to every interface. For some reason, this isn’t inherited from the module itself.
  • Add imports for any symbols used in the declarations. Even if they are in the same module, they still need to be imported (e.g. test_suite_t).
abstract interface
   subroutine print_test_suite_name_interface(test_suite_name)
      implicit none(type, external)
      character(len=*), intent(in) :: test_suite_name
   end subroutine print_test_suite_name_interface

   subroutine print_test_name_interface(test_name)
      implicit none(type, external)
      character(len=*), intent(in) :: test_name
   end subroutine print_test_name_interface

   subroutine print_test_result_passed_interface()
      implicit none(type, external)
   end subroutine print_test_result_passed_interface

   subroutine print_test_result_failed_interface()
      implicit none(type, external)
   end subroutine print_test_result_failed_interface

   subroutine print_test_results_summary_interface(test_results)
      import :: test_results_t
      implicit none(type, external)
      type(test_results_t), intent(in) :: test_results
   end subroutine print_test_results_summary_interface
end interface

We use these interfaces to declare deferred procedures on an abstract progress_printer_t:

type, abstract :: progress_printer_t
contains
   procedure(print_test_suite_name_interface), deferred, nopass :: print_test_suite_name
   procedure(print_test_name_interface), deferred, nopass :: print_test_name
   procedure(print_test_result_passed_interface), deferred, nopass :: print_test_result_passed
   procedure(print_test_result_failed_interface), deferred, nopass :: print_test_result_failed
   procedure(print_test_results_summary_interface), deferred, nopass :: print_test_results_summary
end type progress_printer_t

We then define one concrete progress printer, the default_progress_printer_t, which extends progress_printer_t:

type, extends(progress_printer_t) :: default_progress_printer_t
contains
   procedure, nopass :: print_test_suite_name
   procedure, nopass :: print_test_name
   procedure, nopass :: print_test_result_passed
   procedure, nopass :: print_test_result_failed
   procedure, nopass :: print_test_results_summary
end type default_progress_printer_t

The type-bound procedures of default_progress_printer_t are just the existing subroutines we have. Because we used the nopass attribute, they don’t even need self as their first required argument: the instance on which the procedure is called, will not be passed as an first argument.

We can now start using the new progress printer. Let’s do that by passing it as an argument to both run_test_suites and handle_test_results. Inside those functions we should depend on the abstract progress_printer_t, so we could switch to different implementations when needed:

-function run_test_suites(test_suites) result(test_results)
+function run_test_suites(test_suites, progress_printer) result(test_results)
   type(test_suite_t), dimension(:), intent(in) :: test_suites
+  class(progress_printer_t), intent(in) :: progress_printer

   ! ...
   do test_suite_index = 1, size(test_suites)
-     call print_test_suite_name(test_suites(test_suite_index)%name)
+     call progress_printer%print_test_suite_name(test_suites(test_suite_index)%name)
      unit_tests = test_suites(test_suite_index)%collect()

   ! ...

Note that we just prepend progress_printer% to all existing print_* calls. Finally, we have to instantiate the default_progress_printer_t and pass it as the runtime argument for all the class(progress_printer_t) arguments:

subroutine run_test_suites_and_print_results(test_suites)
   type(test_suite_t), dimension(:), intent(in) :: test_suites

+  class(progress_printer_t), allocatable :: progress_printer
   type(test_results_t) :: test_results

+  progress_printer = default_progress_printer_t()

-  test_results = run_test_suites(test_suites)
+  test_results = run_test_suites(test_suites, progress_printer)

-  call handle_test_results(test_results)
+  call handle_test_results(test_results, progress_printer)
end subroutine run_test_suites_and_print_results

We can easily create alternative progress printers now, by making a new subtype, and providing implementations for its procedures. One type of printer that we may need is the no_output_printer_t, which does nothing.

The only thing in our test program that still prints output directly to the terminal is the assert_equals function. In the next post we’ll fix that, and improve several other aspects of this procedure.