Fortran - Testing - Returning test and assertion errors

Matthias Noback

In the previous post we improved the output handling of the test runner, making output optional by using an abstract progress printer. Only our assert_equals function, which by the way should also be moved to the test_framework module, still prints directly to stdout:

function assert_equals(expected, actual, epsilon) result(assertion_failed)
   ! ... 
   assertion_failed = .false.

   if (abs(actual - expected) > epsilon) then
      print *, 'Actual: ', actual
      print *, 'Expected: ', expected
      print *, 'Epsilon: ', epsilon
      print *, 'Reals are not equal'
      assertion_failed = .true.
   end if
end function assert_equals

To fix this, we should upgrade the return type. One idea would be to return a message (variable-length string) instead of a logical, but a more flexible, future-proof alternative is to define a custom type for an error. This would allow us to pass more information back to the caller than just the message. A suitable name would be assertion_failed_t:

type :: assertion_failed_t
   character(len=:), allocatable :: message
end type assertion_failed_t

Now instead of returning .true. in case of failure, assert_equals should return an instance of assertion_failed_t. We effectively make this return value optional by adding allocatable to it:

function assert_equals(expected, actual, epsilon) result(assertion_failed)
   real(kind=real64), intent(in) :: expected
   real(kind=real64), intent(in) :: actual
   real(kind=real64), intent(in) :: epsilon
-  logical :: assertion_failed
+  type(assertion_failed_t), allocatable :: assertion_failed

-  assertion_failed = .false.

   if (abs(actual - expected) > epsilon) then
-     print *, 'Actual: ', actual
-     print *, 'Expected: ', expected
-     print *, 'Epsilon: ', epsilon
-     print *, 'Reals are not equal'
-     assertion_failed = .true.
+     assertion_failed = assertion_failed_t('Reals are not equal. '// &
+                                           'actual='//real64_to_string(actual)// &
+                                           ', expected='//real64_to_string(expected)// &
+                                           ', epsilon='//real64_to_string(epsilon) &
+                                           )
   end if
end function assert_equals

In order to easily build up the error message, we want to concatenate strings to reals, which in Fortran is impossible without any intermediate steps. This is what the helper function real64_to_string does:

pure function real64_to_string(value) result(str)
   real(kind=real64), intent(in) :: value
   character(len=32) :: temp
   character(len=:), allocatable :: str

   write (temp, *) value
   str = trim(adjustl(temp))
end function real64_to_string

We’ll discuss better ways of doing this in a future post.

The assertion error message is now available in the test, but we don’t want the test to print anything either. It’s better to delegate that responsibility to the test runner, more specifically the progress printer. This means we have to upgrade the return type of each test to return not just a logical but a more advanced type as well, one that can carry the error message to the test runner. Let’s call this type test_failed_t:

type :: test_failed_t
   character(len=:), allocatable :: message
end type test_failed_t

The updated test procedure interface:

abstract interface
   function test_procedure_interface() result(test_failed)
      import test_failed_t
      implicit none(type, external)

      class(test_failed_t), allocatable :: test_failed
   end function test_procedure_interface
end interface

Note: we have to import test_failed_t explicitly, and it has to be defined in the module before this interface.

The corresponding update to one of the existing test procedure:

function test_polyline_with_3_points() result(test_failed)
   class(test_failed_t), allocatable :: test_failed
   ! ...

Before, we could simply assign the logical returned by assert_equals to the return variable of the test procedure:

test_failed = assert_equals(5.0_real64, &
                            polyline_length(two_points), &
                            1.0e-10_real64)

This no longer works, since assertion_failed_t and test_failed_t are different and incompatible types. To make that work again, we have to make sure assertion_failed_t can be used as a test_failed_t. Two options:

  1. Define a (possibly abstract) parent type from which both assertion_failed_t and test_failed_t extend, e.g. test_error_t. We let test procedures return class(test_error_t), so any subtype is allowed.
  2. Let assertion_failed_t extend test_failed_t. We let test procedures return test_failed_t, which allows either of these types to be returned.

Let’s use the second option (I think it’s not clear yet which option will be better in the long run):

-type :: assertion_failed_t
+type, extends(test_failed_t) :: assertion_failed_t
-  character(len=:), allocatable :: message
end type assertion_failed_t

In the test loop we should now catch the return value and check if it’s allocated. If it is, it should be treated in the same way as assertion_failed was before. Note that we should also deallocate the local variable test_failed if needed, so the next test may start with a clean slate:

function run_test_suites(test_suites, progress_printer) result(test_results)
   ! ...
-  logical :: test_failed
+  class(test_failed_t), allocatable :: test_failed

   do test_suite_index = 1, size(test_suites)
      ! ...
      do unit_test_index = 1, size(unit_tests)
         ! ...
         test_failed = unit_tests(unit_test_index)%test_procedure()

         test_results%all_tests = test_results%all_tests + 1
-        if (test_failed) then
+        if (allocated(test_failed)) then
            ! ...
+           deallocate (test_failed)
         else
            ! ...
         end if
      end do
   end do
end function run_test_suites

Everything works well if the assertion fails: the assertion_failed value will be copied to the test_failed variable correctly, because it has been allocated. However, if the assertion succeeds, assertion_failed will not be allocated, and if we try to copy it to test_failed, we’ll get a “segmentation fault”. It makes sense, because we’d be trying to access memory that has not been allocated, which should not be allowed.

The solution is to make a temporary variable for assertion_failed. Only if it’s allocated, we’ll assign it to test_failed:

function test_polyline_with_2_points() result(test_failed)
   class(test_failed_t), allocatable :: test_failed

+  type(assertion_failed_t), allocatable :: assertion_failed
   ! ...
   assertion_failed = assert_equals(5.0_real64, &
                                    polyline_length(two_points), &
                                    1.0e-10_real64)
-  test_failed = assertion_failed
+  if (allocated(assertion_failed)) then
+     test_failed = assertion_failed
+  end if
end function test_polyline_with_2_points

This isn’t great, in my opinion. We’ll take some time to improve this in a future post.

Printing the test error using the progress printer

Finally, we can access the assertion error inside the test runner. Since it’s a value (test_failed_t), we can pass it to the existing print_test_result_failed procedure. We first modify the existing interface print_test_result_failed_interface to accept it as an argument:

subroutine print_test_result_failed_interface(test_failed)
+  import test_failed_t
   implicit none(type, external)
+  class(test_failed_t), intent(in) :: test_failed
end subroutine print_test_result_failed_interface

Then we update the existing implementation:

-subroutine print_test_result_failed()
+subroutine print_test_result_failed(test_failed)
+  class(test_failed_t), intent(in) :: test_failed
   write (stdout, '(A)') 'FAIL'
+  write (stdout, '(A)'), test_failed%message
end subroutine print_test_result_failed

Then we pass the test_failed variable from the test runner to the printer:

-call progress_printer%print_test_result_failed()
+call progress_printer%print_test_result_failed(test_failed)

The output looks much better now:

[Polyline tests]
Calculate length of polyline with 2 points... PASS
Calculate length of polyline with 3 points... FAIL
Reals are not equal. actual=6.41421356237309, expected=4.41421356237309, epsilon=1.000000000000000E-010, 
           2  test(s) executed,            1  test(s) passed

Adding custom messages

It would be nice if we could pass even more context to the error. The generic assertion speaks of reals not being equal, but we could add a quick explanation of what real we’re talking about. We should be able to add this context from within the test itself, that is, when calling assert_equals it should accept an additional string, like 'Incorrect length.'.

It’s best if this is an optional argument, because it may not make sense to provide it in all situations. If the argument is present, we’d put it before the generated message:

-function assert_equals(expected, actual, epsilon) result(assertion_failed)
+function assert_equals(expected, actual, epsilon, custom_message) result(assertion_failed)
   real(kind=real64), intent(in) :: expected
   real(kind=real64), intent(in) :: actual
   real(kind=real64), intent(in) :: epsilon
+  character(len=*), optional, intent(in) :: custom_message

   type(assertion_failed_t), allocatable :: assertion_failed

   character(len=:), allocatable :: message

   if (abs(actual - expected) > epsilon) then
      message = 'Reals are not equal. '// &
                'actual='//real64_to_string(actual)// &
                ', expected='//real64_to_string(expected)// &
                ', epsilon='//real64_to_string(epsilon)
+     if (present(custom_message)) then
+        message = custom_message//' '//message
+     end if
      assertion_failed = assertion_failed_t(message)
   end if
end function assert_equals

We call assert_equals providing the custom message:

test_failed = assert_equals(5.0_real64, &
                            polyline_length(two_points), &
                            1.0e-10_real64, &
                            'Incorrect length.')

And the output becomes this:

Calculate length of polyline with 3 points... FAIL
Incorrect length. Reals are not equal. ...

In the next post we’ll look at adding more assertion functions to the test framework.