Fortran - Testing - More assertion functions

When expanding our test suite, we’ll certainly encounter the need for an assert_equals function that works with other types than real(kind=real64). We’d want to compare values of type logical, character, integer, etc. For example, say we have a utility function str_to_upper for which we are adding a unit-test in the test_string module. This function is supposed to convert the letters in a string to their uppercase alternative. The test looks like this:

function test_letters() result(test_failed)
   class(test_failed_t), allocatable :: test_failed

   type(assertion_failed_t), allocatable :: assertion_failed

   assertion_failed = assert_equals('FOO', &
                                    str_to_upper('foo'))
   if (allocated(assertion_failed)) then
      test_failed = assertion_failed
   end if
end function test_letters

Introducing an interface for assert_equals

This doesn’t work, of course, because assert_equals has a function signature that’s incompatible with how it’s called: we pass two strings, but the existing function expects two reals. We’d need something like an assert_strings_equal(). Still, it would be nice if we don’t have to put the type names in the various assert_*_equal function names that we’ll eventually have; that would be restating the obvious. Instead, we’d like to have a single assert_equals function that can be used for comparing different types of values. assert_equals could then be used as an alias for any of the type-specific assert_*_equal functions we have in our framework.

In previous articles, we’ve already encountered a technique for “aliasing” procedures: the interface definition, with references to module procedures. Let’s start by renaming the assertion function we already to assert_reals_equal:

-function assert_equals(expected, actual, epsilon, custom_message) result(assertion_failed)
+function assert_reals_equal(expected, actual, epsilon, custom_message) result(assertion_failed)

We then introduce the assert_equals interface and specify that assert_reals_equal is one of its possible implementations:

interface assert_equals
    module procedure :: assert_reals_equal
end interface

Note that in order to use this interface in other modules, it has to be made public.

An advantage of using an interface to refer to a single procedure is that we don’t have to update existing call sites. Existing tests can keep calling assert_equals, but the compiler will forward the call to assert_reals_equal. This is a nice trick that we can use to keep backward compatibility when doing refactorings (more about this in a future article). This has no performance penalty at all, because resolving the interface call to its actual procedure is handled at compile-time.

Compiling the test for str_to_upper, we now get an interesting compiler error:

There is no matching specific function for this generic 
function reference.   [ASSERT_EQUALS]
      assertion_failed = assert_equals('FOO', &
-------------------------^

The compiler recognizes that we are calling the interface, but finds no module procedure with a signature that has two string arguments.

An interface with just one module procedure acts like a simple alias for that procedure. But we can make it more powerful if we add alternative procedures, like an assert_strings_equal function that fulfills the needs of our test:

   public :: assert_equals
   
   ! ...
   
   interface assert_equals
      module procedure :: assert_reals_equal
      module procedure :: assert_strings_equal
   end interface

contains

   function assert_strings_equal(expected, actual, custom_message) result(assertion_failed)
      character(len=*), intent(in) :: expected
      character(len=*), intent(in) :: actual
      character(len=*), optional, intent(in) :: custom_message

      type(assertion_failed_t), allocatable :: assertion_failed

      character(len=:), allocatable :: message

      if (len(expected) /= len(actual)) then
         message = 'Strings have different lengths. '// &
                   'actual='//actual// &
                   ', expected='//expected
         if (present(custom_message)) then
            message = custom_message//' '//message
         end if
         assertion_failed = assertion_failed_t(message)
         return
      end if

      if (expected /= actual) then
         message = 'Strings have different content. '// &
                   'actual='//actual// &
                   ', expected='//expected
         if (present(custom_message)) then
            message = custom_message//' '//message
         end if
         assertion_failed = assertion_failed_t(message)
         return
      end if
   end function assert_strings_equal

Note that asserting equality for strings consists of two steps here: first we compare the length, then we compare the actual contents. Eventually we’d likely want to improve the output, because it will be difficult to compare long strings visually and figure out the differences. For now, just print-ing the complete strings is fine.

If the first check fails, we’ll set the assertion_failed return variable, but then we should immediately return from the function. If we don’t do this, the return variable will be overwritten in the next if clause, because if the lengths of the strings aren’t equal, their contents won’t be equal too. This will be the case for any unit test that makes multiple assertions, and will be an important implementation rule: as soon as the first assertion fails, we should jump out of the test.

Most other languages that have unit-testing frameworks don’t have this rule. There, a failed assertion will throw an exception, which already prevents further execution of the test procedure; we don’t have to manually jump out of the function. In Fortran, where we use return values to indicate failure, it’s impossible to implement it like that. The only other option we have is to error stop on failure, and we already saw on multiple occasions that that’s not a good option in most cases. Still, the solution can be improved. In the next article we’ll discuss an approach that can save us from making the mistake of forgetting to return after a failed assertion.

When building the project again, the call to assert_equals in test_string will be inspected by the compiler: since we pass two strings (and an optional custom message), it’s resolved to assert_strings_equal. Since str_to_upper doesn’t work yet, these tests fail, but with helpful output:

[String tests]
Empty string... PASS
Letters converted to uppercase... FAIL
Strings have different content. actual=foo, expected=FOO
Other characters not converted... FAIL
Strings have different content. actual=foo123, expected=FOO123

Minimizing code duplication

The code in assert_strings_equal could be improved somewhat, because it contains some code duplication related to the custom_message argument. It’s optional, yet, before we can use it, we always have to check its presence. We can’t assign a default value to it, that would be used if the caller doesn’t provide the argument. Actually, we can’t assign any value to it, because it’s an intent(in) argument, which makes the variable read-only. The following is impossible:

function assert_strings_equal(expected, actual, custom_message) &
                              result(assertion_failed)
   ! ...

   character(len=*), intent(in) :: custom_message
   
   if (.not. present(custom_message)) then
      ! assigning a "default" value is impossible
      custom_message = ''
   end if

It results in this compiler error:

A dummy argument with the INTENT(IN) attribute shall not 
be defined nor become undefined.

The simplest thing we can do to at least get rid of the duplicate concatenation is to introduce a local variable that gets assigned the value of the custom_message argument, or an empty string. Then we can prepend it to the generated message, without first checking if it’s present. This also removes the need for the temporary message variable:

function assert_strings_equal(expected, actual, custom_message) &
                              result(assertion_failed)
   ! ...

   character(len=:), allocatable :: prepend_message
   
   if (present(custom_message)) then
      prepend_message = custom_message//' '
   else
      prepend_message = ''
   end if

   if (len(expected) /= len(actual)) then
      assertion_failed = assertion_failed_t(prepend_message// &
         'Strings have different lengths. '// &
         'actual='//actual// &
         ', expected='//expected)
      return
   end if

   if (expected /= actual) then
      assertion_failed = assertion_failed_t(prepend_message// &
         'Strings have different content. '// &
         'actual='//actual// &
         ', expected='//expected)
      return
   end if
end function assert_strings_equal

It’s a small change, but it helps keep our code nice and clean.

Supporting multiple kind-types

For every built-in or primitive type we can add a generic, reusable assert_*_equal function, like assert_logicals_equal, assert_integers_equal, etc. For all these types, we may also have to consider their kind. As we saw before when declaring reals, we always pass a specific kind, like real64 which can be imported from iso_fortran_env. The kind always has to be a specific value, and it has to be a compile-time parameter, which means we can’t just make an assert_reals_equal function that can deal with reals of any kind. We have to basically repeat the same code, only with a different kind, e.g.

function assert_reals_32_equal(expected, actual, epsilon, custom_message) result(assertion_failed)
   real(kind=real32), intent(in) :: expected
   real(kind=real32), intent(in) :: actual
   real(kind=real32), intent(in) :: epsilon
   
   ! ...

   if (abs(actual - expected) > epsilon) then
      assertion_failed = assertion_failed_t(prepend_message// &
         'Reals are not equal. '// &
         'actual='//real32_to_string(actual)// &
         ', expected='//real32_to_string(expected)// &
         ', epsilon='//real32_to_string(epsilon))
   end if
end function assert_reals_32_equal

And so on, same for integers and logicals, which could also have different kinds. This is a bit sad, since there’s going to be a lot of code duplication, meaning changes to one of these functions have to be copied manually to all the similar functions.

One way to make it easier to maintain code like this is to use a preprocessor. This is a tool that allows you to modify the code just before you give it to the compiler. You’d list all the kind-types for which you want the assert_reals_equal function to work, and the preprocessor will repeat the same code, into which you can insert the kind-types when needed. It basically treats the code as a template, filling out the details last-minute:

interface assert_equals
#:for real_kind in ['real64', 'real32']
    module procedure assert_${real_kind}_equal$
#:endfor
end interface

#:for real_kind in ['real64', 'real32']
   function assert_${real_kind}_equals(expected, actual, epsilon, custom_message) 
                                        result(assertion_failed)
      real(kind=${real_kind}), intent(in) :: expected
      real(kind=${real_kind}), intent(in) :: actual
      real(kind=${real_kind}), intent(in) :: epsilon
      ! ...
#:endfor

FPM doesn’t support it yet, but once they do you can just add these lines to the fpm.toml configuration file:

[preprocess]
[preprocess.fypp]

If you use some other build process, it should be doable to set this up. With FPM, I haven’t found an elegant alternative. Plus, using pre-processor directives in code is likely to render your IDE helpless when trying to interpret the meaning of the code, since it’s not in its final form.

Comparing derived types

Besides making assertion functions for primitive/built-in types, we’ll have to make a custom function for each derived type that we’d like to compare. Each data component has to be compared separately. That sounds like a lot of work, but it’s often not hard work, since derived types will often have primitive data components, which can be compared using the existing assert_*_equals functions. Revisiting a type we used before, point_t:

type :: point_t
   real(kind=real64) :: x
   real(kind=real64) :: y
end type point_t

When we have a function move_x that works as follows:

pure function move_x(point, dx) result(new_point)
   class(point_t), intent(in) :: point
   real(kind=real64), intent(in) :: dx

   type(point_t) :: new_point

   new_point%x = point%x + dx
   new_point%y = point%y
end function move_x

We would write a unit test like this:

function test_move_point_to_the_right() result(test_failed)
   type(test_failed_t), allocatable :: test_failed

   type(assertion_failed_t), allocatable :: assertion_failed
   type(point_t), allocatable :: expected, actual

   expected = point_t(3.0_real64, 4.0_real64)
   actual = move_x(point_t(1.0_real64, 4.0_real64), 2.0_real64)

   assertion_failed = assert_equals(expected, actual)
   if (allocated(assertion_failed)) then
      test_failed = assertion_failed
   end if
end function test_move_point_to_the_right

By the way, we don’t need local variables for expected and actual, but it reduces the width of the code sample, which is good for this reading format. Of course, we get the already familiar compiler error:

There is no matching specific function for this 
generic function reference.   [ASSERT_EQUALS]
      assertion_failed = assert_equals(expected, actual)
-------------------------^

We have to add another procedure to the interface assert_equals, e.g. assert_points_equal. However, that interface lives inside the test_framework module. It should remain generic, so we shouldn’t add assert_points_equal there. Luckily, we can “extend” an interface definition locally, in our own test module. First, we need to import the interface from the test_framework (we already do):

use test_framework, only: assert_equals

We add an assert_points_equal procedure:

function assert_points_equal(expected, actual) result(assertion_failed)
   class(point_t), intent(in) :: expected
   class(point_t), intent(in) :: actual

   type(assertion_failed_t), allocatable :: assertion_failed

   assertion_failed = assert_equals(expected%x, actual%x, 1.0e-10_real64)
   if (allocated(assertion_failed)) then
      return
   end if
   
    assertion_failed = assert_equals(expected%y, actual%y, 1.0e-10_real64)
end function assert_points_equal

Then we can add our own procedure to the existing interface (this modification of the interface will be local to the module):

interface assert_equals
   module procedure :: assert_points_equal
end interface

This doesn’t override, but extends the assert_equals interface, meaning that this now works:

expected = point_t(3.0_real64, 4.0_real64)
actual = move_x(point_t(1.0_real64, 4.0_real64), 2.0_real64)

assertion_failed = assert_equals(expected, actual)

But comparing reals, strings, etc. also keeps working.

When you want to compare point_t instances in another test module, e.g. test_polyline, you can import the assert_equals interface from test_point. The only requirement is that the interface is public there, but that’s already the case for anything you want to use in other modules. Importing the same interface name from multiple modules, will just merge the list of associated module procedures, which is exactly what we need.

We are really getting somewhere, but while we are adding more tests and more assertion functions, there are some design issues that keep causing friction. We’ll tackle these issues in the next post.

• Fortran
• Unit Testing