Fortran - Testing - Improving the design of the test framework - Part 2

Several articles later, it turns out the little test framework has grown very fast. We’re looking at around 400 lines of code in a single test_framework module. In my opinion, this requires too much scrolling and jumping through the code to understand what’s going on or make changes. I’m aware “real-world” projects suffer from files that are a lot larger, but they are in a lot of trouble because of it. It’s smart to split large modules into smaller ones at a much earlier stage. As we discussed before, the following guidelines may be used when doing so:

• Each derived type and its type-bound module procedures should be in their own module. This would be the case for test_result_t and progress_printer_t, but also default_progress_printer_t, etc.
• Similar functions or functions that reuse some shared but private logic may live in their own module, e.g. the assert_*_equal functions and the assert_equals interface could be in the same assertions module.
• For users, it will be helpful if there is a single “facade” module that exposes the elements they need access to, so they don’t have to dig into the smaller “helper” modules. As an example, we want to make it easy for users to add unit tests to their program, and to define their own test runner application. The procedures and types needed for this, should all be accessible (importable) from a single facade module.

When moving derived types to their own module, we no longer need to alias the module procedures and prefix their names with the name of the type:

module test_framework_test_result
   implicit none(type, external)

   private
   public :: test_result_t

   type :: test_result_t
      private
      logical :: failed = .false.
      logical :: uncaught = .false.
      character(len=:), allocatable :: message
      integer :: assertion_count = 0
   contains
      procedure :: process
      procedure :: fail
      procedure :: has_failed
      procedure :: is_risky
   end type test_result_t

contains

   subroutine process(self, assertion_result)
      ! ...
   end subroutine process
end module test_framework_test_result

While moving things to their own module, there are some small decisions to be made. For example, test_result_t and test_results_t are distinct derived types, so following our rule they should be stored in separate modules. However, modules that use test_results_t also use test_result_t (i.e. any progress printer implementation needs them both), so we could put them in a single module. On the other hand, modules that use only test_result_t don’t usually use test_results_t as well. This would be a good reason to still put them in separate modules.

We gradually start recognizing some kind of dependency graph. By moving things around, we notice that all of our dependencies go in one direction. For instance, a test_result_t doesn’t need the progress_printer_t; it’s the other way around. This means we’ll have so-called leaf modules which are at the end of the line in the dependency graph. There will be higher-level modules that use those leaf modules.

These leaf modules represent concepts that are lower-level than the other modules; they are used only from higher-level modules. They contain implementation details that higher-level modules try to hide from their users, who are generally unaware of them. For instance, the progress printer is a lower-level concept than the top-level subroutine run_unit_tests_and_print_results. Users of this subroutine won’t have to deal with the printer itself, nor with the intermediate test result objects.

The resulting list of module files is:

src/
  test_framework/
    assertions.f90
    default_progress_printer.f90
    progress_printer.f90
    test_result.f90
    test_results.f90
    test_suite.f90
    unit_test.f90

Now we need to define a facade for the test framework, so it’s easy to use for programmers. Which means:

• Users should be allowed to import framework elements they have to interact with from just one module (the facade).
• Users should only need to import procedures that fulfill their actual needs.

One way the test framework is used is to define the test suites and run the tests. So the facade module should offer these as procedures, even though new_test_suite, for instance, is actually defined in another module:

program check
   use test_polyline, only: collect_polyline_tests
   use test_string, only: collect_string_tests
   use test_point, only: collect_point_tests
   use test_framework_facade, only: run_test_suites_and_print_results, &
                                    new_test_suite

   implicit none(type, external)
   call run_test_suites_and_print_results( &
      [ &
      new_test_suite('String tests', collect_string_tests), &
      new_test_suite('Polyline tests', collect_polyline_tests), &
      new_test_suite('Point tests', collect_point_tests) &
      ])
end program check

This can be accomplished by letting test_framework_facade import it from test_framework_test_suite, but also expose it as a public element:

module test_framework_facade
   use test_framework_test_suite, only: new_test_suite

   implicit none(type, external)

   private
   public :: new_test_suite
end module test_framework_facade

Normally, this is not recommended, since it makes it hard to untangle dependencies. It’s not immediately clear where a dependency actually lives (because these dependencies are in a sense transitive). However, a facade module is a good exception to this rule, because it results in a better user/developer experience. Also, we gain the freedom to change our internal module structure and for example move the exposed procedures to a different “private” module. The end user wouldn’t notice this, since they import only public elements exposed by the facade.

Another “use case” for the test framework is for users to write tests with it. For this they need to define unit tests and call assertion functions. So the related elements should be accessible from the facade as well:

module test_point
   use test_framework_facade, only: unit_test_t, assert_equals, test_result_t

   implicit none(type, external)

   private
   public :: collect_point_tests

   ! ...
end module test_point

Here we can use the same trick again: the facade exports elements that live elsewhere.

As we did before, in the logging library, we could also introduce a submodule to hide some more implementation details from the user. For any public procedure that the user should be able to use, we provide an interface in the facade module:

module test_framework_facade
   public :: run_test_suites_and_print_results
   ! ...

   interface
      module subroutine run_test_suites_and_print_results(test_suites)
         import :: test_suite_t
         implicit none(type, external)
         type(test_suite_t), dimension(:), intent(in) :: test_suites
      end subroutine run_test_suites_and_print_results
   end interface
   
   ! ...
end module test_framework_facade

The implementation of the interface lives in the test_framework_facade_implementation submodule:

submodule(test_framework_facade) test_framework_facade_implementation
   ! ...

   implicit none(type, external)

contains

   module subroutine run_test_suites_and_print_results(test_suites)
      type(test_suite_t), dimension(:), intent(in) :: test_suites

      ! The actual implementation
   end subroutine run_test_suites_and_print_results
   
   ! Private helper procedures go here
end submodule test_framework_facade_implementation

With the facade and facade implementation (sub)modules, we add two more files to the test_framework folder:

src/
  test_framework/
    ... 
    facade_implementation.f90
    facade.f90

I think the benefits of splitting large modules into smaller modules are huge:

• Each module offers some public elements, keeping its internals private. This makes it easier to find out what changes can be made safely without affecting users of the module.
• If we want to make a change related to some part of the framework, we can easily find our way to the file where this change should be made: the names are self-explanatory.
• Similarly, for the reader who wants to familiarize themselves with the code base, it’s easier to get a quick overview of the concepts involved in the testing framework. They can even spot concepts that aren’t there. For instance, the ability to use test doubles (mocks, stubs, etc.) can be recognized as a “missing” feature for this framework. Rightly so, because in Fortran, there couldn’t be a standard, reusable solution for them.

Furthermore, splitting modules into modules and submodules has many advantages too. At the very least, it allows you to change the code inside a procedure without triggering recompilation of files that use those procedures (as long as their signatures are still the same, of course). For a library, this is less important, because it doesn’t change often; only if you install a new version and in that case recompilation isn’t a bad thing. But in that case it offers a way to more explicitly define what you want the public API of the library to be.

Publishing the framework as a reusable FPM package

To make the test_framework library easy to use in another project, we should somehow host it separate from our project code base. Since we’re using FPM, this is very easy. We can put it in a GitHub repository. Note that this makes the code publicly accessible, which is fine in this case. Now we should add an fpm.toml file with any number of project settings in it, but at least, it’s good to put a name in it:

name = "test_framework"

Not a great name, but this allows us to use the repository as a project dependency. In our project’s fpm.toml we add the following:

[dev-dependencies]
test_framework.git = "https://github.com/matthiasnoback/fortran-test-framework"

You can and should specify a version control tag as well, which is definitely best practice when including external packages.

Now after removing the same modules from the main project, we just run fpm test again, and FPM will install and build the test_framework dependency for us. Everything works as it did before, but now we can easily reuse the framework in other projects.

Some things to note about this diagram:

• test_result_t is an important type. The module it contains is heavily relied on as dependency. This makes sense for a test framework. There are several implications though. It should have a stable interface. We can’t easily make changes to test_result_t and its procedures, since it’s relied on by many other things. One way to make the interface stable is to encapsulate state and logic, which we did: it can be modified and queried only in the specific ways we offer (i.e. by calling its type-bound procedures).
• Looking at the facade and its implementation submodule, it’s clear that the facade has fewer dependencies than the implementation. It makes sense, but it’s a nice benefit. We can freely change the dependencies used by the submodule, without impacting the users of the facade module itself.

There’s an important thing we’ve been postponing, and it’s not very smart: testing the test framework itself. We’ll look into that in the next post.

• Fortran
• Unit Testing