ScalaTest extra features on waitingforcode.com

Versions: ScalaTest 3.0.3

ScalaTest, as xUnit family testing frameworks, provides a lot of features. Even though several of them are not frequently used, it's always good to know them and the context of their use.

This post is divided in 5 parts. Each of them presents one interesting ScalaTest features. The first described one is tagging. It's followed by fixtures, property-based tests, given/when/then mixin and the pending test markers.

Tagging

The role of tags in ScalaTest is similar to their role in xUnit testing frameworks, i.e. the tags are used to define the specificities of test cases. An example of such specificities can be the character (integration vs unit) of the test, its criticality or the reason of adding (e.g. detected after a regression). The tags are the classes extending org.scalatest.Tag class.

Test cases can be annotated with one or several tags. The tagging method depends on the test family (see more in the post about ScalaTest and testing styles). The following code shows a simple implementation for FlatSpec and FunSpec:

object ImportantTest extends Tag("ImportantTest")

class TaggingTest extends FlatSpec {

  "the flat spec test" should "be tagged" taggedAs(ImportantTest) in {
  }

}

class TaggingFunSpecTest extends FunSpec {
  it("the fun spec test should be also tagged as important", ImportantTest) {
  }
  it("the fun spec test should not be tagged as important") {
  }
}

Another writing style uses the annotations to mark whole test class with given tag:

 
import com.waitingforcode.annotations.ImportantTest

@ImportantTest
class TaggingTest extends FlatSpec {

  "the flat spec test" should "be tagged" in {
  }

}

Fixtures

Very often test cases need to share a specific feature, as for instance an in-memory database simulating distributed data store, sockets connections or test configurations. ScalaTest provides a solution to share them throughout the fixtures. They can be introduced with one of these approaches:

Scala-based methods

Scala-based approach uses native Scala language methods to add such fixtures, as for example factory methods or shared objects, as shown in the following snippets:

      "get-fixture method" should "be used in the test" in {
        val fixture = ScalaFixtures.fixtures

        StorePersister.persist(("a", "b"), fixture.store)

        fixture.store("a") shouldEqual "b"
      }

      "object fixture" should "be used in the test" in new StoreFixtureObject {
        StorePersister.persist(("a", "b"), store)

        store("a") shouldEqual "b"
      }
    object StorePersister {
      def persist(entry: (String, String), store: mutable.Map[String, String]): Unit = {
        store.put(entry._1, entry._2)
      }
    }


    object ScalaFixtures {
      // get-fixture method example
      def fixtures = new {
        val store: mutable.Map[String, String] = new mutable.HashMap[String, String]()
      }
    }

    trait StoreFixtureObject {
      val store: mutable.Map[String, String] = new mutable.HashMap[String, String]()
    }

Another Scala-based method uses the loan fixture pattern. This solution consists on defining the method doing 2 things: fixtures setup/cleaning and the test case execution. Thus it's a pretty useful way to share a single one logic among all tests requiring the same fixtures:

      private val SharedMapFixture = new mutable.HashMap[String, String]()
      def withSampleFixtureLoanPattern(testCode: mutable.Map[String, String] => Any): Unit = {
        try {
          testCode(SharedMapFixture)
        } finally {
          SharedMapFixture.clear()
        }
      }

      "loan fixture" should "be used in the test" in withSampleFixtureLoanPattern { storePersister =>
        StorePersister.persist(("e", "f"), storePersister)

        storePersister("e") shouldEqual "f"
      }

      "loan fixture" should "be used in the other test" in withSampleFixtureLoanPattern { storePersister =>
        StorePersister.persist(("g", "h"), storePersister)

        storePersister("g") shouldEqual "h"
      }

withFixture method overriding

Another strategy is based on the withFixture method overriding. The TestSuite traits brings a method called withFixture(NoArgTest) that is in charge of the test case execution:

      protected def withFixture(test: NoArgTest): Outcome = {
        test()
      }

This method can be freely overridden by any test needing some specific context before and after the execution, as shown in the following code:

    class ScalaFixtureOverridingWithFixtureTest extends FlatSpec with Matchers {

      private val SharedMap = new mutable.HashMap[String, String]()

      override def withFixture(testCaseToExecute: NoArgTest): Outcome = {
        // The test method should always be invoked by the super and not directly here
        // Moreover everything should be wrapped with try-finally block to avoid to skip the cleaning
        // in the case of an exception
        // The [[Outcome]] can be later used to do something with the test according to its result
        try {
          val result = super.withFixture(testCaseToExecute)
          println(s"Result=${result}")
          result
        } finally {
          SharedMap.clear()
        }
      }

      "test case withFixture" should "be executed and clean the shared map at the end" in {
        SharedMap.put("a", "b")

        SharedMap should have size 1
        SharedMap("a") shouldEqual "b"
      }

      "test case withFixture" should "be executed and clean the shared map at the end in the 2nd test" in {
        SharedMap.put("c", "d")

        SharedMap should have size 1
        SharedMap("c") shouldEqual "d"
      }

    }

BeforeAndAfter trait use

The last solution to share the context is the use of before/after (all) hooks provided by BeforeAndAfter or BeforeAndAfterAll traits. Depending on used trait, we can either call specific code before and after the execution of each test case or directly before the first case and after the last one in given suite:

    class ScalaFixtureBeforeAfterTest extends FlatSpec with Matchers with BeforeAndAfter {

      private val SharedMap = new mutable.HashMap[String, String]()

      after {
        SharedMap.clear()
      }

      "test case withFixture" should "be executed and clean the shared map at the end" in {
        SharedMap.put("a", "b")

        SharedMap should have size 1
        SharedMap("a") shouldEqual "b"
      }

      "test case withFixture" should "be executed and clean the shared map at the end in the 2nd test" in {
        SharedMap.put("c", "d")

        SharedMap should have size 1
        SharedMap("c") shouldEqual "d"
      }

    }

    // Same as above but the cleaning is executed only once
    class ScalaFixtureBeforeAfterAllTest extends FlatSpec with Matchers with BeforeAndAfterAll {

      private val SharedMap = new mutable.HashMap[String, String]()

      override def afterAll(): Unit = {
        SharedMap.clear()
      }

      "test case withFixture" should "be executed and clean the shared map at the end" in {
        SharedMap.put("a", "b")

        // Here we can't assert on the size since the cleaning is executed only
        // after the test
        SharedMap("a") shouldEqual "b"
      }

      "test case withFixture" should "be executed and clean the shared map at the end in the 2nd test" in {
        SharedMap.put("c", "d")

        SharedMap("c") shouldEqual "d"
      }

    }

Property-based tests

The most common tests declaration method specifies one tested object per use case. It's the first way to ensure that the code meets the functional expectations. But a complementary approach exists and it's called property-based tests. In such tests instead of defining 1 single tested value, we either define a set of the values or let the test framework to generate them in our place. Thanks to that we can check how the implementation behaves on a much wider variety of options.

In ScalaTest such tests can be declared with the use of org.scalatest.prop.PropertyChecks mixin, directly in the test definition. It can be used for both previously quoted declaration types (explicit and implicit), as shown in the following code:

class PropertyTests extends PropSpec  with Matchers with PropertyChecks {

  property("tuples conversion should be made correctly") {
    forAll { (firstName: String, lastName: String, bornYear: Int) => {
        val currentYear = 2018
        whenever(bornYear < currentYear) {
          val person = Converter.convertTupleToPerson(firstName, lastName, bornYear, currentYear)

          // As you can notice, the logic from the converter is repeated here
          person.firstName shouldEqual firstName
          person.lastName shouldEqual lastName
          person.age shouldEqual (currentYear - bornYear)
        }
      }
    }
  }

  private val PropertyTable = Table(
    "test dataset",
    ("a", "b", 2000, 18),
    ("a", "b", 1990, 28),
    ("a", "b", 1980, 38)
  )

  property("converting dataset from property table") {
    val referenceYear = 2018
    forAll(PropertyTable) { testTuple => {
        val (firstName, lastName, bornYear, expectedAge) = testTuple

        val person = Converter.convertTupleToPerson(firstName, lastName, bornYear, referenceYear)

        person.firstName shouldEqual firstName
        person.lastName shouldEqual lastName
        person.age shouldEqual expectedAge
      }
    }
  }

}


object Converter {

  def convertTupleToPerson(firstName: String, lastName: String, bornYear: Int, referenceYear: Int): Person = {
    val age = referenceYear - bornYear
    Person(firstName, lastName, age)
  }

}

case class Person(firstName: String, lastName: String, age: Int)

In the first test case the data is generated automatically by built-in generators. But we can also generate more specific objects by implementing the Gen[+T] trait. You can also notice the use of a whenever method controlling to the input parameters we want to check. Maybe one of the most important drawbacks of the automatic property checks is the logic duplication. As you can see in the presented (pretty simple though) case, we duplicate the logic of computing the age in the assertion person.age shouldEqual (currentYear - bornYear) and in the implementation val age = referenceYear - bornYear.

For the case of the second test there is much less magic since all tested data is defined in an object.

Given/when/then mixin

Sometimes it's not obvious to transparently structure the test cases with given/when/then style. ScalaTest provides a way to define these boundaries explicitly with the help of org.scalatest.GivenWhenThen mixin:

class GivenThenMixinTest extends FlatSpec with GivenWhenThen with Matchers {

  case class Numbers(nr1: Int, nr2: Int, nr3: Int)

  "a case class" should "be copied with changed properties" in {
    Given("a case class")
    val numbers = Numbers(1, 2, 3)

    When("the case class is copied with changed element")
    val changedNumbers = numbers.copy(nr3 = 30)

    Then("copied case class should have one attribute changed")
    changedNumbers.nr1 shouldEqual 1
    changedNumbers.nr2 shouldEqual 2
    changedNumbers.nr3 shouldEqual 30
  }

}

And it outputs:

Given a case class
When the case class is copied with changed element
Then copied case class should have one attribute changed

However, since the text in Given/When/Then methods is printed, sometimes it can reduce the readability of the test results in the CI reports.

Pending tests

Another ScalaTest's feature concerns pending tests. A pending test is the test that we know the existence but we don't know the implementation, for example because it was specified before as a kind of functional guideline.

Pending tests are different than ignored tests from 2 points of view: philosophical and technical. Ignored tests are the tests that already exist but that, because of some reason, e.g. quickfix or significant refactoring, must be ignored temporary and so inevitably fixed later. The pending tests appear even before the functional logic and thus are written as a kind of specification. For the technical difference ignored tests are not executed at all while pending ones are executed until reaching the TestPendingException thrown by org.scalatest.Assertions#pending() method:

class PendingTestFunSuite extends FunSuite {

  test("a new document should be persisted in the local storage")(pending)


  test("a new document should be persisted in the local storage with some initial implementation") {
    println("Before calling pending method...")

    pending
  }

  ignore("this test should be ignored") {
    println("Before calling ignored test case...")
  }
}

The snippet above outputs:

Test Pending
Before calling pending method...

Test Pending

Test Ignored

Of course, the way of defining tests as pending varies with the used family and for FlatSpec it will be:

class PendingTestFlatSpec extends FlatSpec {

  "a new document" should "be persisted in the local storage" in (pending)

}

ScalaTest brings several facility methods helping to define the tests easier and make them more maintainable. We can use custom tags to classify test executions or one of available fixtures methods to avoid the common setup and cleaning code duplication. We can also go further and test our code against a set of dynamically constructed values. The BDD also has its points in the extended features with given/when/then mixin and the pending tag letting to define the acceptance tests before the implementation.

TAGS: #BDD #ScalaTest