ScalaTest and testing styles

Versions: ScalaTest 3.0.3

At first glance the wide choice of testing families in Scala can scary. After all in JUnit and other xUnit frameworks, the choice of tests declaration is limited. Hopefully after some digging ScalaTest's testing styles become more obvious to understand and to use.

This post describes the test families available in ScalaTest framework. It's divided in 3 parts. The first one represents the code used in the examples. The second one lists and explains testing styles available in the framework. The final part analyzes which styles and how are used in 2 important Open Source Scala projects: Apache Spark and Akka.

Tested code

To illustrate test families we'll use the code below representing a simplified e-commerce shopping cart:

case class ShoppingCart(products: HashSet[Product] = new HashSet[Product](),
                        voucherCode: VoucherCode = ShoppingCart.NoVoucherCode,
                        deliveryMode: Option[DeliveryMode] = None) {

  def totalPrice: Double = (products.map(_.price).sum + deliveryMode.get.price) * voucherCode.discountPercentage

  def addProduct(product: Product): ShoppingCart = {
    val newProductsList = products + product
    this.copy(products = newProductsList)
  }

  def removeProduct(product: Product): ShoppingCart = {
    val newProductsList = products - product
    this.copy(products = newProductsList)
  }

  def setDeliveryMode(newDeliveryMode: DeliveryMode): ShoppingCart = this.copy(deliveryMode=Some(newDeliveryMode))

  def setVoucherCode(newVoucherCode: VoucherCode): ShoppingCart = this.copy(voucherCode=newVoucherCode)

}

object ShoppingCart {
  val NoVoucherCode = VoucherCode("no_code", 1.0)
}

case class Product(name: String, price: Double)

case class VoucherCode(code: String, discountPercentage: Double) {
  assert(discountPercentage <= 1.0, "The discount percentage can't be greater than 100%")
}

case class DeliveryMode(name: String, price: Double)

ScalaTest testing styles

ScalaTest is Scala's popular testing framework. It's characterized by a lot of flexibility in terms of tests definition. This flexibility is brought mainly by the testing styles that can be adapted as well for classical unit testing (xUnit-based) as for more descriptive BDD. These testing styles are expressed by following style traits:

• FunSuite - it's the most basic trait and maybe the most intuitive for the people coming from classical xUnit environment. The test cases are defined inside test("...") method allowing nonetheless a pretty nice descriptiveness through text-like test name. The FunSuite has a flat structure, i.e. tests are defined one after another without any nesting places. An example of this style could be:
  

  class ShoppingCartFunSuiteTest extends FunSuite with Matchers {
  
    private val Pineapple = Product("pineapple", 1.99)
  
    private val Apple = Product("apple", 0.5)
  
    test("an empty shopping cart should add new product to the list") {
      val emptyCart = ShoppingCart()
  
      val cartWithPineapple = emptyCart.addProduct(Pineapple)
  
      cartWithPineapple.products should have size 1
      cartWithPineapple.products.head shouldEqual(Pineapple)
    }
  
    test("a not empty shopping cart should add new product to the list") {
      val cartWithPineapple = ShoppingCart(HashSet(Pineapple))
  
      val cartWithPineappleAndApple = cartWithPineapple.addProduct(Apple)
  
      cartWithPineappleAndApple.products should have size 2
      cartWithPineappleAndApple.products should contain allOf(Pineapple, Apple)
    }
  
    test("a not empty shopping cart should remove already added product") {
      val cartWithPineapple = ShoppingCart(HashSet(Pineapple))
  
      val emptyCart = cartWithPineapple.removeProduct(Pineapple)
  
      emptyCart.products shouldBe empty
    }
  
  }
  

• FlatSpec - this style introduces basic concepts of BDD. So it's well suited for the people with the will to use the BDD but who don't master it. It's translated by the specification-like language composed of: tested object, test verb and expected action. The former one is called subject and defines the entity being tested. The verb can be one of: should, must or can and it helps to define the purpose of the test. The expected action tells what is verified. For instance we could verify the emptiness of a string with that grammar: an empty string must return 0 as its length.
  The BDD character is also emphasized by the instruction behavior of "Subject" and following it + verb + action methods. This very descriptive method defines the expected behavior for tested subject.
  

  class ShoppingCartFlatSpecTest extends FlatSpec with Matchers {
  
    private val Peach = Product("peach", 1.25)
    private val Banana = Product("banana", 0.99)
  
    behavior of "a shopping cart"
  
    it should "accept new product when it's empty" in {
      val emptyCart = ShoppingCart()
  
      val cartWithPeach = emptyCart.addProduct(Peach)
  
      cartWithPeach.products should have size 1
      cartWithPeach.products.head shouldEqual(Peach)
    }
  
    it should "accept new product when it's not empty" in {
      val cartWithPeach = ShoppingCart(HashSet(Peach))
  
      val cartWithPineappleAndApple = cartWithPeach.addProduct(Banana)
  
      cartWithPineappleAndApple.products should have size 2
      cartWithPineappleAndApple.products should contain allOf(Peach, Banana)
    }
  
    it should "remove already added product" in {
      val cartWithPeach = ShoppingCart(HashSet(Peach))
  
      val emptyCart = cartWithPeach.removeProduct(Peach)
  
      emptyCart.products shouldBe empty
    }
  
  }
  

• FunSpec - it's the first style fully belonging to the BDD family. It favors the specification style where the described and tested elements are mixed with describe("...") and it("...") or they("...") (depending if the subject is plural or singular) instructions. They're used as test guideline to define the structure of the test.
  

  class ShoppingCartFunSpecTest extends FunSpec with Matchers {
  
    private val Chocolate = Product("chocolate", 1.90)
    private val Milk = Product("milk", 0.5)
  
    describe("a shopping cart") {
      describe("when is empty") {
        val emptyShoppingCart = ShoppingCart()
        it("should add new product") {
          val shoppingCartWithMilk = emptyShoppingCart.addProduct(Milk)
  
          shoppingCartWithMilk.products should have size 1
          shoppingCartWithMilk.products.head shouldEqual(Milk)
        }
      }
      describe("when is not empty") {
        val shoppingCartWithChocolate = ShoppingCart(HashSet(Chocolate))
        it("should add new product") {
          val shoppingCartWithChocolateAndMilk = shoppingCartWithChocolate.addProduct(Milk)
  
          shoppingCartWithChocolateAndMilk.products should have size 2
          shoppingCartWithChocolateAndMilk.products should contain allOf(Chocolate, Milk)
        }
        it("should remove already added product") {
          val emptyShoppingCart = shoppingCartWithChocolate.removeProduct(Chocolate)
  
          emptyShoppingCart.products shouldBe empty
        }
      }
    }
  
  }
  

• WordSpec - is another BDD-like style and the name comes from the fact that the instructions are always preceded by the strings that can represent either tested situation or the subject. The test starts by the definition of the subject followed either by the instruction defining possible cases (when) or by the instruction defining the expectation character (should/must/can). Later, depending on the previously used instruction, we can either list the tested cases or define the body of the case. The body is defined thanks to the in operator.
  

  class ShoppingCartWordSpecTest extends WordSpec with Matchers {
  
    private val Fish = Product("Fish", 5)
    private val PotatoSalad = Product("potato salad", 6)
  
    "A shopping cart" when {
      "empty" should {
        val emptyShoppingCart = ShoppingCart()
        "accept new product" in {
          val cartWithFish = emptyShoppingCart.addProduct(Fish)
  
          cartWithFish.products should have size 1
          cartWithFish.products.head shouldEqual(Fish)
        }
      }
      "not empty" should {
        val cartWithFish = ShoppingCart(HashSet(Fish))
        "accept new product" in {
          val cartWithFishAndPotatoSalad = cartWithFish.addProduct(PotatoSalad)
  
          cartWithFishAndPotatoSalad.products should have size 2
          cartWithFishAndPotatoSalad.products should contain allOf(PotatoSalad, Fish)
        }
        "remove already added product" in {
          val emptyShoppingCart = cartWithFish.removeProduct(Fish)
  
          emptyShoppingCart.products shouldBe empty
        }
      }
    }
  
  }
  

• FreeSpec - this another BDD-style brings more definition flexibility than the above ones. The test is composed as a sequence of nested blocks built of a text followed by dash. The body for the test is defined inside a block preceded by "in" operation. The test can have any level of nested descriptions so it's well suited for the teams having an experience with BDD and shared testing standards:
  

  class ShoppingCartFreeSpecTest extends FreeSpec with Matchers {
  
    private val Bread = Product("bread", 0.80)
    private val Ham = Product("ham", 1.50)
  
    "A shopping cart" - {
      "when empty" - {
        val emptyShoppingCart = ShoppingCart()
        "should add new product" in {
          val cartWithBread = emptyShoppingCart.addProduct(Bread)
  
          cartWithBread.products should have size 1
          cartWithBread.products.head shouldEqual(Bread)
        }
      }
      "when not empty" - {
        val cartWithBread = ShoppingCart(HashSet(Bread))
        "should add new product" in {
          val cartWithBreadAndHam = cartWithBread.addProduct(Ham)
  
          cartWithBreadAndHam.products should have size 2
          cartWithBreadAndHam.products should contain allOf(Bread, Ham)
        }
        "should remove already added product" in {
          val emptyShoppingCart = cartWithBread.removeProduct(Bread)
  
          emptyShoppingCart.products shouldBe empty
        }
      }
    }
  
  }
  

• PropSpec - the name of this style comes from "property" and it's linked to the property-based tests. This kind of tests verify the behavior of tested method against a set of properties. Each property test is defined inside property("...") method. The tested values are wrapped inside this method by a forAll("...") function followed by the block containing the test logic.
  

  class ShoppingCartPropSpecTest extends PropSpec with TableDrivenPropertyChecks with Matchers {
  
    private val Butter = Product("butter", 1.25)
    private val Cheese = Product("cheese", 2.20)
  
    private val ShoppingCartsToAddTest = Table(
      "shopping carts with with items to add",
      ShoppingCart(),
      ShoppingCart(HashSet(Butter))
    )
  
    private val ShoppingCartsToRemoveTest = Table(
      "shopping carts with items to remove",
      ShoppingCart(HashSet(Butter))
    )
  
    property("a shopping cart should accept new product") {
      forAll(ShoppingCartsToAddTest) { shoppingCart =>
        val shoppingCartWithCheese = shoppingCart.addProduct(Cheese)
  
        shoppingCartWithCheese.products.size shouldEqual shoppingCart.products.size + 1
        shoppingCartWithCheese.products should contain(Cheese)
      }
    }
  
    property("a shopping cart should remove product") {
      forAll(ShoppingCartsToRemoveTest) { shoppingCart =>
        val shoppingCartWithoutButter = shoppingCart.removeProduct(Butter)
  
        shoppingCartWithoutButter.products should not (contain(Butter))
      }
    }
  
  }

• FeatureSpec - it's main purpose are acceptance tests. The test is built on one or more feature containing one or multiple scenarios. The former one defines the tested entity. The latter ones are used to define the tested behavior. The methods like info("..."), written before the features, help to give more descriptive context for the execution and thus to work programmers and non programmers together during the test case definition:
  

  class ShoppingCartFeatureSpecTest extends FeatureSpec with Matchers {
  
    private val Yogurt = Product("yogurt", 1.33)
    private val Kiwifruit = Product("kiwifruit", 0.25)
  
    info("as e-commerce customer")
    info("I want to manipulate the products in my shopping cart")
  
    feature("shopping cart") {
      scenario("customer adds new product to empty cart") {
        val emptyCart = ShoppingCart()
  
        val cartWithYogurt = emptyCart.addProduct(Yogurt)
  
        cartWithYogurt.products should have size 1
        cartWithYogurt.products.head shouldEqual(Yogurt)
      }
      scenario("customer adds new product to not empty cart") {
        val cartWithYogurt = ShoppingCart(HashSet(Yogurt))
  
        val cartWithYogurtAndKiwifruit = cartWithYogurt.addProduct(Kiwifruit)
  
        cartWithYogurtAndKiwifruit.products should have size 2
        cartWithYogurtAndKiwifruit.products should contain allOf(Yogurt, Kiwifruit)
      }
      scenario("customer removes already added product from not empty cart") {
        val cartWithYogurt = ShoppingCart(HashSet(Yogurt))
  
        val emptyCart = cartWithYogurt.removeProduct(Yogurt)
  
        emptyCart.products shouldBe empty
      }
    }
  
  }
  

• RefSpec - this style is also used for BDD testing. The difference with the previously described ones is the performance. The tests in RefSpec are defined as methods within singleton objects. It can lead to faster build time (fewer function literals + fewer generated classes) that can be useful in large projects where this aspect does matter a lot.
  

  class ShoppingCartRefSpecTest extends RefSpec with Matchers {
  
    private val Eggs = Product("eggs", 1.15)
    private val Chicken = Product("chicken", 3)
  
    object `A shopping cart` {
      object `when empty` {
        val emptyCart = ShoppingCart()
        def `should add new product` = {
          val cartWithEggs = emptyCart.addProduct(Eggs)
  
          cartWithEggs.products should have size 1
          cartWithEggs.products.head shouldEqual(Eggs)
        }
      }
      object `with one added product` {
        val cartWithEggs = ShoppingCart(HashSet(Eggs))
        def `should add another product` = {
          val cartWithEggsAndChicken = cartWithEggs.addProduct(Chicken)
  
          cartWithEggsAndChicken.products should have size 2
          cartWithEggsAndChicken.products should contain allOf(Chicken, Eggs)
        }
        def `should remove the added product` = {
          val emptyCart = cartWithEggs.removeProduct(Eggs)
  
          emptyCart.products shouldBe empty
        }
      }
    }
  
  }
  

Testing styles in OS projects

Let's start the analysis of the OS projects using ScalaTest with Apache Spark. The tests in this data processing framework are organized around org.apache.spark.SparkFunSuite . This abstract class extends FunSuite and mixes with BeforeAndAfterAll, ThreadAudit and Logging traits. The purpose of SparkFunSuite is to handle the functionalities shared by the tests as: setting spark.testing to true or cleaning accumulators context. A single exception not using SparkFunSuite directly is org.apache.spark.sql.RowTest that doesn't need the Spark context to execute the tests.

Spark also works with other testing styles as FlatSpec, FunSpec and WordSpec. But, unlike FunSuite, they're not directly used in the written tests. Instead they are defined as the assertions checking if SharedSparkContext works correctly if other testing styles than FunSuite are used by the client applications.

In Akka we can find the preference for WordSpec. It's either used as a base class or, as in the case of SparkFunSuite, as a parent for more specific Akka testing styles (e.g. akka.testkit.typed.scaladsl.AbstractActorSpec). But it was not always true. In the past some tests were using FlatSpec. But it was not estimated good for the codebase consistency and some work was done to keep only WordSpec.

At first glance ScalaTest testing styles seem to be complicated because of their quantity. However after some digging the ideas become more clear. As we could learn, a part of styles are BDD-oriented and another part is more appropriate for the teams coming from classical xUnit ground. Even though the variety of styles, the main OS projects based on Scala prefers always 1 testing style. For Apache Spark it's FunSuite brought with SparkFunSuite base class. Akka in its turn prefers WordSpec.