Testing Scala Projects

This page will discuss common topics around working with test suites using the Mill build tool

Defining Unit Test Suites

package build
import mill._, scalalib._

object foo extends ScalaModule {
  def scalaVersion = "2.13.8"
  object test extends ScalaTests {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
    def testFramework = "utest.runner.Framework"
  }
}

This build defines a single module with a test suite, configured to use "uTest" as the testing framework. Test suites are themselves ScalaModules, nested within the enclosing module, and have all the normal tasks like foo.test.compile available to run, but with an additional .test task that runs the tests. You can also run the test suite directly, in which case it will run the .test task as the default task for that module.

> mill foo.compile
compiling 1 ... source...

> mill foo.test.compile
compiling 1 ... source...

> mill foo.test.test
...foo.FooTests.hello ...
...foo.FooTests.world ...

> mill foo.test
...foo.FooTests.hello ...
...foo.FooTests.world ...

For convenience, you can also use one of the predefined test frameworks:

TestModule.Junit4
TestModule.Junit5
TestModule.TestNg
TestModule.Munit
TestModule.ScalaTest
TestModule.Specs2
TestModule.Utest
TestModule.ZioTest

object bar extends ScalaModule {
  def scalaVersion = "2.13.8"

  object test extends ScalaTests with TestModule.Utest {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
  }
}

> mill bar.test
...bar.BarTests.hello ...
...bar.BarTests.world ...

You can also select multiple test suites in one command using Mill’s Target Query Syntax

> mill __.test
...bar.BarTests.hello ...
...bar.BarTests.world ...
...foo.FooTests.hello ...
...foo.FooTests.world ...
...

Mill provides three ways of running tests

foo.test.test: runs tests in a subprocess in an empty sandbox/ folder.
foo.test.testCached: runs the tests in an empty sandbox/ folder and caches the results if successful. Also allows multiple test modules to be run in parallel e.g. via mill __.testCached
foo.test.testLocal: runs tests in an isolated classloader as part of the main Mill process. This can be faster than .test, but is less flexible (e.g. you cannot pass forkEnv) and more prone to interference (due to sharing the sandbox/ folder provided by the Mill process)

> mill bar.test.test

> mill bar.test.testCached

> mill bar.test.testLocal

Mill provides three ways of running tests

foo.test.test: runs tests in a subprocess in an empty sandbox/ folder, and forkArg and forkEnv can be overridden to pass JVM flags & environment variables.
foo.test.testCached: runs the tests in an empty sandbox/ folder and caches the results if successful. Also allows multiple test modules to be run in parallel e.g. via mill __.testCached. Again, forkEnv and forkArgs can be configured.
foo.test.testLocal: runs tests in an isolated classloader as part of the main Mill process. This can be faster than .test, but is less flexible (e.g. you cannot pass forkEnv) and more prone to interference (due to sharing the sandbox/ folder provided by the Mill process)

> mill bar.test.test

> mill bar.test.testCached

> mill bar.test.testLocal

Note that Mill runs tests with the working directory set to an empty sandbox/ folder by default. Tests can access files from their resource directory via the environment variable MILL_TEST_RESOURCE_FOLDER which provides the path to the resource folder, and additional paths can be provided to test via forkEnv. See Classpath and Filesystem Resources for more details.

If you want to pass any arguments to the test framework, simply put them after foo.test in the command line. e.g. uTest lets you pass in a selector to decide which test to run, which in Mill would be:

> mill bar.test bar.BarTests.hello
...bar.BarTests.hello ...

Test Dependencies

Mill has no test-scoped dependencies!

You might be used to test-scoped dependencies from other build tools like Maven, Gradle or sbt. As test modules in Mill are just regular modules, there is no special need for a dedicated test-scope. You can use ivyDeps and runIvyDeps to declare dependencies in test modules, and test modules can use their moduleDeps to also depend on each other

build.mill (download, browse)

package build
import mill._, scalalib._

object qux extends ScalaModule {
  def scalaVersion = "2.13.8"
  def moduleDeps = Seq(baz)

  object test extends ScalaTests {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
    def testFramework = "utest.runner.Framework"
    def moduleDeps = super.moduleDeps ++ Seq(baz.test)
  }
}


object baz extends ScalaModule {
  def scalaVersion = "2.13.8"

  object test extends ScalaTests {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
    def testFramework = "utest.runner.Framework"
  }
}

In this example, not only does qux depend on baz, but we also make qux.test depend on baz.test. That lets qux.test make use of the BazTestUtils class that baz.test defines, allowing us to re-use this test helper throughout multiple modules' test suites

> ./mill qux.test
Using BazTestUtils.bazAssertEquals
... qux.QuxTests.simple ...
...

> ./mill baz.test
Using BazTestUtils.bazAssertEquals
... baz.BazTests.simple ...
...

Defining Integration Test Suites

You can also define test suites with different names other than test. For example, the build below defines a test suite with the name integration, in addition to that named test.

build.mill (download, browse)

package build
import mill._, scalalib._

object qux extends ScalaModule {
  def scalaVersion = "2.13.8"

  object test extends ScalaTests with TestModule.Utest {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
  }
  object integration extends ScalaTests with TestModule.Utest {
    def ivyDeps = Agg(ivy"com.lihaoyi::utest:0.8.4")
  }
}

These two test modules will expect their sources to be in their respective foo/test and foo/integration folder respectively

> mill 'qux.{test,integration}' # run both test suites
...qux.QuxTests...hello...
...qux.QuxTests...world...
...qux.QuxIntegrationTests...helloworld...

> mill __.integration.testCached # run all integration test suites