Testing Java Projects

This page will discuss common topics around working with test suites using the Mill build tool

Defining Unit Test Suites

package build
import mill._, javalib._

object foo extends JavaModule {
  object test extends JavaTests {
    def testFramework = "com.novocode.junit.JUnitFramework"
    def ivyDeps = Agg(
      ivy"com.novocode:junit-interface:0.11",
      ivy"org.mockito:mockito-core:4.6.1"
    )
  }
}

scala

foo/src/foo/Foo.java (browse)

package foo;

public class Foo {
  public static void main(String[] args) {
    System.out.println(new Foo().hello());
  }

  public String hello() {
    return "Hello World";
  }
}

scala

foo/test/src/foo/FooTests.java (browse)

package foo;

import static org.junit.Assert.assertTrue;
import static org.mockito.Mockito.*;

import org.junit.Test;

public class FooTests {

  @Test
  public void hello() {
    String result = new Foo().hello();
    assertTrue(result.startsWith("Hello"));
  }

  @Test
  public void world() {
    String result = new Foo().hello();
    assertTrue(result.endsWith("World"));
  }

  @Test
  public void testMockito() {
    Foo mockFoo = mock(Foo.class);

    when(mockFoo.hello()).thenReturn("Hello Mockito World");

    String result = mockFoo.hello();

    assertTrue(result.equals("Hello Mockito World"));
    verify(mockFoo).hello();
  }
}

scala

This build defines a single module with a test suite, configured to use "JUnit" as the testing framework, along with Mockito. Test suites are themselves JavaModules, nested within the enclosing module, and have all the normal tasks like foo.test.compile available to run, but with an additional .test task that runs the tests. You can also run the test suite directly, in which case it will run the .test task as the default task for that module.

> mill foo.compile
compiling 1 ... source...

> mill foo.test.compile
compiling 1 ... source...

> mill foo.test.test
...foo.FooTests...hello ...
...foo.FooTests...world ...

> mill foo.test # same as above, `.test` is the default task for the `test` module
...foo.FooTests...hello ...
...foo.FooTests...world ...

> mill foo.test.testOnly foo.FooTests # explicitly select the test class you wish to run
...foo.FooTests...hello ...
...foo.FooTests...world ...

bash

For convenience, you can also use one of the predefined test frameworks:

TestModule.Junit4, using sbt/junit-interface
TestModule.Junit5, using sbt/sbt-jupiter-interface
TestModule.TestNg, using TestNg
TestModule.Munit, using munit
TestModule.ScalaTest, using ScalaTest
TestModule.Specs2, using Specs2
TestModule.Utest, using uTest
TestModule.ZioTest, using ZioTest

Each testing framework has their own flags and configuration options that are documented on their respective websites, so please see the links above for more details on how to use each one from the command line.

object bar extends JavaModule {
  object test extends JavaTests with TestModule.Junit4 {
    def ivyDeps = super.ivyDeps() ++ Agg(
      ivy"org.mockito:mockito-core:4.6.1"
    )
  }
}

scala

> mill bar.test
...bar.BarTests...hello ...
...bar.BarTests...world ...

bash

You can also select multiple test suites in one command using Mill’s Task Query Syntax

> mill __.test
...bar.BarTests...hello ...
...bar.BarTests...world ...
...foo.FooTests...hello ...
...foo.FooTests...world ...
...

bash

Mill provides three ways of running tests

> mill foo.test

bash

foo.test: runs tests in a subprocess in an empty sandbox/ folder. This is short for foo.test.test, as test is the default task for TestModules.

> mill foo.test.testCached

bash

foo.test.testCached: runs the tests in an empty sandbox/ folder and caches the results if successful. This can be handy if you are you working on some upstream modules and only want to run downstream tests which are affected: using testCached, downstream tests which are not affected will be cached after the first run and not re-run unless you change some file upstream of them.

> mill foo.test.testLocal

bash

foo.test.testLocal: runs tests in an isolated classloader as part of the main Mill process. This can be faster than .test, but is less flexible (e.g. you cannot pass forkEnv) and more prone to interference (testLocal runs do not run in sandbox folders)

It is common to run tests with -w/--watch enabled, so that once you edit a file on disk the selected tests get re-run.

Mill runs tests with the working directory set to an empty sandbox/ folder by default. Tests can access files from their resource directory via the environment variable MILL_TEST_RESOURCE_DIR which provides the path to the resource folder, and additional paths can be provided to test via forkEnv. See Classpath and Filesystem Resources for more details.

If you want to pass any arguments to the test framework, you can pass them after foo.test in the command line. e.g. uTest lets you pass in a selector to decide which test to run, which in Mill would be:

> mill bar.test bar.BarTests.hello
...bar.BarTests...hello ...

bash

This command only runs the hello test case in the bar.BarTests test suite class.

Test Dependencies

Mill has no test-scoped dependencies!

You might be used to test-scoped dependencies from other build tools like Maven, Gradle or SBT. As test modules in Mill are just regular modules, there is no special need for a dedicated test-scope. You can use ivyDeps and runIvyDeps to declare dependencies in test modules, and test modules can use their moduleDeps to also depend on each other

build.mill (download, browse)

package build
import mill._, javalib._

object qux extends JavaModule {
  def moduleDeps = Seq(baz)

  object test extends JavaTests with TestModule.Junit4 {
    def moduleDeps = super.moduleDeps ++ Seq(baz.test)
    def ivyDeps = super.ivyDeps() ++ Agg(ivy"com.google.guava:guava:33.3.0-jre")
  }
}

object baz extends JavaModule {
  object test extends JavaTests with TestModule.Junit4 {
    def ivyDeps = super.ivyDeps() ++ Agg(ivy"com.google.guava:guava:33.3.0-jre")
  }
}

scala

In this example, not only does qux depend on baz, but we also make qux.test depend on baz.test.

That lets qux.test make use of the BazTestUtils class that baz.test defines, allowing us to re-use this test helper throughout multiple modules' test suites

> ./mill qux.test
Using BazTestUtils.bazAssertEquals
... qux.QuxTests...simple ...
...

> ./mill baz.test
Using BazTestUtils.bazAssertEquals
... baz.BazTests...simple ...
...

bash

Defining Integration Test Suites

You can also define test suites with different names other than test. For example, the build below defines a test suite with the name integration, in addition to that named test.

build.mill (download, browse)

package build
import mill._, javalib._
object qux extends JavaModule {
  object test extends JavaTests with TestModule.Junit5
  object integration extends JavaTests with TestModule.Junit5
}

scala

The integration suite is just another regular test module within the parent JavaModule (This example also demonstrates using Junit 5 instead of Junit 4)

These two test modules will expect their sources to be in their respective qux/test and qux/integration folder respectively

> mill 'qux.{test,integration}' # run both test suites
...qux.QuxTests...hello...
...qux.QuxTests...world...
...qux.QuxIntegrationTests...helloworld...

> mill __.integration.testCached # run all integration test suites

bash

Test Parallelism

Test parallelism is an opt-in that enables parallel test execution, automatically distributing your test classes across multiple JVM subprocesses, while minimizing the overhead of JVM creation by re-using the subprocesses where possible. Test parallelism should be able to reduce testing times for most project’s test suites

Test parallelism can be enabled by overriding def testParallelism, as demonstrated below. This feature is expected to be enabled by default in future versions of Mill.

build.mill (download, browse)

package build
import mill._, javalib._

object foo extends JavaModule {
  object test extends JavaTests {
    def testFramework = "com.novocode.junit.JUnitFramework"
    def ivyDeps = Agg(
      ivy"com.novocode:junit-interface:0.11",
      ivy"org.mockito:mockito-core:4.6.1"
    )
    def testParallelism = true
  }
}

scala

> mill -j 3 foo.test

> find out/foo/test/test.dest
...
out/foo/test/test.dest/worker-0.log
out/foo/test/test.dest/worker-0
out/foo/test/test.dest/worker-1.log
out/foo/test/test.dest/worker-1
out/foo/test/test.dest/worker-2.log
out/foo/test/test.dest/worker-2
out/foo/test/test.dest/test-classes
out/foo/test/test.dest/test-report.xml
...

bash

Test Grouping

Test Grouping is an opt-in feature that allows you to take a single test module and group the test classes such that each group will execute in parallel in a separate JVM when you call test. Test grouping is enabled by overriding def testForkGrouping, as shown below:

build.mill (download, browse)

package build
import mill._, javalib._

object foo extends JavaModule {
  object test extends JavaTests {
    def testFramework = "com.novocode.junit.JUnitFramework"
    def ivyDeps = Agg(
      ivy"com.novocode:junit-interface:0.11",
      ivy"org.mockito:mockito-core:4.6.1"
    )
    def testForkGrouping = discoveredTestClasses().grouped(1).toSeq
  }
}

scala

foo/test/src/foo/HelloTests.java (browse)

package foo;

import static org.junit.Assert.assertTrue;

import org.junit.Test;

public class HelloTests {

  @Test
  public void hello() throws Exception {
    System.out.println("Testing Hello");
    String result = new Foo().hello();
    assertTrue(result.startsWith("Hello"));
    Thread.sleep(1000);
    System.out.println("Testing Hello Completed");
  }
}

scala

foo/test/src/foo/WorldTests.java (browse)

package foo;

import static org.junit.Assert.assertTrue;

import org.junit.Test;

public class WorldTests {
  @Test
  public void world() throws Exception {
    System.out.println("Testing World");
    String result = new Foo().hello();
    assertTrue(result.endsWith("World"));
    Thread.sleep(1000);
    System.out.println("Testing World Completed");
  }
}

scala

In this example, we have one test module foo.test, and two test classes HelloTests and WorldTests. By default, all test classes in the same module run sequentially in the same JVM, but with testForkGrouping we can break up the module and run each test class in parallel in separate JVMs, each with their own separate sandbox folder and .log file:

> mill foo.test

> find out/foo/test/test.dest
...
out/foo/test/test.dest/foo.HelloTests.log
out/foo/test/test.dest/foo.HelloTests/sandbox
out/foo/test/test.dest/foo.WorldTests.log
out/foo/test/test.dest/foo.WorldTests/sandbox
out/foo/test/test.dest/test-report.xml

bash

Compared to Test Parallelism, test grouping allows you to run tests in parallel while and isolating different test groups into their own subprocesses, at the cost of greater subprocess overhead due to the larger number of isolated subprocesses. Different test groups will not write over each others files in their sandbox, and each one will have a separate set of logs that can be easily read without the others mixed in

In this example, def testForkGrouping = discoveredTestClasses().grouped(1).toSeq assigns each test class to its own group, running in its own JVM. You can also configure testForkGrouping to choose which test classes you want to run together and which to run alone:

If some test classes are much slower than others, you may want to put the slow test classes each in its own group to reduce latency, while grouping multiple fast test classes together to reduce the per-group overhead of spinning up a separate JVM.
Some test classes may have global JVM-wide or filesystem side effects that means they have to run alone, while other test classes may be better-behaved and OK to run in a group

In general, testForkGrouping leaves it up to you how you want to group your tests for execution, based on the unique constraints of your test suite.

Test Grouping & Test Parallelism together

testParallelism respects testForkGrouping, allowing you to use both features in a test module.

build.mill (download, browse)

package build
import mill._, javalib._

object foo extends JavaModule {
  object test extends JavaTests {
    def testFramework = "com.novocode.junit.JUnitFramework"
    def ivyDeps = Agg(
      ivy"com.novocode:junit-interface:0.11",
      ivy"org.mockito:mockito-core:4.6.1"
    )
    def testForkGrouping =
      discoveredTestClasses().groupMapReduce(_.contains("GroupX"))(Seq(_))(_ ++ _).toSeq.sortBy(
        data => !data._1
      ).map(_._2)
    def testParallelism = true
  }
}

scala

> mill -j 4 foo.test

> find out/foo/test/test.dest
...
out/foo/test/test.dest/group-0-foo.GroupX1/worker-...
out/foo/test/test.dest/group-0-foo.GroupX1/test-classes
out/foo/test/test.dest/group-1-foo.GroupY1/worker-...
out/foo/test/test.dest/group-1-foo.GroupY1/test-classes
out/foo/test/test.dest/test-report.xml
...

bash

This example sets testForkGrouping to group test classes into two categories: GroupX and GroupY. Additionally, testParallelism is enabled. Mill ensures each subprocess exclusively claims and runs tests from either GroupX or GroupY, preventing them from mixing. Test classes from GroupX and GroupY will never share the same test runner.

This is useful when you have incompatible tests that cannot run within the same JVM. Test Grouping combined with Test Parallel Scheduler maintains their isolation while maximizing performance through parallel test execution.

Github Actions Test Reports

If you use Github Actions for CI, you can use https://github.com/mikepenz/action-junit-report in your pipeline to render the generated test-report.xml files nicely on Github. See https://github.com/com-lihaoyi/mill/pull/4218/files for an example integration