Testing Python Projects

This page will discuss topics around defining and running Python tests using the Mill build tool

Defining Unit Test Suites

build.mill (download, browse)
package build
import mill._, pythonlib._

object foo extends PythonModule {
  def mainScript = Task.Source { millSourcePath / "src/foo.py" }
  object test extends PythonTests with TestModule.Unittest {
    def mainScript = Task.Source { millSourcePath / "src/test_foo.py" }
  }
}

This build defines a single module with a test suite, configured to use "Unittest" as the testing framework. Test suites are themselves PythonModules, nested within the enclosing module, and have all the normal tasks available to run, but with an additional .test task that runs the tests. You can also run the test suite directly, in which case it will run the .test task as the default task for that module.

> ./mill foo.run
Hello World

> ./mill foo.test.test
test_hello (test_foo.TestScript...) ... ok
test_mock (test_foo.TestScript...) ... ok
test_world (test_foo.TestScript...) ... ok
...Ran 3 tests...
OK

> ./mill foo.test # same as above, `.test` is the default task for the `test` module
...Ran 3 tests...
OK

> ./mill foo.test test_foo.TestScript.test_mock # explicitly select the test class you wish to run
test_mock (test_foo.TestScript...) ... ok
...Ran 1 test...
OK

For convenience, you can also use one of the predefined test frameworks:

Each testing framework has their own flags and configuration options that are documented on their respective websites, so please see the links above for more details on how to use each one from the command line.

object bar extends PythonModule {
  def mainScript = Task.Source { millSourcePath / "src/bar.py" }
  object test extends PythonTests with TestModule.Pytest
}
> ./mill bar.test
...test_bar.py::test_hello PASSED...
...test_bar.py::test_world PASSED...
...test_bar.py::test_mock PASSED...
...3 passed...

You can also select multiple test suites in one command using Mill’s Task Query Syntax

> ./mill __.test
test_hello (test_foo.TestScript...) ... ok
test_mock (test_foo.TestScript...) ... ok
test_world (test_foo.TestScript...) ... ok
...Ran 3 tests...
OK
...test_bar.py::test_hello PASSED...
...test_bar.py::test_world PASSED...
...test_bar.py::test_mock PASSED...
...3 passed...

Mill provides multiple ways of running tests

> ./mill foo.test
test_hello (test_foo.TestScript...) ... ok
test_mock (test_foo.TestScript...) ... ok
test_world (test_foo.TestScript...) ... ok
...Ran 3 tests...
OK

  • foo.test: runs tests in a subprocess in an empty sandbox/ folder. This is short for foo.test.test, as test is the default task for TestModules.

> ./mill foo.test.testCached
test_hello (test_foo.TestScript...) ... ok
test_mock (test_foo.TestScript...) ... ok
test_world (test_foo.TestScript...) ... ok
...Ran 3 tests...
OK

  • foo.test.testCached: runs the tests in an empty sandbox/ folder and caches the results if successful. This can be handy if you are you working on some upstream modules and only want to run downstream tests which are affected: using testCached, downstream tests which are not affected will be cached after the first run and not re-run unless you change some file upstream of them.

It is common to run tests with -w/--watch enabled, so that once you save a file to disk the selected tests get re-run.

Mill runs tests with the working directory set to an empty sandbox/ folder by default. Additional paths can be provided to test via forkEnv. See Pythonpath and Filesystem Resources for more details.

If you want to pass any arguments to the test framework, you can pass them after foo.test in the command line. e.g. Pytest lets you pass in a selector to decide which test to run, which in Mill would be:

> ./mill bar.test bar/test/src/test_bar.py::test_mock # explicitly select the test class you wish to run
...test_bar.py::test_mock PASSED...
...1 passed...

This command only runs the test_mock test case in the bar.test test suite class.

Test Dependencies

You can use pythonDeps to declare dependencies in test modules, and test modules can use their moduleDeps to also depend on each other

build.mill (download, browse)
package build
import mill._, pythonlib._

object foo extends PythonModule {

  def mainScript = Task.Source { millSourcePath / "src/foo.py" }

  def moduleDeps = Seq(bar)

  def pythonDeps = Seq("jinja2==3.1.4")

  object test extends PythonTests with TestModule.Unittest {
    def moduleDeps = super.moduleDeps ++ Seq(bar.test)

    def pythonDeps = Seq("MarkupSafe==3.0.2")

  }

}

object bar extends PythonModule {

  def mainScript = Task.Source { millSourcePath / "src/bar.py" }

  object test extends PythonTests with TestModule.Unittest

}

In this example, not only does foo depend on bar, but we also make foo.test depend on bar.test.

G bar bar bar.test bar.test bar->bar.test foo foo bar->foo foo.test foo.test bar.test->foo.test foo->foo.test

That lets foo.test make use of the BarTestUtils class that bar.test defines, allowing us to re-use this test helper throughout multiple modules test suites

> ./mill foo.test
...Using BarTestUtils.bar_assert_equals...
...test_equal_string (test.TestScript...)...ok...
...Ran 1 test...
...OK...

> ./mill bar.test
...Using BarTestUtils.bar_assert_equals...
...test_mean (test.TestScript...)...ok...
...Ran 1 test...
...OK...

> ./mill foo.run
<h1><XYZ></h1>

> ./mill bar.run
123

Defining Integration Test Suites

You can also define test suites with different names other than test. For example, the build below defines a test suite with the name itest, in addition to that named test.

build.mill (download, browse)
package build
import mill._, pythonlib._

object foo extends PythonModule {

  def mainScript = Task.Source { millSourcePath / "src/foo.py" }

  object test extends PythonTests with TestModule.Unittest

  object itest extends PythonTests with TestModule.Unittest

}

These two test modules will expect their sources to be in their respective foo/test and foo/itest folder respectively.

> ./mill foo.test # run Unit test suite
test_hello (test.TestScript...) ... ok
test_world (test.TestScript...) ... ok
...Ran 2 tests...
...OK...

> ./mill foo.itest # run Integration test suite
...test_hello_world (test.TestScript...) ... ok
...Ran 1 test...
...OK...

> ./mill 'foo.{test,itest}' # run both test suites
...test_hello (test.TestScript...)...ok...
...test_world (test.TestScript...)...ok...
...Ran 2 tests...
...test_hello_world (test.TestScript...)...ok...
...Ran 1 test...
...OK...

> ./mill __.itest.testCached # run all integration test suites
...test_hello_world (test.TestScript...) ... ok
...Ran 1 test...
...OK...