Tasks

Sources are defined using Task.Source{…​} taking one os.Path, or Task.Sources{…​}, taking multiple os.Paths as arguments. A Source's': its build signature/hashCode depends not just on the path it refers to (e.g. foo/bar/baz) but also the MD5 hash of the filesystem tree under that path.

Task.Source and Task.Sources are most common inputs in any Mill build: they watch source files and folders and cause downstream targets to re-compute if a change is detected.

Targets are defined using the def foo = Task {…​} syntax, and dependencies on other targets are defined using foo() to extract the value from them. Apart from the foo() calls, the Task {…​} block contains arbitrary code that does some work and returns a result.

The os.walk and os.read.lines statements above are from the OS-Lib library, which provides all common filesystem and subprocess operations for Mill builds. You can see the OS-Lib library documentation for more details:

If a target’s inputs change but its output does not, e.g. someone changes a comment within the source files that doesn’t affect the classfiles, then downstream targets do not re-evaluate. This is determined using the .hashCode of the Target’s return value.

Furthermore, when code changes occur, targets only invalidate if the code change may directly or indirectly affect it. e.g. adding a comment to lineCount will not cause it to recompute:

But changing the code of the target or any upstream helper method will cause the old value to be invalidated and a new value re-computed (with a new println) next time it is invoked:

For more information on how the bytecode analysis necessary for invalidating targets based on code-changes work, see PR#2417 that implemented it.

The return-value of targets has to be JSON-serializable via uPickle. You can run targets directly from the command line, or use show if you want to see the JSON content or pipe it to external tools. See the uPickle library documentation for more details:

Defined using Task.Command {…​} syntax, Commands can run arbitrary code, with dependencies declared using the same foo() syntax (e.g. classFiles() above). Commands can be parametrized, but their output is not cached, so they will re-evaluate every time even if none of their inputs have changed. A command with no parameter is defined as def myCommand() = Task.Command {…​}. It is a compile error if () is missing.

Targets can take command line params, parsed by the MainArgs library. Thus the signature def run(mainClass: String, args: String*) takes params of the form --main-class <str> <arg1> <arg2> …​ <argn>:

Command line arguments can take most primitive types: String, Int, Boolean, etc., along with Option[T] representing optional values and Seq[T] representing repeatable values, and mainargs.Flag representing flags and mainargs.Leftover[T] representing any command line arguments not parsed earlier. Default values for command line arguments are also supported. See the mainargs documentation for more details:

By default, all command parameters need to be named, except for variadic parameters of type T* or mainargs.Leftover[T]. You can use the flag --allow-positional-command-args to allow arbitrary arguments to be passed positionally, as shown below:

Like Targets, a command only evaluates after all its upstream dependencies have completed, and will not begin to run if any upstream dependency has failed.

Commands are assigned the same scratch/output folder out/run.dest/ as Targets are, and its returned metadata stored at the same out/run.json path for consumption by external tools.

Commands can only be defined directly within a Module body.

Tasks can be overriden, with the overriden task callable via super. You can also override a task with a different type of task, e.g. below we override sourceRoots which is a Task.Sources with a T{} target:

You can define anonymous tasks using the Task.Anon {…​} syntax. These are not runnable from the command-line, but can be used to share common code you find yourself repeating in Targets and Commands.

Anonymous task’s output does not need to be JSON-serializable, their output is not cached, and they can be defined with or without arguments. Unlike Targets or Commands, anonymous tasks can be defined anywhere and passed around any way you want, until you finally make use of them within a downstream target or command.

While an anonymous task foo's own output is not cached, if it is used in a downstream target baz and the upstream target bar hasn’t changed, baz's cached output will be used and foo's evaluation will be skipped altogether.

A generalization of Sources, Task.Inputs are tasks that re-evaluate every time (unlike Anonymous Tasks), containing an arbitrary block of code.

Inputs can be used to force re-evaluation of some external property that may affect your build. For example, if I have a Target bar that calls out to git to compute the latest commit hash and message directly, that target does not have any Task inputs and so will never re-compute even if the external git status changes:

gitStatusTarget will not know that git rev-parse can change, and will not know to re-evaluate when your git log does change. This means gitStatusTarget will continue to use any previously cached value, and gitStatusTarget's output will be out of date!

To fix this, you can wrap your git log in a Task.Input:

This makes gitStatusInput to always re-evaluate every build, and only if the output of gitStatusInput changes will gitStatusTarget2 re-compute

Note that because Task.Inputs re-evaluate every time, you should ensure that the code you put in Task.Input runs quickly. Ideally it should just be a simple check "did anything change?" and any heavy-lifting should be delegated to downstream targets where it can be cached if possible.

Persistent targets defined using Task.Persistent are similar to normal Targets, except their Task.dest folder is not cleared before every evaluation. This makes them useful for caching things on disk in a more fine-grained manner than Mill’s own Target-level caching.

Below is a semi-realistic example of using a Task.Persistent target:

In this example, we implement a compressedData target that takes a folder of files in inputData and compresses them, while maintaining a cache of compressed contents for each file. That means that if the inputData folder is modified, but some files remain unchanged, those files would not be unnecessarily re-compressed when compressedData evaluates.

Since persistent targets have long-lived state on disk that lives beyond a single evaluation, this raises the possibility of the disk contents getting into a bad state and causing all future evaluations to fail. It is left up to the person implementing the Task.Persistent to ensure their implementation is eventually consistent. You can also use mill clean to manually purge the disk contents to start fresh.

Mill workers defined using Task.Worker are long-lived in-memory objects that can persistent across multiple evaluations. These are similar to persistent targets in that they let you cache things, but the fact that they let you cache the worker object in-memory allows for greater performance and flexibility: you are no longer limited to caching only serializable data and paying the cost of serializing it to disk every evaluation. This example uses a Worker to provide simple in-memory caching for compressed files.

Common things to put in workers include:

Workers live as long as the Mill process. By default, consecutive mill commands in the same folder will re-use the same Mill process and workers, unless --no-server is passed which will terminate the Mill process and workers after every command. Commands run repeatedly using --watch will also preserve the workers between them.

Workers can also make use of their Task.dest folder as a cache that persist when the worker shuts down, as a second layer of caching. The example usage below demonstrates how using the --no-server flag will make the worker read from its disk cache, where it would have normally read from its in-memory cache

Mill uses workers to manage long-lived instances of the Zinc Incremental Scala Compiler and the Scala.js Optimizer. This lets us keep them in-memory with warm caches and fast incremental execution.

As Workers may also hold limited resources, it may be necessary to free up these resources once a worker is no longer needed. This is especially the case, when your worker tasks depends on other tasks and these tasks change, as Mill will then also create a new worker instance.

To implement resource cleanup, your worker can implement java.lang.AutoCloseable. Once the worker is no longer needed, Mill will call the close() method on it before any newer version of this worker is created.