Tasks

Sources are defined using Task.Source{…​} taking one os.Path, or Task.Sources{…​}, taking multiple os.Paths as arguments. A Source's: its build signature/hashCode depends not just on the path it refers to (e.g. foo/bar/baz) but also the MD5 hash of the file contents or folder tree at that path.

Task.Source and Task.Sources are common inputs in any Mill build: they watch source files and folders and cause downstream tasks to re-compute if a change is detected. Non-file inputs to the build can also be captured via the more general Input tasks

Taskss are defined using the def foo = Task {…​} syntax, and dependencies on other tasks are defined using foo() to extract the value from them. Apart from the foo() calls, the Task {…​} block contains arbitrary code that does some work and returns a result. Note that tasks cannot have circular dependencies between each other.

The os.walk and os.read.lines statements above are from the OS-Lib library, which provides all common filesystem and subprocess operations for Mill builds. You can see the OS-Lib library documentation for more details:

If a cached task’s inputs change but its output does not, then downstream tasks do not re-evaluate. e.g. Someone may change a comment within an input source file that doesn’t affect the output classfiles. This is determined using the .hashCode of the cached task’s return value.

Furthermore, when code changes occur, cached tasks only invalidate if the code change may directly or indirectly affect it. e.g. adding a comment to lineCount will not cause it to recompute:

But changing the code of the cached task or any upstream helper method will cause the old value to be invalidated and a new value re-computed (with a new println) next time it is invoked:

For more information on how invalidating tasks based on code-changes works, see PR#2417 that implemented it.

The return-value of cached tasks has to be JSON-serializable via uPickle. You can run cached tasks directly from the command line, or use show if you want to see the JSON content or pipe it to external tools. See the uPickle library documentation for more details:

Each task is assigned a unique Task.dest folder on disk (e.g. classFiles is given out/classFiles.dest/). Task.dest is reset every time a task is recomputed, and can be used as scratch space or to store the task’s output files. Any metadata returned from the task is automatically JSON-serialized and stored at out/classFiles.json adjacent to it’s .dest folder. If you want to return a file or a set of files as the result of a Task, write them to disk within your Task.dest folder and return a PathRef() that referencing the files or folders you want to return:

Tasks can depend on other tasks via the foo() syntax.

The output below demonstrates this behavior, with the println defined in def largeFile above running even though the largeFile() branch of the if conditional does not get used:

Defined using Task.Command {…​} syntax, Commands can run arbitrary code, with dependencies declared using the same foo() syntax (e.g. classFiles() above). Commands can be parametrized, but their output is not cached, so they will re-evaluate every time even if none of their inputs have changed. The only times Commands may be skipped is due to Selective Test Execution.

A command with no parameter is defined as def myCommand() = Task.Command {…​}. It is a compile error if () is missing.

Tasks can take command line params, parsed by the MainArgs library. Thus the signature def run(mainClass: String, args: String*) takes params of the form --main-class <str> <arg1> <arg2> …​ <argn>:

Command line arguments can take most primitive types: String, Int, Boolean, etc., along with Option[T] representing optional values and Seq[T] representing repeatable values, and mainargs.Flag representing flags and mainargs.Leftover[T] representing any command line arguments not parsed earlier. Default values for command line arguments are also supported. See the mainargs documentation for more details:

By default, all command parameters need to be named, except for variadic parameters of type T* or mainargs.Leftover[T], or those marked as @arg(positional = true). You can use also the flag --allow-positional-command-args to globally allow arguments to be passed positionally, as shown below:

Like Tasks, a command only evaluates after all its upstream dependencies have completed, and will not begin to run if any upstream dependency has failed.

Commands are assigned the same scratch/output folder out/run.dest/ as Tasks are, and their returned metadata stored at the same out/run.json path for consumption by external tools.

A generalization of Sources, Task.Inputs are tasks that re-evaluate every time (unlike Anonymous Tasks), containing an arbitrary block of code.

Inputs can be used to force re-evaluation of some external property that may affect your build. For example, if I have cached a Task bar that calls out to git to compute the latest commit hash and message directly, that target does not have any Task inputs and so will never re-compute even if the external git status changes:

gitStatusTask will not know that git rev-parse can change, and will not know to re-evaluate when your git log does change. This means gitStatusTask will continue to use any previously cached value, and gitStatusTask's output will be out of date!

To fix this, you can wrap your git log in a Task.Input:

This makes gitStatusInput to always re-evaluate every build, and only if the output of gitStatusInput changes will gitStatusTask2 re-compute

Note that because Task.Inputs re-evaluate every time, you should ensure that the code you put in Task.Input runs quickly. Ideally it should just be a simple check "did anything change?" and any heavy-lifting should be delegated to downstream tasks where it can be cached if possible.

Persistent tasks defined using Task(persistent = true) are similar to normal cached Tasks, except their Task.dest folder is not cleared before every evaluation. This makes them useful for caching things on disk in a more fine-grained manner than Mill’s own Task-level caching: the task can maintain a cache of one or more files on disk, and decide itself which files (or parts of which files!) need to invalidate, rather than having all generated files wiped out every time (which is the default behavior for normal Tasks).

Below is a semi-realistic example of using Task(persistent = true) to compress files in an input folder, cache the previously-compressed files in Task.dest / "cache", and re-use previously-compressed files if a file in the input folder did not change:

In this example, we implement a compressedData task that takes a folder of files in data and compresses them, while maintaining a cache of compressed contents for each file. That means that if the data folder is modified, but some files remain unchanged, those files would not be unnecessarily re-compressed when compressedData evaluates.

Since persistent tasks have long-lived state on disk that lives beyond a single evaluation, this raises the possibility of the disk contents getting into a bad state and causing all future evaluations to fail. It is left up to user of Task(persistent = true) to ensure their implementation is eventually consistent. You can also use mill clean to manually purge the disk contents to start fresh.

Mill workers defined using Task.Worker are long-lived in-memory objects that can persistent across multiple evaluations. These are similar to persistent tasks in that they let you cache things, but the fact that they let you cache the worker object in-memory allows for greater performance and flexibility: you are no longer limited to caching only serializable data and paying the cost of serializing it to disk every evaluation.

Common things to put in workers include:

This example uses a Worker to provide simple two-level cache: in-memory caching for compressed files, in addition to caching them on disk. This means that if the Mill process is persistent (e.g. with --watch/-w) the cache lookups are instant, but even if the Mill process is restarted it can load the cache values from disk without having to recompute them:

Workers live as long as the Mill process. By default, consecutive mill commands in the same folder will re-use the same Mill process and workers, unless --no-server is passed which will terminate the Mill process and workers after every command. Commands run repeatedly using --watch will also preserve the workers between them.

Workers can also make use of their Task.dest folder as a cache that persist when the worker shuts down, as a second layer of caching. The example usage below demonstrates how using the --no-server flag will make the worker read from its disk cache, where it would have normally read from its in-memory cache

Mill uses workers to manage long-lived instances of the Zinc Incremental Scala Compiler and the Scala.js Optimizer. This lets us keep them in-memory with warm caches and fast incremental execution.

Like any other task, you can use ./mill clean to wipe out any cached in-memory or on-disk state belonging to workers. This may be necessary if your worker implementation has bugs that cause the worker disk or in-memory data structures to get into a bad state.

As Workers may also hold limited resources, it may be necessary to free up these resources once a worker is no longer needed. For example, java.net.URLClassLoaders need the .close() method to be called, third-party subprocesses spawned by os.spawn need to .destroy() to be called, otherwise these externally-managed resources may leak causes your build or system to run out of memory.

To implement resource cleanup, your worker can implement java.lang.AutoCloseable. Once the worker is no longer needed, Mill will call the close() method on it before any newer version of this worker is created.

One very common use case for workers is managing long-lived mutable state. The issue with long-lived mutable state is that in the presence of parallelism (the default in Mill), managing such state can be tricky:

The solution to these issues is to maintain an in-memory cache, with proper locking, eviction, and invalidation. Doing so is tedious and error prone, and so Mill provides the CachedFactory helper to make it easier. The example below re-implements the a simplified version of CompressWorker we saw earlier, but using CacheFactory to cache, reset, and re-use the ByteArrayOutputStream btween calls to compressWorker2().compress:

Although this example is synthetic (you don’t actually need to reset, reuse, and teardown ByteArrayOutputStreams) the same techniques would apply to any long-lived mutable state or components you need to manage. This is especially important when running JVM code in classloaders or subprocesses, as those are both expensive to initialize and need to be properly closed or terminated when you are done with them

The above cachedCompressWorker can be used as shown below, with three def compressed* tasks using it to call .compress:

You can define anonymous tasks using the Task.Anon {…​} syntax. These are not runnable from the command-line, but can be used to share common code you find yourself repeating in Tasks and Commands.

Anonymous task’s output does not need to be JSON-serializable, their output is not cached, and they can be defined with or without arguments. Unlike Tasks or Commands, anonymous tasks can be defined anywhere and passed around any way you want, until you finally make use of them within a downstream task or command.

While an anonymous task foo's own output is not cached, if it is used in a downstream task baz and the upstream task bar hasn’t changed, baz's cached output will be used and foo's evaluation will be skipped altogether.