Tasks

Sources are defined using Task.Source{…​} taking one os.Path, or Task.Sources{…​}, taking multiple os.Paths as arguments. A Source's: its build signature/hashCode depends not just on the path it refers to (e.g. foo/bar/baz) but also the MD5 hash of the filesystem tree under that path.

Task.Source and Task.Sources are most common inputs in any Mill build: they watch source files and folders and cause downstream tasks to re-compute if a change is detected.

Cached Taskss are defined using the def foo = Task {…​} syntax, and dependencies on other tasks are defined using foo() to extract the value from them. Apart from the foo() calls, the Task {…​} block contains arbitrary code that does some work and returns a result. Note that tasks cannot have circular dependencies between each other.

The os.walk and os.read.lines statements above are from the OS-Lib library, which provides all common filesystem and subprocess operations for Mill builds. You can see the OS-Lib library documentation for more details:

If a cached task’s inputs change but its output does not, e.g. someone changes a comment within the source files that doesn’t affect the classfiles, then downstream tasks do not re-evaluate. This is determined using the .hashCode of the cached task’s return value.

Furthermore, when code changes occur, cached tasks only invalidate if the code change may directly or indirectly affect it. e.g. adding a comment to lineCount will not cause it to recompute:

But changing the code of the cached task or any upstream helper method will cause the old value to be invalidated and a new value re-computed (with a new println) next time it is invoked:

For more information on how the bytecode analysis necessary for invalidating cached tasks based on code-changes work, see PR#2417 that implemented it.

The return-value of cached tasks has to be JSON-serializable via uPickle. You can run cached tasks directly from the command line, or use show if you want to see the JSON content or pipe it to external tools. See the uPickle library documentation for more details:

Defined using Task.Command {…​} syntax, Commands can run arbitrary code, with dependencies declared using the same foo() syntax (e.g. classFiles() above). Commands can be parametrized, but their output is not cached, so they will re-evaluate every time even if none of their inputs have changed. A command with no parameter is defined as def myCommand() = Task.Command {…​}. It is a compile error if () is missing.

Tasks can take command line params, parsed by the MainArgs library. Thus the signature def run(mainClass: String, args: String*) takes params of the form --main-class <str> <arg1> <arg2> …​ <argn>:

Command line arguments can take most primitive types: String, Int, Boolean, etc., along with Option[T] representing optional values and Seq[T] representing repeatable values, and mainargs.Flag representing flags and mainargs.Leftover[T] representing any command line arguments not parsed earlier. Default values for command line arguments are also supported. See the mainargs documentation for more details:

By default, all command parameters need to be named, except for variadic parameters of type T* or mainargs.Leftover[T], or those marked as @arg(positional = true). You can use also the flag --allow-positional-command-args to globally allow arguments to be passed positionally, as shown below:

Like Cached Tasks, a command only evaluates after all its upstream dependencies have completed, and will not begin to run if any upstream dependency has failed.

Commands are assigned the same scratch/output folder out/run.dest/ as Tasks are, and its returned metadata stored at the same out/run.json path for consumption by external tools.

Commands can only be defined directly within a Module body.

Tasks can be overriden, with the overriden task callable via super. You can also override a task with a different type of task, e.g. below we override sourceRoots which is a Task.Sources with a cached Task{} that depends on the original via super:

You can define anonymous tasks using the Task.Anon {…​} syntax. These are not runnable from the command-line, but can be used to share common code you find yourself repeating in Tasks and Commands.

Anonymous task’s output does not need to be JSON-serializable, their output is not cached, and they can be defined with or without arguments. Unlike Cached Tasks or Commands, anonymous tasks can be defined anywhere and passed around any way you want, until you finally make use of them within a downstream task or command.

While an anonymous task foo's own output is not cached, if it is used in a downstream task baz and the upstream task bar hasn’t changed, baz's cached output will be used and foo's evaluation will be skipped altogether.

A generalization of Sources, Task.Inputs are tasks that re-evaluate every time (unlike Anonymous Tasks), containing an arbitrary block of code.

Inputs can be used to force re-evaluation of some external property that may affect your build. For example, if I have a cached task bar that calls out to git to compute the latest commit hash and message directly, that target does not have any Task inputs and so will never re-compute even if the external git status changes:

gitStatusTask will not know that git rev-parse can change, and will not know to re-evaluate when your git log does change. This means gitStatusTask will continue to use any previously cached value, and gitStatusTask's output will be out of date!

To fix this, you can wrap your git log in a Task.Input:

This makes gitStatusInput to always re-evaluate every build, and only if the output of gitStatusInput changes will gitStatusTask2 re-compute

Note that because Task.Inputs re-evaluate every time, you should ensure that the code you put in Task.Input runs quickly. Ideally it should just be a simple check "did anything change?" and any heavy-lifting should be delegated to downstream tasks where it can be cached if possible.

One major use case of Input tasks is to make your build configurable via JVM system properties of environment variables. If you directly access sys.props or sys.env inside a cached Task{}, the cached value will be used even if the property or environment variable changes in subsequent runs, when you really want it to be re-evaluated. Thus, accessing system properties should be done in a Task.Input, and usage of the property should be done downstream in a cached task:

Again, Task.Input runs every time, and thus you should only do the bare minimum in your Task.Input that is necessary to detect changes. Any further processing should be done in downstreak cached tasks to allow for proper caching and re-use

Like system properties, environment variables should be referenced in Task.Inputs. Unlike system properties, you need to use the special API Task.env to access the environment, due to JVM limitations:

Persistent tasks defined using Task(persistent = true) are similar to normal cached Tasks, except their Task.dest folder is not cleared before every evaluation. This makes them useful for caching things on disk in a more fine-grained manner than Mill’s own Task-level caching: the task can maintain a cache of one or more files on disk, and decide itself which files (or parts of which files!) need to invalidate, rather than having all generated files wiped out every time (which is the default behavior for normal Tasks).

Below is a semi-realistic example of using Task(persistent = true) to compress files in an input folder, and re-use previously-compressed files if a file in the input folder did not change:

In this example, we implement a compressedData task that takes a folder of files in data and compresses them, while maintaining a cache of compressed contents for each file. That means that if the data folder is modified, but some files remain unchanged, those files would not be unnecessarily re-compressed when compressedData evaluates.

Since persistent tasks have long-lived state on disk that lives beyond a single evaluation, this raises the possibility of the disk contents getting into a bad state and causing all future evaluations to fail. It is left up to user of Task(persistent = true) to ensure their implementation is eventually consistent. You can also use mill clean to manually purge the disk contents to start fresh.

Mill workers defined using Task.Worker are long-lived in-memory objects that can persistent across multiple evaluations. These are similar to persistent tasks in that they let you cache things, but the fact that they let you cache the worker object in-memory allows for greater performance and flexibility: you are no longer limited to caching only serializable data and paying the cost of serializing it to disk every evaluation.

This example uses a Worker to provide simple two-level cache: in-memory caching for compressed files, in addition to caching them on disk. This means that if the Mill process is persistent (e.g. with --watch/-w) the cache lookups are instant, but even if the Mill process is restarted it can load the cache values from disk without having to recompute them:

Common things to put in workers include:

Workers live as long as the Mill process. By default, consecutive mill commands in the same folder will re-use the same Mill process and workers, unless --no-server is passed which will terminate the Mill process and workers after every command. Commands run repeatedly using --watch will also preserve the workers between them.

Workers can also make use of their Task.dest folder as a cache that persist when the worker shuts down, as a second layer of caching. The example usage below demonstrates how using the --no-server flag will make the worker read from its disk cache, where it would have normally read from its in-memory cache

Mill uses workers to manage long-lived instances of the Zinc Incremental Scala Compiler and the Scala.js Optimizer. This lets us keep them in-memory with warm caches and fast incremental execution.

Like any other task, you can use ./mill clean to wipe out any cached in-memory or on-disk state belonging to workers. This may be necessary if your worker implementation has bugs that cause the worker disk or in-memory data structures to get into a bad state.

As Workers may also hold limited resources, it may be necessary to free up these resources once a worker is no longer needed. This is especially the case, when your worker tasks depends on other tasks and these tasks change, as Mill will then also create a new worker instance.

To implement resource cleanup, your worker can implement java.lang.AutoCloseable. Once the worker is no longer needed, Mill will call the close() method on it before any newer version of this worker is created.

One very common use case for workers is managing long-lived mutable state. The issue with long-lived mutable state is that in the presence of parallelism (the default in Mill), managing such state can be tricky:

The solution to these issues is to maintain an in-memory cache, with proper locking, eviction, and invalidation. Doing so is tedious and error prone, and so Mill provides the CachedFactory helper to make it easier. The example below re-implements the a simplified version of CompressWorker we saw earlier, but using CacheFactory to cache, reset, and re-use the ByteArrayOutputStream btween calls to compressWorker2().compress:

Although this example is synthetic (you don’t actually need to reset, reuse, and teardown `ByteArrayOutputStream`s) the same techniques would apply to any long-lived mutable state or components you need to manage. This is especially important when running JVM code in classloaders or subprocesses, as those are both expensive to initialize and need to be properly closed or terminated when you are done with them

The above cachedCompressWorker can be used as shown below, with three def compressed* tasks using it to call .compress: