The Mill Process Architecture

This page goes into detail of how the Mill process and application is structured. At a high-level, a simplified version of the main components and data-flows within a running Mill process is shown below:

G cluster_out out/ cluster_mill_server_folder mill-daemon/ cluster_out_foo_folder foo/ cluster_server mill daemon cluster_classloder URLClassLoader cluster_build build cluster_foo foo cluster_client mill launcher launcher-stdin launcher-stdin Socket Socket launcher-stdin->Socket launcher-stdout launcher-stdout launcher-stdout->Socket launcher-stderr launcher-stderr launcher-stderr->Socket launcher-exit launcher-exit MillLauncherMain MillLauncherMain launcher-exit->MillLauncherMain launcher-args launcher-args launcher-args->MillLauncherMain socketPort socketPort Socket->socketPort runArgs runArgs MillLauncherMain->runArgs exitCode exitCode MillLauncherMain->exitCode MillDaemonMain MillDaemonMain runArgs->MillDaemonMain ServerSocket ServerSocket socketPort->ServerSocket exitCode->MillDaemonMain compile.json compile.json foo.compile foo.compile compile.json->foo.compile compile.dest compile.dest compile.dest->foo.compile assembly.json assembly.json foo.assembly foo.assembly assembly.json->foo.assembly assembly.dest assembly.dest assembly.dest->foo.assembly PromptLogger PromptLogger daemon-stdout daemon-stdout PromptLogger->daemon-stdout daemon-stderr daemon-stderr PromptLogger->daemon-stderr Evaluator Evaluator MillDaemonMain->Evaluator ServerSocket->PromptLogger foo.sources foo.sources foo.sources->foo.compile foo.classPath foo.classPath foo.compile->foo.classPath foo.classPath->foo.assembly foo.resources foo.resources foo.resources->foo.assembly

The Mill Client

The Mill launcher is a small Java application that is responsible for launching and delegating work to the Mill daemon, a long-lived process. Each ./mill command spawns a new Mill launcher, but generally re-uses the same Mill daemon where possible in order to reduce startup overhead and to allow the Mill daemon process to warm up and provide good performance

  • The Mill launcher takes all the inputs of a typical command-line application - stdin and command-line arguments - and proxies them to the long-lived Mill daemon process.

  • It then takes the outputs from the Mill daemon - stdout, stderr, and finally the exitcode - and proxies those back to the calling process or terminal.

In this way, the Mill launcher acts and behaves for most all intents and purposes as a normal CLI application, except it is really a thin wrapper around logic that is actually running in the long-lived Mill daemon.

The Mill daemon sometimes is shut down and needs to be restarted, e.g. if Mill version changed, or the user used Ctrl-C to interrupt the ongoing computation. In such a scenario, the Mill launcher will automatically restart the daemon the next time it is run, so apart from a slight performance penalty from starting a "cold" Mill daemon such shutdowns and restarts should be mostly invisibl to the user.

The Mill Daemon

The Mill daemon is a long-lived process that the Mill launcher spawns. Only one Mill daemon should be running in a codebase at a time, and each daemon takes a filelock at startup time to enforce this mutual exclusion.

The Mill daemon compiles your build.mill and package.mill, spawns a URLClassLoader containing the compiled classfiles, and uses that to instantiate the variousModules and Tasks dynamically in-memory. These are then used by the Evaluator, which resolves, plans, and executes the tasks specified by the given runArgs

During execution, both standard output and standard error are captured during evaluation and forwarded to the PromptLogger. PromptLogger annotates the output stream with the line-prefixes, prompt, and ANSI terminal commands necessary to generate the dynamic prompt, and then forwards both streams multi-plexed over a single socket stream back to the Mill launcher. The launcher then de-multiplexes the combined stream to split it back into output and error, which are then both forwarded to the process or terminal that invoked the Mill launcher.

Lastly, when the Mill daemon completes its tasks, it writes the exitCode to a file that is then propagated back to the Mill launcher. The Mill launcher terminates with this exit code, but the Mill daemon remains alive and ready to serve to the next Mill launcher that connects to it

For a more detailed discussion of what exactly goes into "execution", see The Mill Evaluation Model.

The Out Folder

The out/ directory is where most of Mill’s state lives on disk, both build-task state such as the foo/compile.json metadata cache for foo.compile, or the foo/compile.dest which stores any generated files or binaries. It also contains mill-daemon/ folder which is used to pass data back and forth between the launcher and daemon: the runArgs, exitCode, etc.

Each task during evaluation reads and writes from its own designated paths in the out/ folder. Each task’s files are not touched by any other tasks, nor are they used in the rest of the Mill architecture: they are solely meant to serve each task’s caching and filesystem needs.

More documentation on what the out/ directory contains and how to make use of it can be found at The Output Directory.