The Mill Process Architecture

This page goes into detail of how the Mill process and application is structured. At a high-level, a simplified version of the main components and data-flows within a running Mill process is shown below:

G cluster_client mill client cluster_server mill server cluster_classloder URLClassLoader cluster_build build cluster_foo foo cluster_out out/ cluster_out_foo_folder foo/ cluster_mill_server_folder mill-server/ client-stdin client-stdin Socket Socket client-stdin->Socket client-stdout client-stdout client-stdout->Socket client-stderr client-stderr client-stderr->Socket client-exit client-exit MillClientMain MillClientMain client-exit->MillClientMain client-args client-args client-args->MillClientMain socketPort socketPort Socket->socketPort runArgs runArgs MillClientMain->runArgs exitCode exitCode MillClientMain->exitCode MillServerMain MillServerMain runArgs->MillServerMain ServerSocket ServerSocket socketPort->ServerSocket exitCode->MillServerMain compile.json compile.json foo.compile foo.compile compile.json->foo.compile compile.dest compile.dest compile.dest->foo.compile assembly.json assembly.json foo.assembly foo.assembly assembly.json->foo.assembly assembly.dest assembly.dest assembly.dest->foo.assembly PromptLogger PromptLogger server-stdout server-stdout PromptLogger->server-stdout server-stderr server-stderr PromptLogger->server-stderr Evaluator Evaluator MillServerMain->Evaluator ServerSocket->PromptLogger foo.sources foo.sources foo.sources->foo.compile foo.classPath foo.classPath foo.compile->foo.classPath foo.classPath->foo.assembly foo.resources foo.resources foo.resources->foo.assembly

The Mill Client

The Mill client is a small Java application that is responsible for launching and delegating work to the Mill server, a long-lived process. Each ./mill command spawns a new Mill client, but generally re-uses the same Mill server where possible in order to reduce startup overhead and to allow the Mill server process to warm up and provide good performance

  • The Mill client takes all the inputs of a typical command-line application - stdin and command-line arguments - and proxies them to the long-lived Mill server process.

  • It then takes the outputs from the Mill server - stdout, stderr, and finally the exitcode - and proxies those back to the calling process or terminal.

In this way, the Mill client acts and behaves for most all intents and purposes as a normal CLI application, except it is really a thin wrapper around logic that is actually running in the long-lived Mill server.

The Mill server sometimes is shut down and needs to be restarted, e.g. if Mill version changed, or the user used Ctrl-C to interrupt the ongoing computation. In such a scenario, the Mill client will automatically restart the server the next time it is run, so apart from a slight performance penalty from starting a "cold" Mill server such shutdowns and restarts should be mostly invisibl to the user.

The Mill Server

The Mill server is a long-lived process that the Mill client spawns. Only one Mill server should be running in a codebase at a time, and each server takes a filelock at startup time to enforce this mutual exclusion.

The Mill server compiles your build.mill and package.mill, spawns a URLClassLoader containing the compiled classfiles, and uses that to instantiate the variousModules and Tasks dynamically in-memory. These are then used by the Evaluator, which resolves, plans, and executes the tasks specified by the given runArgs

During execution, both standard output and standard error are captured during evaluation and forwarded to the PromptLogger. PromptLogger annotates the output stream with the line-prefixes, prompt, and ANSI terminal commands necessary to generate the dynamic prompt, and then forwards both streams multi-plexed over a single socket stream back to the Mill client. The client then de-multiplexes the combined stream to split it back into output and error, which are then both forwarded to the process or terminal that invoked the Mill client.

Lastly, when the Mill server completes its tasks, it writes the exitCode to a file that is then propagated back to the Mill client. The Mill client terminates with this exit code, but the Mill server remains alive and ready to serve to the next Mill client that connects to it

For a more detailed discussion of what exactly goes into "execution", see The Mill Execution Model.

The Out Folder

The out/ directory is where most of Mill’s state lives on disk, both build-task state such as the foo/compile.json metadata cache for foo.compile, or the foo/compile.dest which stores any generated files or binaries. It also contains mill-server/ folder which is used to pass data back and forth between the client and server: the runArgs, exitCode, etc.

Each task during evaluation reads and writes from its own designated paths in the out/ folder. Each task’s files are not touched by any other tasks, nor are they used in the rest of the Mill architecture: they are solely meant to serve each task’s caching and filesystem needs.

More documentation on what the out/ directory contains and how to make use of it can be found at The Output Directory.