The Mill Evaluation Model
Evaluating a Mill task typically goes through the following phases:
-
Compilation: Mill compiles the
build.mill
to classfiles, following the The Mill Bootstrapping Process to eventually produce aRootModule
object -
Resolution: Mill resolves the list of Tasks given from the command line, e.g.
resolve
orfoo.compile
or{bar,qux}._.test
, to a list of concreteTask
objects nested on Modules within theRootModule
along with their transitive dependencies-
In the process, the relevant Mill
Module
s are lazily instantiated
-
-
Evaluation: Mill evaluates the gathered
Task
s in dependency-order, either serially or in parallel
Limitations of the Mill Evaluation Model
This three-phase evaluation model has consequences for how you structure your build. For example:
-
You can have arbitrary code outside of
Task
s that helps set up your task graph and module hierarchy, e.g. computing what keys exist in aCross
module, or specifying yourdef moduleDeps
-
You can have arbitrary code inside of
Task
s, to perform your build actions -
But your code inside of
Task
s cannot influence the shape of the task graph or module hierarchy, as all Resolving and Planning happens first before anyTask
s are evaluated.
This should not be a problem for most builds, but it is something to be aware
of. In general, we have found that having "two places" to put code - outside of
Task
s to run during Planning or inside of Task
s to run during
Evaluation - is generally enough flexibility for most use cases.
Caching at Each Layer of the Evaluation Model
Apart from fine-grained caching of Task
s during Evaluation, Mill also
performs incremental evaluation of the other phases. This helps ensure
the overall workflow remains fast even for large projects:
-
Compilation:
-
Done on-demand and incrementally using the Scala incremental compiler Zinc.
-
If some of the files
build.mill
imported changed but not others, only the changed files are re-compiled before theRootModule
is re-instantiated -
In the common case where
build.mill
was not changed at all, this step is skipped entirely and theRootModule
object simply re-used from the last run.
-
-
Planning:
-
If the
RootModule
was re-used, then all previously-instantiated modules are simply-re-used
-
-
Evaluation:
-
Task
s are evaluated in dependency order -
Targets only re-evaluate if their input
Task
s change. -
Task.Persistents preserve the
T.dest
folder on disk between runs, allowing for finer-grained caching than Mill’s default task-by-task caching and invalidation -
Task.Workers are kept in-memory between runs where possible, and only invalidated if their input
Task
s change as well. -
Task
s in general are invalidated if the code they depend on changes, at a method-level granularity via callgraph reachability analysis. See #2417 for more details
-
This approach to caching does assume a certain programming style inside your
Mill build: we may-or-may-not re-instantiate the modules in your
build.mill
and we may-or-may-not re-execute any particular task depending on caching,
but your code needs to work either way. Furthermore, task def
s and module `object`s in your
build are instantiated lazily on-demand, and your code needs to work regardless
of which order they are executed in. For code written in a typical Scala style,
which tends to avoid side effects, this is not a problem at all.
One thing to note is for code that runs during Resolution: any reading of
external mutable state needs to be wrapped in an interp.watchValue{…}
wrapper. This ensures that Mill knows where these external reads are, so that
it can check if their value changed and if so re-instantiate RootModule
with
the new value.
The Mill Bootstrapping Process
Mill’s bootstrapping proceeds roughly in the following phases:
-
If using the bootstrap script, it first checks if the right version of Mill is already present, and if not it downloads it to
~/.mill/download
-
It instantiates an in-memory
MillBuildRootModule.BootstrapModule
, which is a hard-codedbuild.mill
used for bootstrapping Mill -
If there is a meta-build present
mill-build/build.mill
, it processes that first and uses theMillBuildRootModule
returned for the next steps. Otherwise it uses theMillBuildRootModule.BootstrapModule
directly -
Mill evaluates the
MillBuildRootModule
to parse thebuild.mill
, generate a list ofivyDeps
as well as appropriately wrapped Scala code that we can compile, and compiles it to classfiles -
Mill loads the compiled classfiles of the
build.mill
into ajava.lang.ClassLoader
to access it’sRootModule
Everything earlier in the doc applies to each level of meta-builds in the Mill bootstrapping process as well.
In general, .sc
files, import $file
, and import $ivy
can be thought of as
a short-hand for configuring the meta-build living in mill-build/build.mill
:
-
.sc
andimport $file
are a shorthand for specifying the.scala
files living inmill-build/src/
-
import $ivy
is a short-hand for configurin thedef ivyDeps
inmill-build/build.mill
Most builds would not need the flexibility of a meta-build’s
mill-build/build.mill
, but it is there if necessary.
Mill supports multiple levels of meta-builds for bootstrapping:
-
Just
build.mill
-
One level of meta-builds:
mill-build/build.mill
andbuild.mill
-
Two level of meta-builds:
mill-build/mill-build/build.mill
,mill-build/build.mill
andbuild.mill
The Mill Meta Build works through a simple use case and example for meta-builds.