Case Study: Mill vs Gradle

Compared to Gradle,

Mill follows Gradle’s conciseness and extensibility: Rather than pages and pages of verbose XML, every line in a Mill build is meaningful. e.g. adding a dependency is 1 line in Mill, like it is in Gradle, and unlike the 5 line <dependency> declaration you find in Maven. Like Gradle, end users can easily customize their build to fit their exact needs without needing to go through the process of writing plugins.
Mill can be 2-4x faster than Gradle: Although both Mill and Gradle automatically cache and parallelize your build, Mill does so with much less fixed overhead. This means less time waiting for your build tool, and more time for the things that really matter to your project.
Mill enforces best practices by default. All parts of a Mill build are cached and incremental by default. All Mill tasks write their output to a standard place. All task inter-dependencies are automatically captured without manual annotation. Where Gradle requires considerable effort and expertise to understand your build and set it up in the right way, Mill’s good IDE experience makes understanding your build easier, and its extensibility model makes configuring your build foolproof, so the easiest thing to do in Mill is usually the right thing to do.

This page compares using Mill to Gradle, using the Mockito Testing Library codebase as the example. Mockito is a medium sized codebase, 100,000 lines of Java split over 22 subprojects. By porting it to Mill, this case study should give you an idea of how Mill compares to Gradle in more realistic, real-world projects.

To do this, we have written a Mill build.mill file for the Mockito project. This can be used with Mill to build and test the various submodules of the Mockito project without needing to change any other files in the repository:

Mockito build.mill file

Completeness

The Mill build for Mockito is not 100% complete, but it covers most of the major parts of Mockito: compiling Java, running JUnit tests. For now, the Android, Kotlin, and OSGI tests are skipped, as support for Building Android apps in Mill and Kotlin with Mill is still experimental.

The goal of this exercise is not to be 100% feature complete enough to replace the Gradle build today. It is instead meant to provide a realistic comparison of how using Mill in a realistic, real-world project compares to using Gradle.

Performance

The Mill build for Mockito is generally snappier than the Gradle build. This applies to most workflows, but the difference matters most for workflows which are short-lived, where the difference in the fixed overhead of the build tool is most noticeable.

For comparison purposes, I disabled the Gradle subprojects that we did not fully implement in Mill (groovyTest, groovyInlineTest, kotlinTest, kotlinReleaseCoroutinesTest, android, osgi-test, java21-test), and added the necessary flags to ensure caching/parallelism/etc. is configured similarly for both tools. This ensures the comparison is fair with both builds compiling the same code and running the same tests in the same way.

For the benchmarks below, each provided number is the median wall time of three consecutive runs on my M1 Macbook Pro. While ad-hoc, these benchmarks are enough to give you a flavor of how Mill’s performance compares to Gradle:

Benchmark	Gradle	Mill	Speedup
Sequential Clean Compile All	17.6s	5.40s	3.3x
Parallel Clean Compile All	12.3s	3.57s	3.4x
Clean Compile Single-Module	4.41s	1.20s	3.7x
Incremental Compile Single-Module	1.37s	0.51s	2.7x
No-Op Compile Single-Module	0.94s	0.46s	2.0x

Benchmark

Gradle

Mill

Speedup

Sequential Clean Compile All

17.6s

5.40s

3.3x

Parallel Clean Compile All

12.3s

3.57s

3.4x

Clean Compile Single-Module

4.41s

1.20s

3.7x

Incremental Compile Single-Module

1.37s

0.51s

2.7x

No-Op Compile Single-Module

0.94s

0.46s

2.0x

The column on the right shows the speedups of how much faster Mill is compared to the equivalent Gradle workflow. In most cases, Mill is 2-4x faster than Gradle. Below, we will go into more detail of each benchmark: how they were run, what they mean, and how we can explain the difference in performing the same task with the two different build tools.

Sequential Clean Compile All

$ ./gradlew clean; time ./gradlew classes testClasses --no-build-cache
17.6s
18.2s
17.4s

$ ./mill clean; time ./mill -j 1 __.compile
5.60s
5.40s
6.13s

This benchmark measures the time taken to sequentially compiled all the Java code in the Mockito code base. The goal of this benchmark is to try and measure the "clean compile everything" step, without the effects of parallelism that can be nondeterministic and vary wildly from machine to machine depending on the number of cores available.

To limit our comparison to compiling Java code only, we avoid using build in favor of classes and testClasses: this skips running tests, lint, jar/docjar generation, and other steps that build performs to make it an apples-to-apples comparison. Furthermore, Gradle parallelizes the build by default and caches things globally under ~/.gradle/caches, while Mill parallelizes by default but does not cache things globally. Again to make it a fair comparison, we use --no-build-cache in Gradle and set org.gradle.parallel=false in gradle.properties, and pass -j 1 to limit Mill to a single thread.

Here we see Mill being about ~3.3x faster than Gradle, to do the equivalent amount of work. As a point of reference, Java typically compiles at 10,000-50,000 lines per second on a single thread, and the Mockito codebase is ~100,000 lines of code, so we would expect compile to take 2-10 seconds without parallelism. The 5-6s taken by Mill seems about what you would expect for a codebase of this size, and the ~17s taken by Gradle is much more than what you would expect from simple Java compilation.

It’s actually not clear to me where the difference in execution time is coming from. Unlike the Mill v.s. Maven comparison, Gradle’s command line output doesn’t show any obvious network requests or jar packing/unpacking/comparing going on. But Gradle’s CLI output is also much less verbose than Maven’s, so it’s possible things are going on under the hood that I’m not aware of.

Parallel Clean Compile All

$ ./gradlew clean; time ./gradlew classes testClasses --no-build-cache
13.8s
12.3s
11.4s

$ ./mill clean; time ./mill __.compile
3.59s
3.57s
3.45s

This benchmark is identical to the Sequential Clean Compile All benchmark above, but enables parallelism: Gradle by default, Mill without -j 1 to run on 10 cores (the number on my Macbook Pro).

Neither Gradle nor Mill benefit hugely from parallelism: both show a moderate ~50% speedup, despite receiving 900% more CPUs. This likely indicates that the module dependency graph of the Mockito codebase is laid out in a way that does not allow huge amounts of compile-time parallelism.

Again, we see Mill being about ~3.4x faster than Gradle, to do the equivalent amount of work. This indicates the the speedup Mill provides over Gradle is unrelated to the parallelism of each tool.

Clean Compile Single-Module

$ ./gradlew clean; time ./gradlew :classes --no-build-cache
4.14s
4.41s
4.41s

$ ./mill clean; time ./mill compile
1.20s
1.12s
1.30s

This benchmark indicates the use case of clean-compiling a single module. In this case, the root module in src/main/java/ containing the bulk of the Mockito library code, excluding the test code in src/test/java/ and all the downstream subprojects in subprojects/.

This benchmark gives us Mill being about ~3.7x faster than Gradle. This is in line with the results above.

Incremental Compile Single-Module

$ echo "" >> src/main/java/org/mockito/BDDMockito.java; time ./gradlew :classes
1.37s
1.39s
1.28s

$ echo "" >> src/main/java/org/mockito/BDDMockito.java; time ./mill compile
compiling 1 Java source to /Users/lihaoyi/Github/netty/out/common/compile.dest/classes ...
0.52s
0.51s
0.52s

This benchmark measures the common case of making a tiny change to a single file and re-compiling just that module. This is the common workflow that most software developers do over and over day-in and day-out. We simulate this by appending a new line to the file src/main/java/org/mockito/BDDMockito.java.

Both Mill and Gradle are able to take advantage of the small code change and re-compile only the single files needing re-compilation, demonstrating substantial speedups over the Clean Compile Single-Module benchmark above. Mill remains faster than Gradle, showing a ~2.7x speedup for this task

No-Op Compile Single-Module

$ time ./gradlew :classes
0.95s
0.93s
0.94s

$ time ./mill common.compile
0.46s
0.50s
0.45s

This benchmark is meant to measure the pure overhead of running the build tool: given a single module that did not change, the build tool should need to do nothing in response, and so any time taken is pure overhead.

For both Mill and Gradle, we see small speedups relative to the Incremental Compile Single-Module benchmark above, which likely comes from not having to compile any Java source files at all. Mill remains faster than Gradle by about 2.0x.

IDE Experience

One area that Mill does better than Gradle is providing a seamless IDE experience. For example, consider the snippet below where we are using Gradle to configure the javac compiler options. Due to .gradle files being untyped Groovy, the autocomplete and code-assist experience working with these files is hit-or-miss. In the example below, we can see that IntelliJ is able to identify that compileArgs exists and has the type List<String>:

But if you try to jump to definition or find out anything else about it you hit a wall:

Often working with build configurations feels like hitting dead ends: if you don’t have options.compilerArgs memorized in your head, there is literally nothing you can do in your editor to make progress to figure out what it is or what it is used for. That leaves you googling for answers, which can be a frustrating experience that distracts you from the task at hand.

The fundamental problem with tools like Gradle is that the code you write does not actually perform the build: rather, you are just setting up some data structure that is used to configure the real build engine that runs later. Thus when you explore the Gradle build in an IDE, the IDE can only explore the configuration logic (which is usually un-interesting) and is unable to explore the actual build logic (which is what you actually care about!)

In comparison, Mill’s .mill files are all statically typed, and as a result IntelliJ is easily able to pull up the documentation for def javacOptions, even though it doesn’t have any special support for Mill built into the IDE:

Apart from static typing, the way Mill builds are structured also helps the IDE: Mill code actually performs your build, rather than configuring some opaque build engine. While that sounds academic, one concrete consequence is that IntelliJ is able to take your def javacOptions override and find the original definitions that were overridden, and show you where they are defined:

Furthermore, because task dependencies in Mill are just normal method calls, IntelliJ is able to find usages, showing you where the task is used. Below, we can see the method call in the def compile task, which uses javacOptions() along with a number of other tasks:

From there, if you are curious about any of the other tasks used alongside javacOptions, it’s easy for you to pull up their documentation, jump to their definition, or find their usages. For example we can pull up the docs of compileClasspath() below:

Or we can use find usages on def compile to see where it is used, both in this build and upstream in the Mill libraries:

Unlike most other build tools, Mill builds are extremely easy to explore interactively in your IDE. If you do not know what something does, it’s documentation, definition, or usages is always one click away in IntelliJ or VSCode. That’s not to say Mill builds aren’t complex - as we saw above, compilation has to deal with upstream outputs, classpaths, flags, reporters, and so on - but at least in Mill your IDE can help you explore, understand and manage the complexity in a way that no other build tool supports.

Note that the IDE experience that Mill provides should already be very familiar to anyone writing Java, Kotlin, or Scala:

of course you can find the overridden definitions!
of course you can pull up the documentation in a click!
of course you can navigate around the codebase with your IDE, up and down the call graph, to see who calls who!

What Mill provides isn’t rocket science, but rather it is just about taking your existing experience and existing IDE tooling working with application codebases, and lets you use it to manage your build system as well.

Mill IDE support isn’t perfect - you may have noticed the spurious red squigglies above - but it’s already better than most other build systems like Gradle or Maven. And that is with approximately ~zero custom integrations with the various IDEs: with some additional work, we can expect the Mill IDE experience to improve even more over time.

Extensibility

Another facet of Mill is that is worth exploring is the ease of making custom tasks or build steps. For example, in Mill, overriding the resources to duplicate a file can be done as follows:

def resources = Task {
  os.copy(
    compile().classes.path / "org/mockito/internal/creation/bytebuddy/inject/MockMethodDispatcher.class",
    Task.dest / "org/mockito/internal/creation/bytebuddy/inject/MockMethodDispatcher.raw",
    createFolders = true
  )
  super.resources() ++ Seq(PathRef(Task.dest))
}

In Gradle, it is written as:

tasks.register('copyMockMethodDispatcher', Copy) {
    dependsOn compileJava

    from "${sourceSets.main.java.classesDirectory.get()}/org/mockito/internal/creation/bytebuddy/inject/MockMethodDispatcher.class"
    into layout.buildDirectory.dir("generated/resources/inline/org/mockito/internal/creation/bytebuddy/inject")

    rename '(.+)\\.class', '$1.raw'
}

classes.dependsOn("copyMockMethodDispatcher")

sourceSets.main {
    resources {
        output.dir(layout.buildDirectory.dir("generated/resources/inline"))
    }
}

At a first glance, both of these snippets do the same thing, just with different syntaxes and helper method names. However, on a deeper look, a few things are worth noting:

In Mill, you do not need to manually add dependsOn clauses, unlike Gradle:
- In Mill, referencing the value of compile(), we both explicitly get access to the value of compile and also add a dependency on it. In Gradle, you need to separately add dependsOn compile to mark the dependency, and rename '(.+)\\.class', '$1.raw' to make use of it implicitly.
- In Mill, overriding def resources is enough to make all tasks that previously depended on resources now depend on the override (e.g. run, test, jar, assembly, etc.) as is the norm for object-oriented overrides. In Gradle, you need to explicitly call classes.dependsOn("copyMockMethodDispatcher") to make the downstream classes task depend on copyMockMethodDispatcher, and sourcesSets.main resources output.dir to wire up the generated files to the resources of the module
In Mill, the resources task is given a unique Task.dest folder that is unique to it. In contrast, Gradle’s copyMockMethodDispatcher puts things in a global generated/ folder
- This means that in Mill, you do not need to worry about filesystem collisions, since every task’s Task.dest is unique. In contrast, in Gradle you need to make sure that no other task in the entire build is scribbling over generated/, otherwise they could interfere with one another in confusing ways
- This also means that in Mill, you always know where the output of a particular task is - foo.bar.resources writes to out/foo/bar/resources.dest/ - so you can always easily find the output of a particular task. In Gradle, you have to dig through the source code to find where the task is implemented and see where it is writing to.
Mill passes typed structured Paths and PathRefs between each other, while Gradle often uses raw path strings
- In Mill, def resources returns a PathRef(Task.dest) for downstream tasks to use, so downstream tasks can use it directly (similar to how it makes use of compile().classes.path directly). This means different tasks can refer to each other in a foolproof way without room for error
- In Gradle, sourcesSets.map resources output.dir needs to refer to the path generated by copyMockMethodDispatcher via it’s string "generated/resources/inline". That adds a lot of room for error, since the strings can easily get out of sync accidentally.

In general, although the two snippets aren’t that different superficially, Mill makes it easy to do the right thing by default:

Upstream task dependencies are recorded automatically when used
Overridden definitions and automatically used by downstream tasks
Every task is automatically assigned a place on disk so you don’t need to worry about collisions and can easily find outputs
Tasks interact with each other via typed structured values - Paths, PathRefs, etc. - rather than magic strings

Although in Gradle it is possible for an expert to customize their build in a way that mitigates these issues, Mill does it automatically and in a way that is foolproof even for non-experts. This helps democratize the build so that any engineer can contribute fixes or improvements without needing to be a build-system expert and learn all the best practices first.

Lastly, as mentioned earlier, the Gradle script has limited IDE support: it can autocomplete things for you, but once you try to jump-to-definition or otherwise navigate your build you hit a wall: it tells you some minimal documentation about the identifier, but nothing about how it is implemented or where it is used:

IntellijGradleResourcesClassesDefinition

In contrast, IntelliJ is able to navigate straight to the definition of compile() in the Mill build (as we saw earlier in IDE Experience), and from there can continue to traverse the build via jump to definition (which we saw earlier) or find usages, as we saw earlier:

Mill build scripts are written in Scala, but you do not need to be an expert in Scala to use Mill, just like you do not need to be an expert in Groovy to use Gradle. Because Mill has great IDE support, and does the right things by default, I hope it would be much easier for a non-expert to contribute to a Mill build than it would be for a non-expert to contribute to Gradle

Conclusion

Both the Mill and Gradle builds we discussed in this case study do the same thing: they compile Java code and run tests. Sometimes they perform additional configuration, tweaking JVM arguments or doing ad-hoc classpath mangling.

In general, building projects with Mill is significantly faster than Gradle, but the gap is not as big as when comparing Mill v.s. Maven. Mill builds do all the same things as gradle builds, and need to manage the same kind of complexity. But where Mill shines over Gradle is just the understandability of the build: while Gradle is famously confusing and opaque, Mill’s great IDE support allows the user to explore and understand their build as easily as any application codebase, and its fool-proof approach to extensibility means non-experts can confidently modify or add to their build system without worrying about getting it wrong.

Again, the Mill build used in this comparison is for demonstration purposes, and more work would be necessary to make the Mill build production ready: publishing configuration, code coverage integration, and so on. Furthermore, Mill is definitely not perfect, and it is a work in progress to improve the user experience and iron out bugs. However, hopefully this comparison demonstrates the potential value, and convinces you to give it a try!