Case Study: Mill vs Gradle

This page compares using Mill to Gradle, using the Mockito Testing Library codebase as the example. Mockito is a medium sized codebase, 100,000 lines of Java split over 22 subprojects. By porting it to Mill, this case study should give you an idea of how Mill compares to Gradle in more realistic, real-world projects.

To do this, we have written a Mill build.mill file for the Mockito project. This can be used with Mill to build and test the various submodules of the Mockito project without needing to change any other files in the repository:

Mockito build.mill file

Completeness

The Mill build for Mockito is not 100% complete, but it covers most of the major parts of Mockito: compiling Java, running JUnit tests. For now, the Android, Kotlin, and OSGI tests are skipped.

The goal of this exercise is not to be 100% feature complete enough to replace the Gradle build today. It is instead meant to provide a realistic comparison of how using Mill in a realistic, real-world project compares to using Gradle.

Performance

The Mill build for Mockito is generally snappier than the Gradle build. This applies to most workflows, but the difference matters most for workflows which are short-lived, where the difference in the fixed overhead of the build tool is most noticeable.

For comparison purposes, I disabled the Gradle subprojects that we did not fully implement in Mill (groovyTest, groovyInlineTest, kotlinTest, kotlinReleaseCoroutinesTest, android, osgi-test, java21-test).

For the benchmarks below, each provided number is the median wall time of three consecutive runs on my M1 Macbook Pro. While ad-hoc, these benchmarks are enough to give you a flavor of how Mill’s performance compares to Gradle:

Benchmark	Gradle	Mill	Speedup
Sequential Clean Compile All	17.6s	5.40s	3.3x
Parallel Clean Compile All	12.3s	3.57s	3.4x
Clean Compile Single-Module	4.41s	1.20s	3.7x
Incremental Compile Single-Module	1.37s	0.51s	2.7x
No-Op Compile Single-Module	0.94s	0.46s	2.0x

Benchmark

Gradle

Mill

Speedup

Sequential Clean Compile All

17.6s

5.40s

3.3x

Parallel Clean Compile All

12.3s

3.57s

3.4x

Clean Compile Single-Module

4.41s

1.20s

3.7x

Incremental Compile Single-Module

1.37s

0.51s

2.7x

No-Op Compile Single-Module

0.94s

0.46s

2.0x

The column on the right shows the speedups of how much faster Mill is compared to the equivalent Gradle workflow. In most cases, Mill is 5-10x faster than Gradle. Below, we will go into more detail of each benchmark: how they were run, what they mean, and how we can explain the difference in performing the same task with the two different build tools.

Sequential Clean Compile All

$ ./gradlew clean; time ./gradlew classes testClasses --no-build-cache
17.6s
18.2s
17.4s

$ ./mill clean; time ./mill -j 1 __.compile
5.60s
5.40s
6.13s

This benchmark measures the time taken to sequentially compiled all the Java code in the Mockito code base. The goal of this benchmark is to try and measure the "clean compile everything" step, without the effects of parallelism that can be nondeterministic and vary wildly from machine to machine depending on the number of cores available.

To limit our comparison to compiling Java code only, we avoid using build in favor of classes and testClasses: this skips running tests, lint, jar/docjar generation, and other steps that build performs to make it an apples-to-apples comparison. Furthermore, Gradle parallelizes the build by default and caches things globally under ~/.gradle/caches, while Mill parallelizes by default but does not cache things globally. Again to make it a fair comparison, we use --no-build-cache in Gradle and set org.gradle.parallel=false in gradle.properties, and pass -j 1 to limit Mill to a single thread.

Here we see Mill being about ~3.3x faster than Gradle, to do the equivalent amount of work.

Parallel Clean Compile All

$ ./gradlew clean; time ./gradlew classes testClasses --no-build-cache
13.8s
12.3s
11.4s

$ ./mill clean; time ./mill -j 10  __.compile
3.59s
3.57s
3.45s

This benchmark is identical to the Sequential Clean Compile All benchmark above, but enables parallelism: Gradle by default, Mill via -j 10 to run on 10 cores (the number on my Macbook Pro).

Neither Gradle nor Mill benefit hugely from parallelism: both show a moderate ~50% speedup, despite receiving 900% more CPUs. This likely indicates that the module dependency graph of the Mockito codebase is laid out in a way that does not allow huge amounts of compile-time parallelism.

Again, we see Mill being about ~3.4x faster than Gradle, to do the equivalent amount of work. This indicates the the speedup Mill provides over Gradle is unrelated to the parallelism of each tool.

Clean Compile Single-Module

$ ./gradlew clean; time ./gradlew :classes --no-build-cache
4.14s
4.41s
4.41s

$ ./mill clean; time ./mill compile
1.20s
1.12s
1.30s

This benchmark indicates the use case of clean-compiling a single module. In this case, the root module in src/main/java/ containing the bulk of the Mockito library code, exluding the test code in src/test/java/ and all the downstream subprojects in subprojects/.

This benchmark gives us Mill being about ~3.7x faster than Gradle. This is in line with the results above.

Incremental Compile Single-Module

$ echo "" >> src/main/java/org/mockito/BDDMockito.java; time ./gradlew :classes
1.37s
1.39s
1.28s

$ echo "" >> src/main/java/org/mockito/BDDMockito.java; time ./mill compile
compiling 1 Java source to /Users/lihaoyi/Github/netty/out/common/compile.dest/classes ...
0.52s
0.51s
0.52s

This benchmark measures the common case of making a tiny change to a single file and re-compiling just that module. This is the common workflow that most software developers do over and over day-in and day-out. We simulate this by appending a new line to the file src/main/java/org/mockito/BDDMockito.java.

Both Mill and Gradle are able to take advantage of the small code change and re-compile only the single files needing re-compilation, demonstrating substantial speedups over the Clean Compile Single-Module benchmark above. Mill remains faster than Gradle, showing a ~2.7x speedup for this task

No-Op Compile Single-Module

$ time ./gradlew :classes
0.95s
0.93s
0.94s

$ time ./mill common.compile
0.46s
0.50s
0.45s

This benchmark is meant to measure the pure overhead of running the build tool: given a single module that did not change, the build tool should need to do nothing in response, and so any time taken is pure overhead.

For both Mill and Gradle, we see small speedups relative to the Incremental Compile Single-Module benchmark above, which likely comes from not having to compile any Java source files at all. Mill remains faster than Gradle by about 2.0x.

Debugging Tooling

Another area that Mill does better than Gradle is providing builtin tools for you to understand what your build is doing. For example, the Mockito project build discussed has 22 submodules and associated test suites, but how do these different modules depend on each other? With Mill, you can run ./mill visualize __.compile, and it will show you how the compile task of each module depends on the others:

Apart from the static dependency graph, another thing of interest may be the performance profile and timeline: where the time is spent when you actually compile everything. With Mill, when you run a compilation using ./mill -j 10 __.compile, you automatically get a out/mill-chrome-profile.json file that you can load into your chrome://tracing page and visualize where your build is spending time and where the performance bottlenecks are:

If you want to inspect the tree of third-party dependencies used by any module, the built in ivyDepsTree command lets you do that easily:

$ ./mill subprojects.junit-jupiter.ivyDepsTree
├─ org.junit.jupiter:junit-jupiter-api:5.10.3
│  ├─ org.apiguardian:apiguardian-api:1.1.2
│  ├─ org.junit.platform:junit-platform-commons:1.10.3
│  │  └─ org.apiguardian:apiguardian-api:1.1.2
│  └─ org.opentest4j:opentest4j:1.3.0
└─ org.objenesis:objenesis:3.3

None of these tools are rocket science, but Mill provides all of them out of the box in a convenient package for you to use. Whether you want a visual graph layout, a parallel performance profile, or a third-party dependency tree of your project, Mill makes it easy and convenient without needing to fiddle with custom configuration or third party plugins. This helps make it easy for you to explore, understand, and take ownership of the build tool.

Conclusion

Both the Mill and Gradle builds we discussed in this case study do the same thing: they compile Java code and run tests. Sometimes they perform additional configuration, tweaking JVM arguments or doing ad-hoc classpath mangling.

Mill doesn’t try to do more than Gradle does, but it tries to do it better: faster compiles, shorter and easier to read configs, easier extensibility via libraries.

Again, the Mill build used in this comparison is for demonstration purposes, and more work would be necessary to make the Mill build production ready: publishing configuration, code coverage integration, and so on. However, hopefully it demonstrates the potential value: significantly improved performance, so that you spend less time waiting for your code to compile and more time doing the work that actually matters, with builtin debugging tools to help turn normally opaque "build config" into something that’s transparent and easily understandable.