Mill Build Performance

To compare Mill vs Maven, we took the Netty project and ported its Maven build to Mill as a proof of concept, complete enough that we could get all tests passing on a M1 Macbook Pro. Netty is not a small project: ~500,000 lines of code, ~50 submodules, and build customizations such as compiling C code, running Groovy code generators, and other things. After porting all of that from Maven to Mill, we then ran the same common workflows on both equivalent builds to compare the performance: compiling everything, compiling a single module, etc.

Overall across our benchmarks, Mill is 4-6x faster than Maven for clean compiles, both parallel and sequential, and for many modules or for a single module:

First, let’s look at Parallel Clean Compile All. This benchmark involves running clean to delete all generated files and re-compiling everything in parallel. Mill sees a significant ~5x speedup over Maven for this benchmark. The same number of Java files are compiled, with the same classpath, in the same module layout, with the same Java compiler: the only difference is the reduction in build tool overhead giving the 5x speedup.

To get a feel for this difference, play recording below to see Mill (Left) and Maven (Right) running side by side.

The second benchmark worth noting is Incremental Compile Single Module. This benchmark involves adding a single newline to the end of a single already-compiled file in common, and re-compiling common and common.test. Mill sees a huge speedup for this benchmark, because Mill’s incremental compiler (Zinc) is able to detect that only one file in one module has changed, and that the change is small enough to not require other files to re-compile. In contrast, Maven re-compiles all files in both modules, even though only one file was touched and the change was trivial.

Looking at these numbers, it is not surprising that the Java platform has a reputation for being slow and clunky. Even in the No-Op Compile Single Module case, it takes ~5 seconds for Maven to realize there’s nothing to do!

However, with Mill it looks very different:

If Java developers had this kind of fast turnaround working on their code, no doubt Java would have a very different reputation than it has today.

For more details on these benchmarks, including steps to reproduce them, see Mill vs Maven In Depth

We used the Mockito codebase as the case study of Mill vs Gradle. Again, we ported it from Gradle to Mill and got all code compiling and all tests passing. Then we ran benchmarks of common workflows comparing the time taken for both Mill and Gradle to do the same work.

Mill’s various "clean compile" workflows 3-4x faster than Gradle’s, while it’s incremental and no-op compile workflows are 7-9x faster. Both Gradle and Mill appear to do a good job limiting the compilation to only the changed file, but Mill has less fixed overhead than Gradle does, finishing in about ~0.2s rather than ~1.5 seconds.

In general, these benchmarks don’t show Mill doing anything that Maven or Gradle do not: these are equivalent builds for the same projects (Netty and Mockito respectively), compiling the same number of files using the same Java compiler, in the same module structure and passing the same suite of tests. Rather, what we are seeing is Mill simply having less build-tool overhead than Maven or Gradle, so the performance of the underlying JVM and Java compiler (which is actually pretty fast!) can really shine through.

For more details on these Gradle benchmarks, with steps to reproduce them, see Mill vs Gradle In Depth

Compilation times are only one part of the time taken during development. Mill also provides features that help speed up other parts of your development cycle:

None of these features are rocket science, and they are all things that you can in theory set up with other build tools. However, Mill provides these features built-in without needing to first hunt down plugins or third-party integrations, and makes all of them easy to set up and use.

This section goes into the weeds comparing the Mill and Maven build tool performance. To do this, we have written a Mill build.mill file for the Netty project. This can be used with Mill to build and test the various submodules of the Netty project without needing to change any other files in the repository:

To reproduce these benchmarks locally, you can download and unzip

And run the commands listed in the below sections yourself. Note that as Mill runs a JVM background daemon, it may require a few runs for the JVM to warm up to peak performance, and do be aware that benchmarked performance is expected to differ on different operating systems and hardware

The Mill build for Netty is not 100% complete, but it covers most of the major parts of Netty: compiling Java, compiling and linking C code via JNI, custom codegen using Groovy scripts, running JUnit tests and some integration tests using H2Spec. All 47 Maven subprojects are modelled using Mill, with the entire Netty codebase being approximately 500,000 lines of code.

The goal of this exercise is not to be 100% feature complete enough to replace the Maven build today. It is instead meant to provide a realistic comparison of how using Mill in a large, complex project compares to using Maven.

Both Mill and Maven builds end up compiling the same set of files, although the number being reported by the command line is slightly higher for Mill (2915 files) than Maven (2822) due to minor differences in the reporting (e.g. Maven does not report package-info.java files as part of the compiled file count).

The Mill build for Netty is much more performant than the default Maven build. This applies to most workflows.

For the benchmarks below, each provided number is the wall time of three consecutive runs on my M1 Macbook Pro using Java 17 and Mill 0.12.9-native. While ad-hoc, these benchmarks are enough to give you a flavor of how Mill’s performance compares to Maven:

The column on the right shows the speedups of how much faster Mill is compared to the equivalent Maven workflow. In most cases, Mill is 4-6x faster than Maven. Below, we will go into more detail of each benchmark: how they were run, what they mean, and how we can explain the difference in performing the same task with the two different build tools.

This section compares using Mill to Gradle using the Mockito Testing Library codebase as the example. Mockito is a medium sized codebase, 100,000 lines of Java split over 22 subprojects. By porting it to Mill, this case study should give you an idea of how Mill compares to Gradle in more realistic, real-world projects.

To do this, we have written a Mill build.mill file for the Mockito project. This can be used with Mill to build and test the various submodules of the Mockito project without needing to change any other files in the repository:

To reproduce these benchmarks locally, you can download and unzip

And run the commands listed in the below sections yourself. Note that as Mill runs a JVM background daemon, it may require a few runs for the JVM to warm up to peak performance, and do be aware that benchmarked performance is expected to differ on different operating systems and hardware

The Mill build for Mockito is not 100% complete, but it covers most of the major parts of Mockito: compiling Java, running JUnit tests. For now, the Android, Kotlin, and OSGI tests are skipped, as support for Building Android apps in Mill and Kotlin with Mill is still experimental.

The goal of this exercise is not to be 100% feature complete enough to replace the Gradle build today. It is instead meant to provide a realistic comparison of how using Mill in a realistic, real-world project compares to using Gradle.

The Mill build for Mockito is generally snappier than the Gradle build. This applies to most workflows, but the difference matters most for workflows which are short-lived, where the difference in the fixed overhead of the build tool is most noticeable.

For comparison purposes, I disabled the Gradle subprojects that we did not fully implement in Mill (groovyTest, groovyInlineTest, kotlinTest, kotlinReleaseCoroutinesTest, android, osgi-test, java21-test), and added the necessary flags to ensure caching/parallelism/etc. is configured similarly for both tools. This ensures the comparison is fair with both builds compiling the same code and running the same tests in the same way.

For the benchmarks below, each provided number is the median wall time of three consecutive runs on my M1 Macbook Pro with Java 17 and Mill 0.12.9-native. While ad-hoc, these benchmarks are enough to give you a flavor of how Mill’s performance compares to Gradle:

The column on the right shows the speedups of how much faster Mill is compared to the equivalent Gradle workflow. In most cases, Mill is 2-4x faster than Gradle. Below, we will go into more detail of each benchmark: how they were run, what they mean, and how we can explain the difference in performing the same task with the two different build tools.

The Mill, Gradle, and Maven builds we discussed in this case study do the same thing: they compile Java code and run tests. Sometimes they perform additional configuration, tweaking JVM arguments or doing ad-hoc classpath mangling. While the Mill builds are not 100% feature complete - e.g. the Mill build for Netty doesn’t support building on a half-dozen different operating systems like the Maven build does - they are complete enough to pass all tests and be a realistic comparison.

But the bottom line is that, building projects with Mill is significantly faster than Maven or Gradle. Things like 3-6x faster compiles, selective test execution, incremental assembly jars, automatic build profiling all add up to make Mill a substantially snappier build tool than what you might be used to. If you’ve ever been frustrated by slowness in your JVM development process, you should try Mill to see if it can give you a very different experience!