Case Study: Mill vs Maven
Compared to Maven, Mill tries to improve in the following ways:
-
Performance: Mill compiles the same project 4-6x faster than Maven. A Maven "clean install" workflow taking over a minute might take just a few seconds in Mill. It does this by aggressively parallelizing and caching your build at all levels.
-
Extensibility: Mill makes customizing the build tool much easier than Maven. Projects usually grow beyond just compiling a single language: needing custom code generation, linting workflows, tool integrations, output artifacts, or support for additional languages. Mill’s extensibility and IDE experience makes doing this yourself easy and safe, with type-checked code and sandboxed tasks.
-
IDE Support: Jump-to-definition, autocomplete, show-documentation, etc. are all much more useful in
build.millfiles than in `pom.xml`s. This makes it easier to navigate, understand, and maintain your build configuration without being limited by the quantity and quality of online documentation (or the lack thereof!)
This page compares using Mill to Maven using the Netty Network Server
codebase as the example. Netty is a large, old codebase. 500,000 lines of Java, written by
over 100 contributors across 15 years, split over 47 subprojects, with over 10,000 lines of
Maven pom.xml configuration alone. By porting it to Mill, this case study should give you
an idea of how Mill compares to Maven in larger, real-world projects.
To do this, we have written a Mill build.mill file for the Netty project. This can be used
with Mill to build and test the various submodules of the Netty project without needing to
change any other files in the repository:
Completeness
The Mill build for Netty is not 100% complete, but it covers most of the major parts of Netty: compiling Java, compiling and linking C code via JNI, custom codegen using Groovy scripts, running JUnit tests and some integration tests using H2Spec. All 47 Maven subprojects are modelled using Mill, with the entire Netty codebase being approximately 500,000 lines of code.
$ git ls-files | grep \\.java | xargs wc -l
...
513805 totalThe goal of this exercise is not to be 100% feature complete enough to replace the Maven build today. It is instead meant to provide a realistic comparison of how using Mill in a large, complex project compares to using Maven.
Both Mill and Maven builds end up compiling the same set of files, although the number being
reported by the command line is slightly higher for Mill (2915 files) than Maven (2822) due
to minor differences in the reporting (e.g. Maven does not report package-info.java files
as part of the compiled file count).
Performance
The Mill build for Netty is much more performant than the default Maven build. This applies to most workflows.
For the benchmarks below, each provided number is the wall time of three consecutive runs
on my M1 Macbook Pro using Java 17 and Mill 0.12.9-native. While ad-hoc, these benchmarks
are enough to give you a flavor of how Mill’s performance compares to Maven:
| Benchmark | Maven | Mill | Speedup |
|---|---|---|---|
98.80s |
23.41s |
4.2x |
|
48.92s |
9.29s |
5.3x |
|
4.89s |
0.88s |
5.6x |
|
6.82s |
0.18s |
37.9x |
|
5.25s |
0.12s |
43.8x |
The column on the right shows the speedups of how much faster Mill is compared to the equivalent Maven workflow. In most cases, Mill is 4-6x faster than Maven. Below, we will go into more detail of each benchmark: how they were run, what they mean, and how we can explain the difference in performing the same task with the two different build tools.
Sequential Clean Compile All
$ ./mvnw clean; time ./mvnw -Pfast -Dcheckstyle.skip -Denforcer.skip=true -DskipTests install
98.80s
96.14s
99.95s
$ ./mill clean; time ./mill -j1 __.compile
22.83s
23.41s
23.47sThis benchmark exercises the simple "build everything from scratch" workflow, with all remote
artifacts already in the local cache. The actual files
being compiled are the same in either case (as mentioned in the Completeness section).
I have explicitly disabled the various linters and tests for the Maven build, to just focus
on the compilation of Java source code making it an apples-to-apples comparison. As Mill
runs tasks in parallel by default, I have disabled parallelism explicitly via -j1
As a point of reference, Java typically compiles at 10,000-50,000 lines per second on a single thread, and the Netty codebase is ~500,000 lines of code, so we would expect compile to take 10-50 seconds without parallelism. The 20-30s taken by Mill seems about what you would expect for a codebase of this size, and the ~100s taken by Maven is far beyond what you would expect from simple Java compilation.
Maven Compile vs Install
In general, the reason we have to use ./mvnw install rather than ./mvnw compile is that
Maven’s main mechanism for managing inter-module dependencies is via the local artifact cache
at ~/.m2/repository. Although many workflows work with compile, some don’t, and
./mvnw clean compile on the Netty repository fails with:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.10:unpack-dependencies (unpack) on project netty-resolver-dns-native-macos: Artifact has not been packaged yet. When used on reactor artifact, unpack should be executed after packaging: see MDEP-98. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :netty-resolver-dns-native-macos
In contrast, Mill builds do not rely on the local artifact cache, even though Mill is able
to publish to it. That means Mill builds are able to work directly with classfiles on disk,
simply referencing them and using them as-is without spending time packing and unpacking them
into .jar files. But even if we did want Mill to generate the .jars, the
overhead of doing so is just a few seconds, far less than the minutes that
Maven’s overhead adds to the clean build:
$ ./mill clean; time ./mill -j1 __.jar
26.74s
26.02s
26.53sFrom this benchmark, we can see that although both Mill and Maven are doing the same work,
Mill takes about as long as it should for this task of compiling 500,000 lines of Java source
code, while Maven takes considerably longer. This difference is purely build tool overhead
in Maven - in the install workflow, in the JVM process warmup, etc. - that Mill manages
to avoid.
Parallel Clean Compile All
$ ./mvnw clean; time ./mvnw -T 10 -Pfast -DskipTests -Dcheckstyle.skip -Denforcer.skip=true install
48.92s
48.41s
49.50s
$ ./mill clean; time ./mill __.compile
10.95s
8.51s
9.29sThis example compares Maven v.s. Mill, when performing the clean build on 10 threads.
Both build tools support parallelism (-T 10 in Maven, by default in Mill), and both
tools see a similar ~2x speedup for building the Netty project using 4 threads.Again,
this tests a clean build using ./mvnw clean or ./mill clean.
This comparison shows that much of Mill’s speedup over Maven is unrelated to parallelism. Whether sequential or parallel, Mill has approximately the same 4-5x speedup over Maven when performing a clean build of the Netty repository.
Clean Compile Single-Module
$ ./mvnw clean; time ./mvnw -pl common -Pfast -DskipTests -Dcheckstyle.skip -Denforcer.skip=true -Dmaven.test.skip=true install
4.85s
4.96s
4.89s
$ ./mill clean common; time ./mill common.compile
0.88s
0.97s
0.73sThis exercise limits the comparison to compiling a single module, in this case common/,
ignoring test sources.
Again, we can see a significant speedup of Mill v.s. Maven remains even when compiling a
single module: a clean compile of common/ is about 6x faster with Mill than with Maven!
Again, common/ is about 30,000 lines of Java source code, so at 10,000-50,000 lines per
second we would expect it to compile in about 1-4s. That puts Mill’s compile times right
at what you would expect, whereas Maven’s has a significant overhead.
Incremental Compile Single-Module
$ echo "" >> common/src/main/java/io/netty/util/AbstractConstant.java
$ time ./mvnw -pl common -Pfast -DskipTests -Dcheckstyle.skip -Denforcer.skip=true install
Compiling 174 source files to /Users/lihaoyi/Github/netty/common/target/classes
Compiling 60 source files to /Users/lihaoyi/Github/netty/common/target/test-classes
6.89s
6.34s
6.82s
$ echo "" >> common/src/main/java/io/netty/util/AbstractConstant.java
$ time ./mill common.test.compile
compiling 1 Java source to /Users/lihaoyi/Github/netty/out/common/compile.dest/classes ...
0.18s
0.18s
0.21sThis benchmark explores editing a single file and re-compiling common/.
Maven by default takes about as long to re-compile common/s main/ and test/ sources
after a single-line edit as it does from scratch, about 20 seconds. However, Mill
takes just about 0.5s to compile and be done! Looking at the logs, we can see it is
because Mill only compiles the single file we changed, and not the others.
For this incremental compilation, Mill uses the Zinc Incremental Compiler. Zinc is able to analyze the dependencies between files to figure out what needs to re-compile: for an internal change that doesn’t affect downstream compilation (e.g. changing a string literal) Zinc only needs to compile the file that changed, taking barely half a second:
$ git diff
diff --git a/common/src/main/java/io/netty/util/AbstractConstant.java b/common/src/main/java/io/netty/util/AbstractConstant.java
index de16653cee..9818f6b3ce 100644
--- a/common/src/main/java/io/netty/util/AbstractConstant.java
+++ b/common/src/main/java/io/netty/util/AbstractConstant.java
@@ -83,7 +83,7 @@ public abstract class AbstractConstant<T extends AbstractConstant<T>> implements
return 1;
}
- throw new Error("failed to compare two different constants");
+ throw new Error("failed to compare two different CONSTANTS!!");
}
}$ time ./mill common.test.compile
[info] compiling 1 Java source to /Users/lihaoyi/Github/netty/out/common/compile.dest/classes ...
0m 00.55s6In contrast, a change to a class or function public signature (e.g. adding a method) may require downstream code to re-compile, and we can see that below:
$ git diff
diff --git a/common/src/main/java/io/netty/util/AbstractConstant.java b/common/src/main/java/io/netty/util/AbstractConstant.java
index de16653cee..f5f5a93e0d 100644
--- a/common/src/main/java/io/netty/util/AbstractConstant.java
+++ b/common/src/main/java/io/netty/util/AbstractConstant.java
@@ -41,6 +41,10 @@ public abstract class AbstractConstant<T extends AbstractConstant<T>> implements
return name;
}
+ public final String name2() {
+ return name;
+ }
+
@Override
public final int id() {
return id;$ time ./mill common.test.compile
[25/48] common.compile
[info] compiling 1 Java source to /Users/lihaoyi/Github/netty/out/common/compile.dest/classes ...
[info] compiling 2 Java sources to /Users/lihaoyi/Github/netty/out/common/compile.dest/classes ...
[info] compiling 4 Java sources to /Users/lihaoyi/Github/netty/out/common/compile.dest/classes ...
[info] compiling 3 Java sources to /Users/lihaoyi/Github/netty/out/common/test/compile.super/mill/scalalib/JavaModule/compile.dest/classes ...
[info] compiling 1 Java source to /Users/lihaoyi/Github/netty/out/common/test/compile.super/mill/scalalib/JavaModule/compile.dest/classes ...
0m 00.81s2Here, we can see that Zinc ended up re-compiling 7 files in common/src/main/ and 3 files
in common/src/test/ as a result of adding a method to AbstractConstant.java.
In general, Zinc is conservative, and does not always end up selecting the minimal set of
files that need re-compiling: e.g. in the above example, the new method name2 does not
interfere with any existing method, and the ~9 downstream files did not actually need to
be re-compiled! However, even conservatively re-compiling 9 files is much faster than
Maven blindly re-compiling all 234 files, and as a result the iteration loop of
editing-compiling-testing your Java projects in Mill can be much faster than doing
the same thing in Maven
No-Op Compile Single-Module
$ time ./mvnw -pl common -Pfast -DskipTests -Dcheckstyle.skip -Denforcer.skip=true install
5.08s
5.25s
5.26s
$ time ./mill common.test.compile
0.14s
0.12s
0.12sThis last benchmark explores the boundaries of Maven and Mill: what happens if we ask to compile a single module that has already been compiled? In this case, there is literally nothing to do. For Maven, "doing nothing" takes ~17 seconds, whereas for Mill we can see it complete and return in less than 0.5 seconds
Grepping the logs, we can confirm that both build tools skip re-compilation of the
common/ source code. In Maven, skipping compilation only saves us ~2 seconds,
bringing down the 19s we saw in Clean Compile Single-Module to 17s here. This
matches what we expect about Java compilation speed, with the 2s savings on
40,000 lines of code telling us Java compiles at ~20,000 lines per second. However,
we still see Maven taking 17 entire seconds before it can decide to do nothing!
In contrast, doing the same no-op compile using Mill, we see the timing from 2.2s in Clean Compile Single-Module to 0.5 seconds here. This is the same ~2s reduction we saw with Maven, but due to Mill’s minimal overhead, in the end the command finishes in less than half a second.
Extensibility
Even though Maven is designed to be declarative, in many real-world codebases you end up needing to run ad-hoc scripts and logic. This section will explore one such scenario, so you can see how Mill differs from Maven in the handling of these requirements.
The Maven build for the common/ subproject
uses a Groovy script for code generation. This is configured via:
<properties>
<collection.template.dir>${project.basedir}/src/main/templates</collection.template.dir>
<collection.template.test.dir>${project.basedir}/src/test/templates</collection.template.test.dir>
<collection.src.dir>${project.build.directory}/generated-sources/collections/java</collection.src.dir>
<collection.testsrc.dir>${project.build.directory}/generated-test-sources/collections/java</collection.testsrc.dir>
</properties>
<plugin>
<groupId>org.codehaus.gmaven</groupId>
<artifactId>groovy-maven-plugin</artifactId>
<version>2.1.1</version>
<dependencies>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy</artifactId>
<version>3.0.9</version>
</dependency>
<dependency>
<groupId>ant</groupId>
<artifactId>ant-optional</artifactId>
<version>1.5.3-1</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>generate-collections</id>
<phase>generate-sources</phase>
<goals>
<goal>execute</goal>
</goals>
<configuration>
<source>${project.basedir}/src/main/script/codegen.groovy</source>
</configuration>
</execution>
</executions>
</plugin>In contrast, the Mill build configures the code generation as follows:
import $ivy.`org.codehaus.groovy:groovy:3.0.9`
import $ivy.`org.codehaus.groovy:groovy-ant:3.0.9`
import $ivy.`ant:ant-optional:1.5.3-1`
object common extends NettyModule{
...
def script = Task.Source("src" / "main" / "script")
def generatedSources = Task {
val shell = new groovy.lang.GroovyShell()
val context = new java.util.HashMap[String, Object]
context.put("collection.template.dir", s"${Task.workspace}/common/src/main/templates")
context.put("collection.template.test.dir", s"${Task.workspace}/common/src/test/templates")
context.put("collection.src.dir", (Task.dest / "src").toString)
context.put("collection.testsrc.dir", (Task.dest / "testsrc").toString)
shell.setProperty("properties", context)
shell.setProperty("ant", new groovy.ant.AntBuilder())
shell.evaluate((script().path / "codegen.groovy").toIO)
PathRef(Task.dest / "src")
}
}The number of lines of code written is not that different, and in fact both Mill and Maven
configs need to do similar things: setting collection.src.dir, invoking org.codehaus.groovy,
and so on. Where things differ is the amount of indirection: while Maven has us
configuring a third-party groovy-maven-plugin artifact, in Mill
we can import org.codehaus.groovy:groovy:3.0.9 and instantiate a GroovyShell directly.
to evaluate our codegen.groovy script.
This direct control means you are not beholden to third party plugins: rather than being limited to what an existing plugin allows you to do, Mill allows you to directly write the code necessary to do what you need to do. In this case, if we need to invoke Groovy and Groovy-Ant, Mill allows us to direct import $ivy the relevant JVM artifacts from Maven Central
Mill gives you the full power of the JVM ecosystem to use in your build: any Java library
on Maven central is just an import $ivy away, and can be used with the full IDE support
and tooling experience you are used to in the JVM ecosystem.
IDE Support
Mill build files contain code written in a strongly-typed fashion, with full autocomplete and code assistance. As a baseline, consider this Maven configuration where we are generating some source files:
If you weren’t sure what build-helper-maven-plugin does, the obvious thing
to reach for is to try and jump-to-definition in your IDE. If you do that in Intellij,
you get:
This gives you the signature of the config value: sources is an array of files,
it is required, and its description is "additional source directories". While
this is better than nothing, it is less than what you would expect if you use
jump-to-definition in any application codebase.
In contrast, when using Mill, not only do you get full autocomplete to explore available methods, their signatures, and their documentation:
We can jump to definition, e.g. going from our own def generatedSources
to the original definition that was overridden:
From here, you can trace through the data flow, seeing how def generatedSources
gets used by def sources, then def allSources:
An allSources eventually gets used in def compile:
None of this should be a surprise to anyone using a JVM language: Java, Kotlin, and Scala have had this kind of IDE experience for decades! However, it stands in stark contrast to the experience using IDEs with tools like Maven, where although the IDE is able to superficially understand what XML entries are allowed where, they do not help at all in understand how the various configuration values actually end up affecting your build.
With Mill, you get your full IDE experience working with your build: autocomplete, code assistance, navigation, and so on. You can explore and navigate around your build just like you would any application codebase, avoiding the feeling of "hitting a wall" that often occurs when trying to figure out why a Maven build or plugin does not behave quite as you expect. Where with Maven you may find yourself searching online docs or digging through plugin source code on Github, with Mill you can comfortably navigate the build logic in your IDE as easily as any application codebase.
Conclusion
Both the Mill and Maven builds we discussed in this case study do the same thing: they
compile Java code, zip them into Jar files, run tests. Sometimes they compile and link
C code or run make or Groovy to accomplish what they need to do.
Mill doesn’t do more than Maven does, but it tries to do it better: faster compiles,
easier extensibility via libraries (e.g. org.codehaus.groovy:groovy), better IDE
support for working with your build.
Again, the Mill build used in this comparison is for demonstration purposes, and more work would be necessary to make the Mill build production ready: compatibility with different operating system architectures, Java versions, and so on. However, hopefully it demonstrates the potential value: greatly improved performance, easy extensibility, and a much better IDE experience for working with your build. Mill provides builtin tools to help you navigate, visualize, and understand your build, turning a normally opaque "build config" into something that’s transparent and easily understandable.