Zero-Setup All-in-One Java Tooling via Mill Bootstrap Scripts

Perhaps the most common way software is installed is via package managers like apt, yum, or brew. For example, the incantation to install git in an Amazon-Linux machine is:

Depending on how nicely the software you are installing is packaged, this may or may not require additional commands to install transitive dependencies. For example, when setting up a codebase for development, you may need to:

In the JVM ecosystem, it is common to need to:

Such multi-step workflows are common when building a software project, as the codebase and its dependencies are never as nicely packaged as distributed binaries like git. Using a language-specific package manager or build tool can help, but since the build tool itself needs to be installed it remains a multi-step workflow getting everything set up and ready to use.

There are other ways to install things apart from package managers: curl <url> | bash is common, as is manually downloading binaries to put on your PATH. But all of these have a similar problem: the installation must be done manually and happen before you can begin working on your project. This gives a lot of room for things to go wrong, for example:

Multi-step setup workflows are the norm, an 1-step setup workflows are something people often strive towards. But it’s worth asking: could we do better?

One interesting innovation on the installation process is the use of a bootstrap script. These were popularized by the Gradle build tool as a ./gradlew bootstrap script you commit to the repository root. The bootstrap script embeds the version of Gradle you want to use, and ensures to download and cache that specific version when it is invoked. That means you can checkout a project’s code and run:

And be sure you are using the same version of Gradle that everyone else is also using to build that project. This can be very handy: you now no longer need to worry about installing the "right version" of Gradle on your colleagues' laptops, on CI, etc. The bootstrap script ensures that anyone working on the project - human or otherwise - will be using the same version.

Furthermore, as tools like Gradle automatically resolve the application-level dependencies required by the project they are building, the user does not need to install those manually. Any build or install or test command results in all necessary dependencies being automatically downloaded and cached as necessary. More recently, the Maven build tool has adopted a similar convention with ./mvnw scripts serving the same purpose.

However, one limitation of the Maven and Gradle approach to bootstrap scripts is that they rely on java being pre-installed to begin the bootstrapping process. Without java, they cannot run at all, as shown below:

So even with the ./gradlew or ./mvnw bootstrap scripts, working with Gradle or Maven still ends up being a 1-step installation process: you need to install java (and the right version of Java!) before you begin, possibly using SdkMan or Jenv, each of which themselves need to be installed first. Thus although these bootstrap scripts mitigate the problem - differing Java versions tend to be more forgiving than differing Maven/Gradle versions - they haven’t completely solved it.

Why do these bootstrap scripts have require java to be installed? It’s because they don’t want to put non-trivial bootstrapping logic into .sh or .bat scripts, and as JVM build tools writing their bootstrapping logic in Java running on the JVM makes sense. But that doesn’t seem like a hard requirement, and it should be possible to make a bootstrapping binary that can run without java or any other runtime pre-installed. That is the approach that Mill takes.

Mill’s ./mill bootstrap scripts are similar to ./mvnw or ./gradle, but differ in that by default they do not require java pre-installed in order to run. Instead, ./mill downloads a native platform-specific binary that then performs the bootstrapping process:

These .exe files are JVM executables, but compiled to native platform-specific binaries using the Graal Native Image compiler. Apart from the benefits of reduced startup time and memory usage, the key property we care about is that native image binaries also can run on bare environments without a java runtime pre-installed. This lets us write our non-trivial bootstrapping logic in Java and run it without needing a system-wide java distribution pre-installed on the machine.

As native image binaries are OS/CPU-specific, we need some logic to pick the right binary for the machine the bootstrap script it running on, and that logic needs to run in the .sh or .bat bootstrap script because we need it to run before the native image binary has been downloaded. The .sh version of this implemented using uname is as follows:

The bootstrap script can then assemble this into a download URL to curl down the relevant file from the Maven Central package repository:

We can then execute the downloaded file, taking any command line arguments given to the bootstrap script and forwarding them to the native binary:

The snippets above are somewhat simplified - the actual bootstrap script contains a lot more logic to handle backwards compatibility, version configuration, Windows support, and other necessary details. But at a high level, they illustrate what Mill’s bootstrap script does: it picks the downloads the native binary of the configured version, operating system, and CPU architecture, and executes it to begin the Mill bootstrapping process. This lets it bootstrap from shell/bat script to native image binary without any prior installation of java or other system-wide dependencies, and from there we can bootstrap the rest of the way.

Once we execute our native image binary, we then have an opportunity to run real JVM code (as opposed to sketchy shell scripts) to proceed with bootstrapping. When someone runs ./mill __.compile to compile all modules in a repository, and the native image bootstrap launcher has been downloaded as described above, we can then use it to:

Once we have the Mill daemon process running, further steps are necessary to bootstrap the Mill build dependencies and user code dependencies

The various .jar files are typically downloaded from Maven Central, which is the standard package repository for JVM libraries. The JVMs themselves come from the various provider download URLs that we reference via the Coursier JVM Index. Apart from libraries and JVMs, all tools necessary for your Java/Scala/Kotlin development are also bootstrapped the same way - Checkstyle, ErrorProne, ScalaFmt, KtLint, etc. - so you can use them without needing prior system-wide setup or installation.

Note that we only do these steps once the native image bootstrap launcher has been downloaded as they require non-trivial logic: resolving JVM versions to download URLs, resolving .jar files from group-artifact-version coordinates, adjudicating version conflicts, etc. This is too complicated to implement in .sh and .bat scripts, so Mill handles that using Coursier which is a common JVM dependency resolution library also used by Bazel and SBT.

The final bootstrapping process of ./mill __.compile looks something like this, with the solid lines indicating local steps in the bootstrapping process, and the dashed lines indicating downloads from package repositories:

Although this may seem like a lot of steps, all of them are completely automatic, and generally invisible to the user:

This is unlike packages installed via brew or apt or yum, where installation often has to be done manually, and typically only a single version of a package can be "installed" or "active" globally on a system at any one point in time. While traditional package management and program installation often involves manual work to set up and maintain, Mill’s handling of dependencies in this bootstrap process is largely hands-off and automated.

Despite the complexity described above, Mill’s zero-install bootstrap process means that the user never needs to deal with any of it. They can immediately start using ./mill __.compile or or any other command on a clean system, and the only indication noticeable difference would be the first command taking longer than normal and logging indicating that these downloads are happening. And once caches are warm, running ./mill feels just as fast as running any pre-installed binary or executable.

In this article, we discussed how the Mill build tool implements its zero-step setup process. This removes the zoo of manual installs that a Java developer would traditionally need to setup and maintain (mvn, jenv, sdkman, java, etc.), and replaces it with a single ./mill script that automatically bootstraps all necessary tools and runtimes for the user, letting them begin their work on a codebase without any prior setup.

This is done by carefully arranging the bootstrapping process for the Mill project: starting from a .sh script (or .bat on windows), using it to bootstrap a native binary, using the native binary to bootstrap a JVM, and using the JVM to bootstrap the user-defined dependencies they need to build their project. Although both the Mill build tool itself and user projects built with Mill both may have large transitive dependency trees, the bootstrapping process is arranged in a way that it can all be handled entirely automatically.

For the purposes of this article, we simplified and skimmed over a lot of things:

Although this article covers bootstrapping Java and JVM applications, the same principles could apply to bootstrap any non-trivial project and its dependencies: starting from a shell script, bootstrapping a native binary, which then bootstraps the messy dependencies that are required for any real-world project. With Mill, we take advantage of this to try and simplify Java development: codebases built using Mill can be built via ./mill out of the box, providing everything you need for development without any prior setup. We hope that this will make it easier to people to contribute to such projects, whether in a proprietary setting or open-source.

Zero-step installation workflows are really the only thing that scales as a project grows. While multiple 1-step installs can add up and become a long N-step installation process, multiple zero-step installs will always remain zero-step even if added together, regardless of how large and messy the project gets. Hopefully you’ve come away from this article with an appreciation for how Mill builds upon prior art to come up with its zero-step setup process, so next time the opportunity arises you can implement something similar in your own projects.