Running Dynamic JVM Code

This example demonstrates how to resolve a third-party library from Maven Central, but instead of using import $ivy (which loads the library as part of the main build) we use:

While OS-Lib's os.call and os.spawn APIs can be used to create any processes, JVM subprocesses are common enough have enough idiosyncracies (e.g. classpaths) that Mill provides helper methods specifically for them.

For this example, we use the Groovy interpreter as our example third-party library. While often used as a groovy CLI command, Groovy is also available on Maven Central at the org.codehaus.groovy:groovy:3.0.9 coordinates. This lets us pull it into our build as a classpath comprising PathRefs to files on disk, and then run the Groovy JVM main method (in the class groovy.ui.GroovyMain) passing it our script file generate.groovy (wired into our build using an Source Task groovyScript) and arguments used to configure the generated file and tell the script where to write it to. generate.groovy generates a file on disk that we then wire into def resources, which is read at runtime by foo.run and printed to the terminal output as shown below:

As mentioned above, defaultResolver().resolveDeps and Jvm.runSubprocess are an alternative to import $ivy, providing you more flexibility to resolve dependencies on-demand as part of your task graph only when necessary, and keeping it isolated from the build in a subprocess preventing classpath collisions.

This example is similar to the earlier example running the Groovy interpreter in a subprocess, but:

Note that unlike Jvm.runSubprocess, Jvm.runClassloader does not take a workingDir on mainArgs: it instead provides you an in-memory classLoader that contains the classpath you gave it. From there, you can use .loadClass and .getMethod to fish out the classes and methods you want, and .invoke to call them.

Jvm.runClassloader has significantly less overhead than Jvm.runSubprocess: both in terms of wall-clock time and in terms of memory footprint. However, it does have somewhat less isolation, as the code is running inside your JVM and cannot be configured to have a separate working directory, environment variables, and other process-global configs. Which one is better to use differs on a case-by-case basis.

Althought running JVM bytecode via a one-off isolated classloader has less overhead than running it in a subprocess, the fact that the classloader needs to be created each time adds overhead: newly-created classloaders contain code that is not yet optimized by the JVM. When performance matters, you can put the classloader in a Task.Worker to keep it around, allowing the code internally to be optimized and stay optimized without being thrown away each time

This example is similar to the earlier example running the Groovy interpreter in a subprocess, but instead of using Jvm.runSubprocess we use ClassLoader.create to load the Groovy interpreter classpath files:

Here we have two modules foo and bar, each of which makes use of groovyWorker to evaluate a groovy script to generate some resources. In this case, we invoke the main method of groovy.ui.GroovyMain, which also happens to require us to set the ContextClassLoader to work.

Because the URLClassLoader within groovyWorker is long-lived, the code within the classloader can be optimized by the JVM runtime, and would have less overhead than if run in separate classloaders via Jvm.runClassloader. And because URLClassLoader already extends AutoCloseable, groovyWorker gets treated as an Autocloseable Worker automatically.

This example demonstrates using Mill ScalaModules as build tasks: rather than pulling the code we need off of Maven Central, we instead build the code within the bar module as bar/src/Bar.scala.

In this example, we use Bar.scala as a source-code pre-processor on the foo module source code: we override foo.sources, passing the super.sources() and bar.runClasspath to bar.runner().run along with a Task.dest, and returning a PathRef(Task.dest) as the new foo.sources. bar also depends on a third party library OS-Lib. The runner().run subprocess runs inside the Task.dest folder of the enclosing task automatically.

This example does a trivial string-replace of "hello" with "HELLO", but is enough to demonstrate how you can use Mill ScalaModules to implement your own arbitrarily complex transformations. This is useful for build logic that may not fit nicely inside a build.mill file, whether due to the sheer lines of code or due to dependencies that may conflict with the Mill classpath present in build.mill

bar.runner().run by default inherits the mainClass, forkEnv, forkArgs, from the owning module bar, and the working directory from the calling task’s Task.dest. You can also pass in these parameters explicitly to run() as named arguments if you wish to override the defaults.

While the previously example showed how to use the runner().run helpers to run a ScalaModule's code, but you can also use JavaModules for this purpose, with a source code generator written in Java. We also run the bar code within an in-memory classloader via Jvm.runClassloader as we saw earlier:

As mentioned in the section on In-process Isolated Classloaders, this provides less overhead over running bar's classpath in a subprocess, at the expense of the classloader providing weaker isolation than a subprocess. Thus we cannot rely on the working directory inside the bar.Bar code to be in the right place, and instead we need to pass in the Task.dest path explicitly.

Java Virtual Machines are expensive to initialize and with a large memory footprint, therefore if you tend to be doing a lot of classloader or subprocess operations it makes sense to re-use the JVM. You can do this using the CachedFactory helper class, which makes it easy to cache, re-use, and teardown these expensive long-lived components.

The example below is similar to Running a JavaModule in a Classloader above, but in this case the bar.Bar class relies on class-level mutable state in its implementation, and so sharing the same URLClassLoader across different foo* tasks running on different threads is not thread-safe. To resolve this, we make the barWorker contain an instance of mill.api.CachedFactory, which ensures that the classloaders are created when necessary, cached/re-used where possible, and torn down properly when no longer necessary.