Mill Sandboxing
Task Sandboxing
In order to help manage your build, Mill performs some rudimentary filesystem
sandboxing to keep different tasks and modules from interfering with each other.
This tries to ensure your tasks only read and write from their designated .dest/
folders, which are unique to each task and thus guaranteed not to collide with
the filesystem operations of other tasks that may be occurring in parallel.
Task.dest
The standard way of working with a task’s .dest/
folder is through the Task.dest
property. This is available within any task, and gives you access to the
out/<module-names>/<task-name>.dest/
folder to use. The .dest/
folder for
each task is lazily initialized when Task.dest
is referenced and used:
package build
import mill._
object foo extends Module{
def tDestTask = Task { println(Task.dest.toString) }
}
> ./mill foo.tDestTask
.../out/foo/tDestTask.dest
Task os.pwd
redirection
Mill also redirects the os.pwd
property from OS-Lib,
such that that also points towards a running task’s own .dest/
folder
def osPwdTask = Task { println(os.pwd.toString) }
> ./mill osPwdTask
.../out/osPwdTask.dest
The redirection of os.pwd
applies to os.proc
, os.call
, and os.spawn
methods
as well. In the example below, we can see the python3
subprocess we spawn prints
its os.getcwd()
, which is our osProcTask.dest/
sandbox folder:
def osProcTask = Task {
println(os.call(("python3", "-c", "import os; print(os.getcwd())"), cwd = Task.dest).out.trim())
}
> ./mill osProcTask
.../out/osProcTask.dest
Non-task os.pwd
redirection
Lastly, there is the possibily of calling os.pwd
outside of a task. When outside of
a task there is no .dest/
folder associated, so instead Mill will redirect os.pwd
towards an empty sandbox/
folder in out/mill-server/…
:
val externalPwd = os.pwd
def externalPwdTask = Task { println(externalPwd.toString) }
> ./mill externalPwdTask
.../out/mill-server/.../sandbox
Limitations of Mill’s Sandboxing
Mill’s approach to filesystem sandboxing is designed to avoid accidental interference
between different Mill tasks. It is not designed to block intentional misbehavior, and
tasks are always able to traverse the filesystem and do whatever they want. Furthermore,
Mill’s redirection of os.pwd
does not apply to java.io
or java.nio
APIs, which are
outside of Mill’s control.
However, by setting os.pwd
to safe sandbox folders, we hope to minimize the cases where
someone accidentally causes issues with their build by doing the wrong thing.
Test Sandboxing
Mill also creates sandbox folders for test suites to run in. Consider the
following build with two modules foo
and bar
, and their test suites
foo.test
and bar.test
:
package build
import mill._, javalib._
trait MyModule extends JavaModule{
object test extends JavaTests with TestModule.Junit4
}
object foo extends MyModule{
def moduleDeps = Seq(bar)
}
object bar extends MyModule
For the sake of the example, both test modules contain tests that exercise the
logic in their corresponding non-test module, but also do some basic filesystem
operations at the same time, writing out a generated.html
file and then reading it:
package foo;
public class Foo {
public static String generateHtml(String text) {
return "<h1>" + text + "</h1>";
}
}
package foo;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import java.nio.file.*;
public class FooTests {
@Test
public void simple() throws Exception {
String result = Foo.generateHtml("hello");
Path path = Paths.get("generated.html");
Files.write(path, result.getBytes());
assertEquals("<h1>hello</h1>", Files.readString(path));
}
}
package bar;
public class Bar {
public static String generateHtml(String text) {
return "<p>" + text + "</p>";
}
}
package bar;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import java.nio.file.*;
public class BarTests {
@Test
public void simple() throws Exception {
String result = Bar.generateHtml("world");
Path path = Paths.get("generated.html");
Files.write(path, result.getBytes());
assertEquals("<p>world</p>", Files.readString(path));
}
}
Both test suites can be run via
> ./mill __.test
Without sandboxing, due to the tests running in parallel, there is a race condition:
it’s possible that FooTests
may write the file, BarTests
write over it, before
FooTests
reads the output from BarTests
. That would cause non-deterministic
flaky failures in your test suite that can be very difficult to debug and resolve.
With Mill’s test sandboxing, each test runs in a separate folder: the .dest
folder
of the respective task and module. For example:
-
foo.test
runs inout/foo/test/test.dest/
-
bar.test
runs inout/bar/test/test.dest/
As a result, each test’s generated.html
file is written to its own dedicated
working directory, without colliding with each other on disk:
> find . | grep generated.html
.../out/foo/test/test.dest/sandbox/generated.html
.../out/bar/test/test.dest/sandbox/generated.html
> cat out/foo/test/test.dest/sandbox/generated.html
<h1>hello</h1>
> cat out/bar/test/test.dest/sandbox/generated.html
<p>world</p>
As each test suite runs in a different working directory by default, naive usage reading and writing to the filesystem does not cause tests to interefere with one another, which helps keep tests stable and deterministic even when run in parallel
Like Mill’s Task sandboxing, Mill’s Test sandboxing does not guard against
intentional misbehavior: tests can still walk the filesystem from the
sandbox folder via ..
or from the root folder /
or home folder ~/
.
Nevertheless, it should add some simple guardrails to prevent many common
causes of inter-test interference, letting your test suite run in parallel both
quickly and reliably
Breaking Out Of Sandbox Folders
Mill’s sandboxing approach is best effort: while it tries to guide you into using
isolated sandbox folders, Mill cannot guarantee it, and in fact provides the
Task.workspace
property and MILL_WORKSPACE_ROOT
environment variable to reference the
project root folder for scenarios where you may need it. This can be useful for a variety
of reasons:
-
Migrating legacy applications that assume access to the workspace root
-
Scenarios where writing the the original source repository is necessary: code auto-formatters, auto-fixers, auto-updaters. etc.
Task.workspace
can be used in tasks:
package build
import mill._, javalib._
def tWorkspaceTask = Task { println(Task.workspace) }
> ./mill tWorkspaceTask
Whereas MILL_WORKSPACE_ROOT
as well as in tests, which can access the
workspace root via the MILL_WORKSPACE_ROOT
environment variable
object foo extends JavaModule{
object test extends JavaTests with TestModule.Junit4
}
package foo;
public class Foo {
public static String generateHtml(String text) {
return "<h1>" + text + "</h1>";
}
}
package foo;
import java.util.stream.Collectors;
import java.nio.file.*;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
public class FooTests {
@Test
public void simple() throws Exception {
String workspaceRoot = System.getenv("MILL_WORKSPACE_ROOT");
for(Path subpath: Files.list(Paths.get(workspaceRoot)).collect(Collectors.toList())){
String result = Foo.generateHtml(subpath.getFileName().toString());
Path tmppath = Paths.get(subpath.getFileName() + ".html");
Files.write(tmppath, result.getBytes());
assertEquals(
"<h1>" + subpath.getFileName() + "</h1>",
Files.readString(tmppath)
);
}
}
}
> ./mill __.test
> find . | grep .html
...
.../out/foo/test/test.dest/sandbox/foo.html
> cat out/foo/test/test.dest/sandbox/foo.html
<h1>foo</h1>
Limitations
Mill’s approach to filesystem sandboxing is designed to avoid accidental interference
between different Mill tasks. It is not designed to block intentional misbehavior, and
tasks are always able to traverse the filesystem and do whatever they want. Furthermore,
Mill’s redirection of os.pwd
does not apply to java.io
or java.nio
APIs, which are
outside of Mill’s control.
However, by setting os.pwd
to safe sandbox folders, we hope to minimize the cases where
someone accidentally causes issues with their build by doing the wrong thing.