Cross Builds

Cross-building refers to taking the same sources and configuration and building it multiple times with minor changes. This could mean taking the same Scala codebase and building it across multiple Scala versions, taking the same application and building twice for dev/release, or taking the same module config and building it across a variety of source folders.

Simple Cross Modules

Mill handles cross-building of all sorts via the Cross[T] module.

build.mill (download, browse)
package build
import mill._

object foo extends Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
  def suffix = Task { "_" + crossValue }
  def bigSuffix = Task { "[[[" + suffix() + "]]]" }
  def sources = Task.Sources(millSourcePath)
}
G cluster_0 foo[2.10] cluster_2 foo[2.12] cluster_1 foo[2.11] foo[2.12].suffix foo[2.12].suffix foo[2.12].bigSuffix foo[2.12].bigSuffix foo[2.12].suffix->foo[2.12].bigSuffix foo[2.12].sources foo[2.12].sources foo[2.11].suffix foo[2.11].suffix foo[2.11].bigSuffix foo[2.11].bigSuffix foo[2.11].suffix->foo[2.11].bigSuffix foo[2.11].sources foo[2.11].sources foo[2.10].suffix foo[2.10].suffix foo[2.10].bigSuffix foo[2.10].bigSuffix foo[2.10].suffix->foo[2.10].bigSuffix foo[2.10].sources foo[2.10].sources

Cross modules defined using the Cross[T] class allow you to define multiple copies of the same module, differing only in some input key. This is very useful for building the same module against different versions of a language or library, or creating modules to represent folders on the filesystem.

This example defines three copies of FooModule: "2.10", "2.11" and "2.12", each of which has their own suffix task. You can then run them as shown below. Note that by default, sources returns foo for every cross module, assuming you want to build the same sources for each. This can be overridden.

> mill show foo[2.10].suffix
"_2.10"

> mill show foo[2.10].bigSuffix
"[[[_2.10]]]"

> mill show foo[2.10].sources
[
  ".../foo"
]

> mill show foo[2.12].suffix
"_2.12"

> mill show foo[2.12].bigSuffix
"[[[_2.12]]]"

> mill show foo[2.12].sources
[
  ".../foo"
]

Cross Modules Defaults

build.mill (download, browse)
package build
import mill._

object foo extends Cross[FooModule]("2.10", "2.11", "2.12")

trait FooModule extends Cross.Module[String] {
  def suffix = Task { "_" + crossValue }
}

object bar extends Cross[FooModule]("2.10", "2.11", "2.12") {
  def defaultCrossSegments = Seq("2.12")
}

For convenience, you can omit the selector for the default cross segment. By default, this is the first cross value specified, but you can override it by specifying def defaultCrossSegments

> mill show foo[2.10].suffix # explicit cross value given
"_2.10"

> mill show foo[].suffix # no cross value given, defaults to first cross value
"_2.10"

> mill show bar[].suffix # no cross value given, defaults to overriden `defaultCrossSegments`
"_2.12"

Cross Modules Source Paths

If you want to have dedicated millSourcePaths, you can add the cross parameters to it as follows:

build.mill (download, browse)
package build
import mill._

object foo extends Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
  def millSourcePath = super.millSourcePath / crossValue
  def sources = Task.Sources(millSourcePath)
}

By default, cross modules do not include the cross key as part of the millSourcePath for each module. This is the common case, where you are cross-building the same sources across different input versions. If you want to use a cross module to build different folders with the same config, you can do so by overriding millSourcePath as shown above.

> mill show foo[2.10].sources
[
  ".../foo/2.10"
]

> mill show foo[2.11].sources
[
  ".../foo/2.11"
]

> mill show foo[2.12].sources
[
  ".../foo/2.12"
]

Before Mill 0.11.0-M5, Cross modules which were not also CrossScalaModules, always added the cross parameters to the millSourcePath. This often led to setups like this:

def millSourcePath = super.millSourcePath / os.up

Using Cross Modules from Outside Tasks

You can refer to tasks defined in cross-modules using the foo("2.10") syntax, as given below:

build.mill (download, browse)
package build
import mill._

object foo extends Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
  def suffix = Task { "_" + crossValue }
}

def bar = Task { s"hello ${foo("2.10").suffix()}" }

def qux = Task { s"hello ${foo("2.10").suffix()} world ${foo("2.12").suffix()}" }
G cluster_0 foo[2.10] cluster_1 foo[2.11] cluster_2 foo[2.12] foo[2.12].suffix foo[2.12].suffix qux qux foo[2.12].suffix->qux foo[2.11].suffix foo[2.11].suffix foo[2.10].suffix foo[2.10].suffix foo[2.10].suffix->qux bar bar foo[2.10].suffix->bar

Here, def bar uses foo("2.10") to reference the "2.10" instance of FooModule. You can refer to whatever versions of the cross-module you want, even using multiple versions of the cross-module in the same task as we do in def qux.

> mill show foo[2.10].suffix
"_2.10"

> mill show bar
"hello _2.10"

> mill show qux
"hello _2.10 world _2.12"

Using Cross Modules from other Cross Modules

Tasks in cross-modules can use one another the same way they are used from external tasks:

build.mill (download, browse)
package build
import mill._

object foo extends mill.Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
  def suffix = Task { "_" + crossValue }
}

object bar extends mill.Cross[BarModule]("2.10", "2.11", "2.12")
trait BarModule extends Cross.Module[String] {
  def bigSuffix = Task { "[[[" + foo(crossValue).suffix() + "]]]" }
}
G cluster_2 foo[2.12] cluster_4 bar[2.11] cluster_5 bar[2.12] cluster_1 foo[2.11] cluster_3 bar[2.10] cluster_0 foo[2.10] foo[2.12].suffix foo[2.12].suffix bar[2.12].bigSuffix bar[2.12].bigSuffix foo[2.12].suffix->bar[2.12].bigSuffix foo[2.11].suffix foo[2.11].suffix bar[2.11].bigSuffix bar[2.11].bigSuffix foo[2.11].suffix->bar[2.11].bigSuffix foo[2.10].suffix foo[2.10].suffix bar[2.10].bigSuffix bar[2.10].bigSuffix foo[2.10].suffix->bar[2.10].bigSuffix

Rather than passing in a literal "2.10" to the foo cross module, we pass in the crossValue property that is available within every Cross.Module. This ensures that each version of bar depends on the corresponding version of foo: bar("2.10") depends on foo("2.10"), bar("2.11") depends on foo("2.11"), etc.

> mill showNamed foo[__].suffix
{
  "foo[2.10].suffix": "_2.10",
  "foo[2.11].suffix": "_2.11",
  "foo[2.12].suffix": "_2.12"
}

> mill showNamed bar[__].bigSuffix
{
  "bar[2.10].bigSuffix": "[[[_2.10]]]",
  "bar[2.11].bigSuffix": "[[[_2.11]]]",
  "bar[2.12].bigSuffix": "[[[_2.12]]]"
}

Multiple Cross Axes

You can have a cross-module with multiple inputs using the Cross.Module2 trait:

build.mill (download, browse)
package build
import mill._

val crossMatrix = for {
  crossVersion <- Seq("2.10", "2.11", "2.12")
  platform <- Seq("jvm", "js", "native")
  if !(platform == "native" && crossVersion != "2.12")
} yield (crossVersion, platform)

object foo extends mill.Cross[FooModule](crossMatrix)
trait FooModule extends Cross.Module2[String, String] {
  val (crossVersion, platform) = (crossValue, crossValue2)
  def suffix = Task { "_" + crossVersion + "_" + platform }
}

def bar = Task { s"hello ${foo("2.10", "jvm").suffix()}" }
G cluster_5 foo[2.12,js] cluster_0 foo[2.10,jvm] cluster_1 foo[2.11,jvm] cluster_6 foo[2.12,native] cluster_3 foo[2.10,js] cluster_2 foo[2.12,jvm] foo[2.12,native].suffix foo[2.12,native].suffix foo[2.10,js].suffix foo[2.10,js].suffix foo[2.11,js].suffix foo[2.11,js].suffix foo[2.12,js].suffix foo[2.12,js].suffix foo[2.10,jvm].suffix foo[2.10,jvm].suffix bar bar foo[2.10,jvm].suffix->bar foo[2.11,jvm].suffix foo[2.11,jvm].suffix foo[2.12,jvm].suffix foo[2.12,jvm].suffix

This example shows off using a for-loop to generate a list of cross-key-tuples, as a Seq[(String, String)] that we then pass it into the Cross constructor. These can be referenced from the command line as shown below, or referenced in other parts of your build.mill as shown in def bar above.

In this example we assigned crossValue and crossValue2 to the names crossVersion and platform for readability.

> mill show foo[2.10,jvm].suffix
"_2.10_jvm"

> mill showNamed foo[__].suffix
{
  "foo[2.10,jvm].suffix": "_2.10_jvm",
  "foo[2.10,js].suffix": "_2.10_js",
  "foo[2.11,jvm].suffix": "_2.11_jvm",
  "foo[2.11,js].suffix": "_2.11_js",
  "foo[2.12,jvm].suffix": "_2.12_jvm",
  "foo[2.12,js].suffix": "_2.12_js",
  "foo[2.12,native].suffix": "_2.12_native"
}

Extending Cross Modules

You can also take an existing cross module and extend it with additional cross axes as shown:

build.mill (download, browse)
package build
import mill._

object foo extends Cross[FooModule]("a", "b")
trait FooModule extends Cross.Module[String] {
  def param1 = Task { "Param Value: " + crossValue }
}

object foo2 extends Cross[FooModule2](("a", 1), ("b", 2))
trait FooModule2 extends Cross.Module2[String, Int] {
  def param1 = Task { "Param Value: " + crossValue }
  def param2 = Task { "Param Value: " + crossValue2 }
}

object foo3 extends Cross[FooModule3](("a", 1, true), ("b", 2, false))
trait FooModule3 extends FooModule2 with Cross.Module3[String, Int, Boolean] {
  def param3 = Task { "Param Value: " + crossValue3 }
}

Starting from an existing cross module with Cross.Module{N-1}, you can extend Cross.ModuleN to add a new axis to it.

Multi-axis cross modules take their input as tuples, and each element of the tuple beyond the first is bound to the crossValueN property defined by the corresponding Cross.ArgN trait. Providing tuples of the wrong arity to the Cross[] constructor is a compile error.

The Cross module’s axes can take any type T with Cross.ToSegments[T] defined. There are default implementations for strings, chars, numbers, booleans, and lists; the example above demonstrates cross axes of type String, Int, and Boolean. You can define additional ToPathSegments for your own user-defined types that you wish to use in a Cross module

> mill show foo[a].param1
"Param Value: a"

> mill show foo[b].param1
"Param Value: b"

> mill show foo2[a,1].param1
"Param Value: a"

> mill show foo2[b,2].param2
"Param Value: 2"

> mill show foo3[b,2,false].param3
"Param Value: false"

> sed -i.bak 's/, true//g' build.mill

> sed -i.bak 's/, false//g' build.mill

> mill show foo3[b,2,false].param3
error: ...object foo3 extends Cross[FooModule3](("a", 1), ("b", 2))
error: ...                                      ^
error: ...value _3 is not a member of (String, Int)

Inner Cross Modules

build.mill (download, browse)
package build
import mill._

trait MyModule extends Module {
  def crossValue: String
  def name: T[String]
  def param = Task { name() + " Param Value: " + crossValue }
}

object foo extends Cross[FooModule]("a", "b")
trait FooModule extends Cross.Module[String] {
  object bar extends MyModule with CrossValue {
    def name = "Bar"
  }
  object qux extends MyModule with CrossValue {
    def name = "Qux"
  }
}

def baz = Task { s"hello ${foo("a").bar.param()}" }
G root-module root-module foo foo root-module->foo foo[a] foo[a] foo->foo[a] foo[b] foo[b] foo->foo[b] foo[a].bar foo[a].bar foo[a]->foo[a].bar foo[a].qux foo[a].qux foo[a]->foo[a].qux foo[b].bar foo[b].bar foo[b]->foo[b].bar foo[b].qux foo[b].qux foo[b]->foo[b].qux foo[a].bar.name foo[a].bar.name foo[a].bar->foo[a].bar.name foo[a].bar.param foo[a].bar.param foo[a].bar->foo[a].bar.param ... ... foo[a].bar.name->foo[a].bar.param

You can use the CrossValue trait within any Cross.Module to propagate the crossValue defined by an enclosing Cross.Module to some nested module. In this case, we use it to bind crossValue so it can be used in def param.

This lets you reduce verbosity by defining the Cross once for a group of modules rather than once for every single module in that group. In the example above, we define the cross module once for object foo extends Cross, and then the nested modules bar and qux get automatically duplicated once for each crossValue = "a" and crossValue = "b"

> mill show foo[a].bar.param
"Bar Param Value: a"

> mill show foo[b].qux.param
"Qux Param Value: b"

> mill show baz
"hello Bar Param Value: a"

Cross Resolvers

build.mill (download, browse)
package build
import mill._

trait MyModule extends Cross.Module[String] {
  implicit object resolver extends mill.define.Cross.Resolver[MyModule] {
    def resolve[V <: MyModule](c: Cross[V]): V = c.valuesToModules(List(crossValue))
  }
}

object foo extends mill.Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends MyModule {
  def suffix = Task { "_" + crossValue }
}

object bar extends mill.Cross[BarModule]("2.10", "2.11", "2.12")
trait BarModule extends MyModule {
  def bigSuffix = Task { "[[[" + foo().suffix() + "]]]" }
}

You can define an implicit mill.define.Cross.Resolver within your cross-modules, which would let you use a shorthand foo() syntax when referring to other cross-modules with an identical set of cross values.

While the example resolver simply looks up the task Cross value for the cross-module instance with the same crossVersion, you can make the resolver arbitrarily complex. E.g. the resolver for mill.scalalib.CrossScalaModule looks for a cross-module instance whose scalaVersion is binary compatible (e.g. 2.10.5 is compatible with 2.10.3) with the current cross-module.

> mill show bar[2.10].bigSuffix
...
"[[[_2.10]]]"

Please be aware that some shells like zsh interpret square brackets differently, so quoting or masking might be needed.

mill show foo\[2.10\].suffix
mill show 'foo[2.10].suffix'
mill show "foo[2.10].suffix"

The suffix tasks will have the corresponding output paths for their metadata and files:

out/
├── foo/
│     ├── 2.10/
│     │     ├── bigSuffix.json
│     │     └── suffix.json
│     ├── 2.11/
│     │     ├── bigSuffix.json
│     │     └── suffix.json
│     └── 2.12/
│         ├── bigSuffix.json
│         └── suffix.json

Dynamic Cross Modules

build.mill (download, browse)
package build
import mill._, scalalib._

val moduleNames = interp.watchValue(os.list(millSourcePath / "modules").map(_.last))

object modules extends Cross[FolderModule](moduleNames)
trait FolderModule extends ScalaModule with Cross.Module[String] {
  def millSourcePath = super.millSourcePath / crossValue
  def scalaVersion = "2.13.8"
}

It is sometimes necessary for the instances of a cross-module to vary based on some kind of runtime information: perhaps the list of modules is stored in some config file, or is inferred based on the folders present on the filesystem.

In those cases, you can write arbitrary code to populate the cross-module cases, as long as you wrap the value in a interp.watchValue. This ensures that Mill is aware that the module structure depends on that value, and will re-compute the value and re-create the module structure if the value changes.

> mill resolve modules[_]
modules[bar]
modules[foo]
modules[qux]

> mill modules[bar].run
Hello World Bar

> mill modules[new].run
error: Cannot resolve modules[new]...

> cp -r modules/bar modules/new

> sed -i.bak 's/Bar/New/g' modules/new/src/Example.scala

> mill resolve modules[_]
modules[bar]
modules[foo]
modules[qux]
modules[new]

> mill modules[new].run
Hello World New

Note that because the inputs to the Cross constructor affects the number of cross-modules that are generated, it has to be a raw value e.g. List[T] and not a task T[List[T]]. That also means that the list of cross-modules cannot depend on the output of any tasks.

Use Case: Static Blog

The following example demonstrates a use case: using cross modules to turn files on disk into blog posts. To begin with, we import $ivy two third-party libraries - Commonmark and Scalatags - to deal with Markdown parsing and HTML generation respectively:

build.mill (download, browse)
package build
import $ivy.`com.lihaoyi::scalatags:0.12.0`, scalatags.Text.all._
import $ivy.`com.atlassian.commonmark:commonmark:0.13.1`
import org.commonmark.parser.Parser
import org.commonmark.renderer.html.HtmlRenderer

Next, we use os.list and interp.watchValue on the post/ folder to build a Cross[PostModule] whose entries depend no the markdown files we find in that folder. Each post has a source pointing at the markdown file, and a render task that parses the file’s markdown and generates a HTML output file

import mill._

def mdNameToHtml(s: String) = s.toLowerCase.replace(".md", ".html")
def mdNameToTitle(s: String) =
  s.split('-').drop(1).mkString(" ").stripSuffix(".md")

val posts = interp.watchValue {
  os.list(millSourcePath / "post").map(_.last).sorted
}

object post extends Cross[PostModule](posts)
trait PostModule extends Cross.Module[String] {
  def source = Task.Source(millSourcePath / crossValue)
  def render = Task {
    val doc = Parser.builder().build().parse(os.read(source().path))
    val title = mdNameToTitle(crossValue)
    val rendered = doctype("html")(
      html(
        body(
          h1(a("Blog", href := "../index.html"), " / ", title),
          raw(HtmlRenderer.builder().build().render(doc))
        )
      )
    )

    os.write(Task.dest / mdNameToHtml(crossValue), rendered)
    PathRef(Task.dest / mdNameToHtml(crossValue))
  }
}

The last page we need to generate is the index page, listing out the various blog posts and providing links so we can navigate into them. To do this, we need to wrap the posts value in a Task.Input, as it can change depending on what os.list finds on disk. After that, it’s straightforward to render the index.html file we want:

def postsInput = Task.Input { posts }

def renderIndexEntry(mdName: String) = {
  h2(a(mdNameToTitle(mdName), href := ("post/" + mdNameToHtml(mdName))))
}

def index = Task {
  val rendered = doctype("html")(
    html(body(h1("Blog"), postsInput().map(renderIndexEntry)))
  )
  os.write(Task.dest / "index.html", rendered)
  PathRef(Task.dest / "index.html")
}

Lastly we copy the individual post HTML files and the index.html file into a single task’s .dest folder, and return it:

def dist = Task {
  for (post <- Task.traverse(post.crossModules)(_.render)()) {
    os.copy(post.path, Task.dest / "post" / post.path.last, createFolders = true)
  }
  os.copy(index().path, Task.dest / "index.html")
  PathRef(Task.dest)
}

Now, you can run mill dist to generate the blog:

> mill dist

> cat out/dist.dest/index.html                    # root index page
...
...<a href="post/1-my-first-post.html">My First Post</a>...
...<a href="post/2-my-second-post.html">My Second Post</a>...
...<a href="post/3-my-third-post.html">My Third Post</a>...

> cat out/dist.dest/post/1-my-first-post.html     # blog post page
...
...<p>Text contents of My First Post</p>...
BlogIndex.png
BlogPost.png

This static blog automatically picks up new blog posts you add to the post/ folder, and when you edit your posts it only re-parses and re-renders the markdown files that you changed. You can use -w to watch the posts folder to automatically re-run the dist command if a post changes, or -j e.g. mill -j 4 dist to enable parallelism if there are enough posts that the build is becoming noticeably slow.

You can also build each individual post directly:

> mill show "post[1-My-First-Post.md].render"
".../out/post/1-My-First-Post.md/render.dest/1-my-first-post.html"

> cat out/post/1-My-First-Post.md/render.dest/1-my-first-post.html
...
...<p>Text contents of My First Post</p>...

All caching, incremental re-computation, and parallelism is done using the Mill task graph. For this simple example, the graph is as follows

G 1 - Foo.md 1 - Foo.md post[1]\nrender post[1] render 1 - Foo.md->post[1]\nrender dist dist post[1]\nrender->dist 2 - Foo.md 2 - Foo.md post[2]\nrender post[2] render 2 - Foo.md->post[2]\nrender post[2]\nrender->dist 3 - Foo.md 3 - Foo.md post[3]\nrender post[3] render 3 - Foo.md->post[3]\nrender post[3]\nrender->dist index index index->dist

This example use case is taken from the following blog post, which contains some extensions and fun exercises to further familiarize yourself with Mill