Cross Builds
Cross-building refers to taking the same sources and configuration and building it multiple times with minor changes. This could mean taking the same Scala codebase and building it across multiple Scala versions, taking the same application and building twice for dev/release, or taking the same module config and building it across a variety of source folders.
Simple Cross Modules
Mill handles cross-building of all sorts via the Cross[T]
module.
package build
import mill._
object foo extends Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
def suffix = Task { "_" + crossValue }
def bigSuffix = Task { "[[[" + suffix() + "]]]" }
def sources = Task.Sources(millSourcePath)
}
Cross modules defined using the Cross[T]
class allow you to define
multiple copies of the same module, differing only in some input key. This
is very useful for building the same module against different versions of a
language or library, or creating modules to represent folders on the
filesystem.
This example defines three copies of FooModule
: "2.10"
, "2.11"
and
"2.12"
, each of which has their own suffix
task. You can then run
them as shown below. Note that by default, sources
returns foo
for every
cross module, assuming you want to build the same sources for each. This can
be overridden.
> mill show foo[2.10].suffix
"_2.10"
> mill show foo[2.10].bigSuffix
"[[[_2.10]]]"
> mill show foo[2.10].sources
[
".../foo"
]
> mill show foo[2.12].suffix
"_2.12"
> mill show foo[2.12].bigSuffix
"[[[_2.12]]]"
> mill show foo[2.12].sources
[
".../foo"
]
Cross Modules Defaults
package build
import mill._
object foo extends Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
def suffix = Task { "_" + crossValue }
}
object bar extends Cross[FooModule]("2.10", "2.11", "2.12") {
def defaultCrossSegments = Seq("2.12")
}
For convenience, you can omit the selector for the default cross segment.
By default, this is the first cross value specified, but you can override
it by specifying def defaultCrossSegments
> mill show foo[2.10].suffix # explicit cross value given
"_2.10"
> mill show foo[].suffix # no cross value given, defaults to first cross value
"_2.10"
> mill show bar[].suffix # no cross value given, defaults to overriden `defaultCrossSegments`
"_2.12"
Cross Modules Source Paths
If you want to have dedicated millSourcePath
s, you can add the cross
parameters to it as follows:
package build
import mill._
object foo extends Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
def millSourcePath = super.millSourcePath / crossValue
def sources = Task.Sources(millSourcePath)
}
By default, cross modules do not include the cross key as part of the
millSourcePath
for each module. This is the common case, where you are
cross-building the same sources across different input versions. If you want
to use a cross module to build different folders with the same config, you
can do so by overriding millSourcePath
as shown above.
> mill show foo[2.10].sources
[
".../foo/2.10"
]
> mill show foo[2.11].sources
[
".../foo/2.11"
]
> mill show foo[2.12].sources
[
".../foo/2.12"
]
Before Mill 0.11.0-M5,
|
Using Cross Modules from Outside Tasks
You can refer to tasks defined in cross-modules using the foo("2.10")
syntax,
as given below:
package build
import mill._
object foo extends Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
def suffix = Task { "_" + crossValue }
}
def bar = Task { s"hello ${foo("2.10").suffix()}" }
def qux = Task { s"hello ${foo("2.10").suffix()} world ${foo("2.12").suffix()}" }
Here, def bar
uses foo("2.10")
to reference the "2.10"
instance of
FooModule
. You can refer to whatever versions of the cross-module you want,
even using multiple versions of the cross-module in the same task as we do
in def qux
.
> mill show foo[2.10].suffix
"_2.10"
> mill show bar
"hello _2.10"
> mill show qux
"hello _2.10 world _2.12"
Using Cross Modules from other Cross Modules
Tasks in cross-modules can use one another the same way they are used from external tasks:
package build
import mill._
object foo extends mill.Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends Cross.Module[String] {
def suffix = Task { "_" + crossValue }
}
object bar extends mill.Cross[BarModule]("2.10", "2.11", "2.12")
trait BarModule extends Cross.Module[String] {
def bigSuffix = Task { "[[[" + foo(crossValue).suffix() + "]]]" }
}
Rather than passing in a literal "2.10"
to the foo
cross module, we pass
in the crossValue
property that is available within every Cross.Module
.
This ensures that each version of bar
depends on the corresponding version
of foo
: bar("2.10")
depends on foo("2.10")
, bar("2.11")
depends on
foo("2.11")
, etc.
> mill showNamed foo[__].suffix
{
"foo[2.10].suffix": "_2.10",
"foo[2.11].suffix": "_2.11",
"foo[2.12].suffix": "_2.12"
}
> mill showNamed bar[__].bigSuffix
{
"bar[2.10].bigSuffix": "[[[_2.10]]]",
"bar[2.11].bigSuffix": "[[[_2.11]]]",
"bar[2.12].bigSuffix": "[[[_2.12]]]"
}
Multiple Cross Axes
You can have a cross-module with multiple inputs using the Cross.Module2
trait:
package build
import mill._
val crossMatrix = for {
crossVersion <- Seq("2.10", "2.11", "2.12")
platform <- Seq("jvm", "js", "native")
if !(platform == "native" && crossVersion != "2.12")
} yield (crossVersion, platform)
object foo extends mill.Cross[FooModule](crossMatrix)
trait FooModule extends Cross.Module2[String, String] {
val (crossVersion, platform) = (crossValue, crossValue2)
def suffix = Task { "_" + crossVersion + "_" + platform }
}
def bar = Task { s"hello ${foo("2.10", "jvm").suffix()}" }
This example shows off using a for-loop to generate a list of
cross-key-tuples, as a Seq[(String, String)]
that we then pass it into the
Cross
constructor. These can be referenced from the command line as shown
below, or referenced in other parts of your build.mill
as shown in def bar
above.
In this example we assigned crossValue
and crossValue2
to the names
crossVersion
and platform
for readability.
> mill show foo[2.10,jvm].suffix
"_2.10_jvm"
> mill showNamed foo[__].suffix
{
"foo[2.10,jvm].suffix": "_2.10_jvm",
"foo[2.10,js].suffix": "_2.10_js",
"foo[2.11,jvm].suffix": "_2.11_jvm",
"foo[2.11,js].suffix": "_2.11_js",
"foo[2.12,jvm].suffix": "_2.12_jvm",
"foo[2.12,js].suffix": "_2.12_js",
"foo[2.12,native].suffix": "_2.12_native"
}
Extending Cross Modules
You can also take an existing cross module and extend it with additional cross axes as shown:
package build
import mill._
object foo extends Cross[FooModule]("a", "b")
trait FooModule extends Cross.Module[String] {
def param1 = Task { "Param Value: " + crossValue }
}
object foo2 extends Cross[FooModule2](("a", 1), ("b", 2))
trait FooModule2 extends Cross.Module2[String, Int] {
def param1 = Task { "Param Value: " + crossValue }
def param2 = Task { "Param Value: " + crossValue2 }
}
object foo3 extends Cross[FooModule3](("a", 1, true), ("b", 2, false))
trait FooModule3 extends FooModule2 with Cross.Module3[String, Int, Boolean] {
def param3 = Task { "Param Value: " + crossValue3 }
}
Starting from an existing cross module with Cross.Module{N-1}
,
you can extend Cross.ModuleN
to add a new axis to it.
Multi-axis cross modules take their input as tuples, and each element of the
tuple beyond the first is bound to the crossValueN
property defined by the
corresponding Cross.ArgN
trait. Providing tuples of the wrong arity to the
Cross[]
constructor is a compile error.
The Cross module’s axes can take any type T
with Cross.ToSegments[T]
defined. There are default implementations for strings, chars, numbers,
booleans, and lists; the example above demonstrates cross axes of type
String
, Int
, and Boolean
. You can define additional ToPathSegments
for your own user-defined types that you wish to use in a Cross module
> mill show foo[a].param1
"Param Value: a"
> mill show foo[b].param1
"Param Value: b"
> mill show foo2[a,1].param1
"Param Value: a"
> mill show foo2[b,2].param2
"Param Value: 2"
> mill show foo3[b,2,false].param3
"Param Value: false"
> sed -i.bak 's/, true//g' build.mill
> sed -i.bak 's/, false//g' build.mill
> mill show foo3[b,2,false].param3
error: ...object foo3 extends Cross[FooModule3](("a", 1), ("b", 2))
error: ... ^
error: ...value _3 is not a member of (String, Int)
Inner Cross Modules
package build
import mill._
trait MyModule extends Module{
def crossValue: String
def name: T[String]
def param = Task { name() + " Param Value: " + crossValue }
}
object foo extends Cross[FooModule]("a", "b")
trait FooModule extends Cross.Module[String] {
object bar extends MyModule with CrossValue{
def name = "Bar"
}
object qux extends MyModule with CrossValue{
def name = "Qux"
}
}
def baz = Task { s"hello ${foo("a").bar.param()}" }
You can use the CrossValue
trait within any Cross.Module
to
propagate the crossValue
defined by an enclosing Cross.Module
to some
nested module. In this case, we use it to bind crossValue
so it can be
used in def param
.
This lets you reduce verbosity by defining the Cross
once for a group of modules rather than once for every single module in
that group. In the example above, we define the cross module once for
object foo extends Cross
, and then the nested modules bar
and qux
get automatically duplicated once for each crossValue = "a"
and crossValue = "b"
> mill show foo[a].bar.param
"Bar Param Value: a"
> mill show foo[b].qux.param
"Qux Param Value: b"
> mill show baz
"hello Bar Param Value: a"
Cross Resolvers
package build
import mill._
trait MyModule extends Cross.Module[String] {
implicit object resolver extends mill.define.Cross.Resolver[MyModule] {
def resolve[V <: MyModule](c: Cross[V]): V = c.valuesToModules(List(crossValue))
}
}
object foo extends mill.Cross[FooModule]("2.10", "2.11", "2.12")
trait FooModule extends MyModule {
def suffix = Task { "_" + crossValue }
}
object bar extends mill.Cross[BarModule]("2.10", "2.11", "2.12")
trait BarModule extends MyModule {
def bigSuffix = Task { "[[[" + foo().suffix() + "]]]" }
}
You can define an implicit mill.define.Cross.Resolver
within your
cross-modules, which would let you use a shorthand foo()
syntax when
referring to other cross-modules with an identical set of cross values.
While the example resolver
simply looks up the task Cross
value for
the cross-module instance with the same crossVersion
, you can make the
resolver arbitrarily complex. E.g. the resolver
for
mill.scalalib.CrossScalaModule
looks for a cross-module instance whose
scalaVersion
is binary compatible (e.g. 2.10.5 is compatible with 2.10.3)
with the current cross-module.
> mill show bar[2.10].bigSuffix
...
"[[[_2.10]]]"
Please be aware that some shells like
|
The suffix
tasks will have the corresponding output paths for their
metadata and files:
out/
├── foo/
│ ├── 2.10/
│ │ ├── bigSuffix.json
│ │ └── suffix.json
│ ├── 2.11/
│ │ ├── bigSuffix.json
│ │ └── suffix.json
│ └── 2.12/
│ ├── bigSuffix.json
│ └── suffix.json
Dynamic Cross Modules
package build
import mill._, scalalib._
val moduleNames = interp.watchValue(os.list(millSourcePath / "modules").map(_.last))
object modules extends Cross[FolderModule](moduleNames)
trait FolderModule extends ScalaModule with Cross.Module[String]{
def millSourcePath = super.millSourcePath / crossValue
def scalaVersion = "2.13.8"
}
It is sometimes necessary for the instances of a cross-module to vary based on some kind of runtime information: perhaps the list of modules is stored in some config file, or is inferred based on the folders present on the filesystem.
In those cases, you can write arbitrary code to populate the cross-module
cases, as long as you wrap the value in a interp.watchValue
. This ensures
that Mill is aware that the module structure depends on that value, and will
re-compute the value and re-create the module structure if the value changes.
> mill resolve modules[_]
modules[bar]
modules[foo]
modules[qux]
> mill modules[bar].run
Hello World Bar
> mill modules[new].run
error: Cannot resolve modules[new]...
> cp -r modules/bar modules/new
> sed -i.bak 's/Bar/New/g' modules/new/src/Example.scala
> mill resolve modules[_]
modules[bar]
modules[foo]
modules[qux]
modules[new]
> mill modules[new].run
Hello World New
Note that because the inputs to the Cross
constructor affects the number
of cross-modules that are generated, it has to be a raw value e.g.
List[T]
and not a task T[List[T]]
. That also means that the list of
cross-modules cannot depend on the output of any tasks.
Use Case: Static Blog
The following example demonstrates a use case: using cross modules to turn files on disk into blog posts. To begin with, we import $ivy two third-party libraries - Commonmark and Scalatags - to deal with Markdown parsing and HTML generation respectively:
package build
import $ivy.`com.lihaoyi::scalatags:0.12.0`, scalatags.Text.all._
import $ivy.`com.atlassian.commonmark:commonmark:0.13.1`
import org.commonmark.parser.Parser
import org.commonmark.renderer.html.HtmlRenderer
Next, we use os.list
and interp.watchValue
on the post/
folder to
build a Cross[PostModule]
whose entries depend no the markdown files we
find in that folder. Each post has a source
pointing at the markdown file,
and a render
task that parses the file’s markdown and generates a HTML
output file
import mill._
def mdNameToHtml(s: String) = s.toLowerCase.replace(".md", ".html")
def mdNameToTitle(s: String) =
s.split('-').drop(1).mkString(" ").stripSuffix(".md")
val posts = interp.watchValue {
os.list(millSourcePath / "post").map(_.last).sorted
}
object post extends Cross[PostModule](posts)
trait PostModule extends Cross.Module[String]{
def source = Task.Source(millSourcePath / crossValue)
def render = Task {
val doc = Parser.builder().build().parse(os.read(source().path))
val title = mdNameToTitle(crossValue)
val rendered = doctype("html")(
html(
body(
h1(a("Blog", href := "../index.html"), " / ", title),
raw(HtmlRenderer.builder().build().render(doc))
)
)
)
os.write(Task.dest / mdNameToHtml(crossValue), rendered)
PathRef(Task.dest / mdNameToHtml(crossValue))
}
}
The last page we need to generate is the index page, listing out the various
blog posts and providing links so we can navigate into them. To do this, we
need to wrap the posts
value in a Task.Input
, as it can change depending on
what os.list
finds on disk. After that, it’s straightforward to render the
index.html
file we want:
def postsInput = Task.Input{ posts }
def renderIndexEntry(mdName: String) = {
h2(a(mdNameToTitle(mdName), href := ("post/" + mdNameToHtml(mdName))))
}
def index = Task {
val rendered = doctype("html")(
html(body(h1("Blog"), postsInput().map(renderIndexEntry)))
)
os.write(Task.dest / "index.html", rendered)
PathRef(Task.dest / "index.html")
}
Lastly we copy the individual post HTML files and the index.html
file
into a single task’s .dest
folder, and return it:
def dist = Task {
for (post <- Task.traverse(post.crossModules)(_.render)()) {
os.copy(post.path, Task.dest / "post" / post.path.last, createFolders = true)
}
os.copy(index().path, Task.dest / "index.html")
PathRef(Task.dest)
}
Now, you can run mill dist
to generate the blog:
> mill dist
> cat out/dist.dest/index.html # root index page
...
...<a href="post/1-my-first-post.html">My First Post</a>...
...<a href="post/2-my-second-post.html">My Second Post</a>...
...<a href="post/3-my-third-post.html">My Third Post</a>...
> cat out/dist.dest/post/1-my-first-post.html # blog post page
...
...<p>Text contents of My First Post</p>...
This static blog automatically picks up new blog posts you add to the
post/
folder, and when you edit your posts it only re-parses and
re-renders the markdown files that you changed. You can use -w
to watch
the posts folder to automatically re-run the dist
command if a post
changes, or -j
e.g. mill -j 4 dist
to enable parallelism if there are
enough posts that the build is becoming noticeably slow.
You can also build each individual post directly:
> mill show "post[1-My-First-Post.md].render"
".../out/post/1-My-First-Post.md/render.dest/1-my-first-post.html"
> cat out/post/1-My-First-Post.md/render.dest/1-my-first-post.html
...
...<p>Text contents of My First Post</p>...
All caching, incremental re-computation, and parallelism is done using the Mill task graph. For this simple example, the graph is as follows
This example use case is taken from the following blog post, which contains some extensions and fun exercises to further familiarize yourself with Mill