Support for New Languages
This section walks through the process of adding support for a new programming
language to Mill. We will be adding a small trait TypeScriptModule
with the
ability to resolve dependencies, typecheck local code, and optimize a final
bundle.
The TypeScript integration here is not intended for production usage, but is instead meant for illustration purposes of the techniques typically used in implementing language toolchains.
Basic TypeScript Pipeline
This example demonstrates basic integration of Typescript compilation into a Mill build to compile Node.js apps. Mill does not come bundled with a Typescript integration, so here we begin setting one up from first principles using the NPM command line tool and package repository
Installing TypeScript
First, we need to use the npm
CLI tool to install typescript and the @types/node
library necessary for accessing Node.js APIs:
package build
import mill._
def npmInstall = Task{
os.call(("npm", "install", "--save-dev", "typescript@5.6.3", "@types/node@22.7.8"))
PathRef(Task.dest)
}
The npmInstall
task runs npm install
to install TypeScript locally, following
the TypeScript installation instructions.
The os.call
above by default runs inside the npmInstall
task’s unique Task.dest
output directory due to task sandboxing. Note that we
use an explicit version on each of the modules to ensure the Task
is reproducible.
We then return a PathRef
to the Task.dest
so downstream tasks can make use of it.
Note that as written, the npmInstall
task will never invalidate unless you change its
code. This is what we should expect, since npmInstall
has no upstream tasks it depends
on and the versions of typescript
and @types/node
are fully specified in the task.
This assumes that the npm
package repository always returns the same artifacts for
the given name and version, which is a reasonable assumption for most package repositories.
Defining our sources
Next, we define the sources
of our Typescript build using a
source task. Here sources
refers to the
source folder, and the subsequent allSources
walks that folder and picks up all
the individual typescript files within. This is a common pattern to give flexibility,
e.g. someone can later easily override allSources
to add additional filtering
on exactly which files within the source root they wish to pick up.
def sources = Task.Source(millSourcePath / "src")
def allSources = Task{
os.walk(sources().path).filter(_.ext == "ts").map(PathRef(_))
}
Compilation
Next, we define our compile
task. This is again a relatively straightforward subprocess
call invoking the typescript/bin/tsc
executable within the node_modules
folder from the
upstream npmInstall
task, passing it the sources, --outDir
, --types
, and --typeRoots
Again we return a PathRef
to the Task.dest
folder we output the compiled JS files to
def compile = Task{
os.call(
(
npmInstall().path / "node_modules/typescript/bin/tsc",
allSources().map(_.path),
"--outDir", Task.dest,
"--typeRoots", npmInstall().path / "node_modules/@types"
)
)
PathRef(Task.dest)
}
At this point, we have a minimal working build, with a build graph that looks like this:
Given an input file below, we can run
mill compile
and demonstrate it is installing typescript locally and using it to compile
the .ts
files in out src/
folder:
interface User {
firstName: string
lastName: string
role: string
}
const user: User = {
firstName: process.argv[2],
lastName: process.argv[3],
role: "Professor",
}
console.log("Hello " + user.name + " " + user.lastName)
> mill compile
error: .../src/hello.ts(...): error ... Property 'name' does not exist on type...
> sed -i.bak 's/user.name/user.firstName/g' src/hello.ts
> mill compile
> cat out/compile.dest/hello.js # Output is stripped of types and converted to javascript
var user = {
firstName: process.argv[2],
lastName: process.argv[3],
role: "Professor",
};
console.log("Hello " + user.firstName + " " + user.lastName);
Running
The last step here is to allow the ability to run our compiled JavaScript file.
To do this, we need a mainFileName
task to tell Mill which file should be used
as the program entrypoint, and a run
command taking arguments that get used to
call node
along with the main Javascript file:
def mainFileName = Task{ "hello.js" }
def run(args: mill.define.Args) = Task.Command{
os.call(
("node", compile().path / mainFileName(), args.value),
stdout = os.Inherit
)
}
Note that we use stdout = os.Inherit
since we want to display any output to
the user, rather than capturing it for use in our command.
> mill run James Bond
Hello James Bond
So that’s a minimal example of implementing a single TypeScript to JavaScript build
pipeline locally. Next, we will look at turning it into a TypeScriptModule
that
can be re-used
Re-usable TypeScriptModule
In this example, we will explore how to take the one-off typescript build pipeline
we wrote above, and turn it into a re-usable TypeScriptModule
.
To do this, we take all the code we wrote earlier and surround it with
trait TypeScriptModule extends Module
wrapper:
package build
import mill._
trait TypeScriptModule extends Module{
def npmInstall = Task{
os.call(("npm", "install", "--save-dev", "typescript@5.6.3", "@types/node@22.7.8"))
PathRef(Task.dest)
}
def sources = Task.Source(millSourcePath / "src")
def allSources = Task{
os.walk(sources().path).filter(_.ext == "ts").map(PathRef(_))
}
def compile = Task{
val tsc = npmInstall().path / "node_modules/typescript/bin/tsc"
val types = npmInstall().path / "node_modules/@types"
os.call((tsc, allSources().map(_.path), "--outDir", Task.dest, "--typeRoots", types))
PathRef(Task.dest)
}
def mainFileName = Task{ s"${millSourcePath.last}.js" }
def run(args: mill.define.Args) = Task.Command{
val mainFile = compile().path / mainFileName()
os.call(("node", mainFile, args.value), stdout = os.Inherit)
}
}
We can then instantiate the module three times. Module can be adjacent or nested, as shown belo:
object foo extends TypeScriptModule{
object bar extends TypeScriptModule
}
object qux extends TypeScriptModule
interface User { firstName: string }
const user: User = { firstName: process.argv[2] }
console.log("Hello " + user.firstName + " Foo")
interface User { firstName: string }
const user: User = { firstName: process.argv[2] }
console.log("Hello " + user.firstName + " Bar")
interface User { firstName: string }
const user: User = { firstName: process.argv[2] }
console.log("Hello " + user.firstName + " Qux")
And then invoke the .run
method on each module from the command line:
> mill foo.run James
Hello James Foo
> mill foo.bar.run James
Hello James Bar
> mill qux.run James
Hello James Qux
At this point, we have multiple TypeScriptModule
s, with bar
nested inside foo
,
but they are each independent and do not depend on one another.
Next, we will look at how to wire them up using
moduleDeps
.
TypeScriptModule moduleDeps
This example extends TypeScriptModule
to support moduleDeps
.
-
The
def compile
task is considerably fleshed out: rather than using command line flags, we generate atsconfig.json
file using theujson.Obj
/ujson.Arr
JSON factory methods from the bundled uPickle library. -
def compile
now returns twoPathRef`s: one containing the `.js
output to use indef run
, and one containing the.d.ts
output to use in downstreamcompile
s. -
def moduleDeps
is used to allow differentTypeScriptModule
s to depend on each other, and we useTask.traverse
to combine the upstreamcompiledDefinitions
for use incompile
, andcompiledJavascript
for use inrun
package build
import mill._
trait TypeScriptModule extends Module{
def moduleDeps: Seq[TypeScriptModule] = Nil
def npmInstall = Task{
os.call(("npm", "install", "--save-dev", "typescript@5.6.3", "@types/node@22.7.8"))
PathRef(Task.dest)
}
def sources = Task.Source(millSourcePath / "src")
def allSources = Task{ os.walk(sources().path).filter(_.ext == "ts").map(PathRef(_)) }
def compile: T[(PathRef, PathRef)] = Task{
val nodeTypes = npmInstall().path / "node_modules/@types"
val javascriptOut = Task.dest / "javascript"
val declarationsOut = Task.dest / "declarations"
val upstreamPaths =
for(((jsDir, dTsDir), mod) <- Task.traverse(moduleDeps)(_.compile)().zip(moduleDeps))
yield (mod.millSourcePath.subRelativeTo(build.millSourcePath) + "/*", dTsDir.path)
val allPaths = upstreamPaths ++ Seq("*" -> sources().path)
os.write(
Task.dest / "tsconfig.json",
ujson.Obj(
"compilerOptions" -> ujson.Obj(
"outDir" -> javascriptOut.toString,
"declaration" -> true,
"declarationDir" -> declarationsOut.toString,
"typeRoots" -> ujson.Arr(nodeTypes.toString),
"paths" -> ujson.Obj.from(allPaths.map{case (k, v) => (k, ujson.Arr(s"$v/*"))})
),
"files" -> allSources().map(_.path.toString),
)
)
os.call(npmInstall().path / "node_modules/typescript/bin/tsc")
(PathRef(javascriptOut), PathRef(declarationsOut))
}
def mainFileName = Task{ s"${millSourcePath.last}.js" }
def run(args: mill.define.Args) = Task.Command{
val upstream = Task.traverse(moduleDeps)(_.compile)().zip(moduleDeps)
for(((jsDir, tTsDir), mod) <- upstream) {
os.copy(jsDir.path, Task.dest / mod.millSourcePath.subRelativeTo(build.millSourcePath))
}
val mainFile = compile()._1.path / mainFileName()
os.call(
("node", mainFile, args.value),
stdout = os.Inherit,
env = Map("NODE_PATH" -> Seq(".", compile()._1.path).mkString(":"))
)
}
}
Note the use of Task.traverse(moduleDeps)
in order to aggregate the compile
output of the upstream modules, which is necessary both to configure the tsc
TypeScript
compiler in compile
and also to set up the node
working directory in run
. This
is a common pattern when defining language modules, whose module-level dependencies need
to be translated into task-level dependencies
Again, we can instantiate TypeScriptModule
three times, but now foo/src/foo.ts
and foo/bar/src/bar.ts
export their APIs which are then imported in qux/src/qux.ts
:
object foo extends TypeScriptModule{
object bar extends TypeScriptModule
}
object qux extends TypeScriptModule{
def moduleDeps = Seq(foo, foo.bar)
}
interface User {
firstName: string
lastName: string
role: string
}
export {User}
const defaultRole = "Professor"
export {defaultRole}
import {User} from "foo/foo.js"
import {defaultRole} from "foo/bar/bar.js"
const user: User = {
firstName: process.argv[2],
lastName: process.argv[3],
role: defaultRole,
}
console.log("Hello " + user.firstName + " " + user.lastName + " " + user.role)
We can then invoke the qux.run
method on each module from the command line:
> mill qux.run James Bond
Hello James Bond Professor
The dependency graph of tasks now looks like this, with the output of
foo.compile
and bar.compile
now being fed into qux.compile
(and
ultimately qux.run
):
NPM dependencies and bundling
This example expands TypeScriptModule
in two ways:
-
Previously,
def npmInstall
was hardcoded to installtypescript
and@types/node
, because that was what was needed to compile Typescript against the builtin Node.js APIs. In this example, we add adef npmDeps
task, that is aggregated usingTask.traverse
into ` def transitiveNpmDeps`, that are then included in the body ofdef npmInstall
. ThenpmInstall
destination folder in then used both indef compile
to provide thetsc
compiler and supporting installed type definitions, as well as indef run
in order to provide the necessary files to thenode
runtime. -
We include
esbuild@0.24.0
as part of ournpm install
, for use in adef bundle
task that uses it to callesbuild
to bundle our 3TypeScriptModule
s into a singlebundle.js
file. The logic shared betweendef run
anddef bundle
has been extracted into adef prepareRun
task.
package build
import mill._
trait TypeScriptModule extends Module{
def moduleDeps: Seq[TypeScriptModule] = Nil
def npmDeps: T[Seq[String]] = Task{ Seq.empty[String] }
def transitiveNpmDeps: T[Seq[String]] = Task{
Task.traverse(moduleDeps)(_.npmDeps)().flatten ++ npmDeps()
}
def npmInstall = Task{
os.call((
"npm", "install", "--save-dev",
"typescript@5.6.3", "@types/node@22.7.8", "esbuild@0.24.0",
transitiveNpmDeps()
))
PathRef(Task.dest)
}
def sources = Task.Source(millSourcePath / "src")
def allSources = Task{ os.walk(sources().path).filter(_.ext == "ts").map(PathRef(_)) }
def compile: T[(PathRef, PathRef)] = Task{
val nodeTypes = npmInstall().path / "node_modules/@types"
val javascriptOut = Task.dest / "javascript"
val declarationsOut = Task.dest / "declarations"
val upstreamPaths =
for(((jsDir, dTsDir), mod) <- Task.traverse(moduleDeps)(_.compile)().zip(moduleDeps))
yield (mod.millSourcePath.subRelativeTo(build.millSourcePath) + "/*", dTsDir.path)
val allPaths = upstreamPaths ++ Seq("*" -> sources().path, "*" -> npmInstall().path)
os.write(
Task.dest / "tsconfig.json",
ujson.Obj(
"compilerOptions" -> ujson.Obj(
"outDir" -> javascriptOut.toString,
"declaration" -> true,
"declarationDir" -> declarationsOut.toString,
"typeRoots" -> ujson.Arr(nodeTypes.toString),
"paths" -> ujson.Obj.from(allPaths.map{case (k, v) => (k, ujson.Arr(s"$v/*"))})
),
"files" -> allSources().map(_.path.toString),
)
)
os.call((npmInstall().path / "node_modules/typescript/bin/tsc"))
(PathRef(javascriptOut), PathRef(declarationsOut))
}
def mainFileName = Task{ s"${millSourcePath.last}.js" }
def prepareRun = Task.Anon{
val upstream = Task.traverse(moduleDeps)(_.compile)().zip(moduleDeps)
for(((jsDir, tTsDir), mod) <- upstream) {
os.copy(jsDir.path, Task.dest / mod.millSourcePath.subRelativeTo(build.millSourcePath))
}
val mainFile = compile()._1.path / mainFileName()
val env = Map("NODE_PATH" -> Seq(".", compile()._1.path, npmInstall().path).mkString(":"))
(mainFile, env)
}
def run(args: mill.define.Args) = Task.Command{
val (mainFile, env) = prepareRun()
os.call(("node", mainFile, args.value), stdout = os.Inherit, env = env)
}
def bundle = Task{
val (mainFile, env) = prepareRun()
val esbuild = npmInstall().path / "node_modules/esbuild/bin/esbuild"
val bundle = Task.dest / "bundle.js"
os.call((esbuild, mainFile, "--bundle", s"--outfile=$bundle"), env = env)
PathRef(bundle)
}
}
object foo extends TypeScriptModule{
object bar extends TypeScriptModule{
def npmDeps = Seq("immutable@4.3.7")
}
}
object qux extends TypeScriptModule{
def moduleDeps = Seq(foo, foo.bar)
}
We can now not only invoke the qux.run
to run the TypeScriptModule
immediately
using node
, we can also use qux.bundle
to generate a bundle.js
file we can run
standalone using node
:
> mill qux.run James Bond prof
Hello James Bond Professor
> mill show qux.bundle
".../out/qux/bundle.dest/bundle.js"
> node out/qux/bundle.dest/bundle.js James Bond prof
Hello James Bond Professor
The final module tree and task graph is now as follows, with the
additional npmDeps
tasks upstream and the bundle
tasks downstream:
As mentioned earlier, the TypeScriptModule
examples on this page are meant for
demo purposes: to show what it looks like to add support in Mill for a new
programming language toolchain. It would take significantly more work to flesh out
the featureset and performance of TypeScriptModule
to be usable in a real world
build. But this should be enough to get you started working with Mill to add support
to any language you need: whether it’s TypeScript or some other language, most programming
language toolchains have similar concepts of compile
, run
, bundle
, etc.