mill-integrationtest - Integration test plugin for mill plugins

Integration testing for Mill plugins.

Table of Contents

Quickstart
Configuration and Targets
How can I …
How is mill-integrationtest tesing itself?
Download
License
Version Compatibility Matrix
About
Changelog

Quickstart

Here we assume, you use mill 0.11.0 and develop a mill plugin named demo

// build.sc
import mill._, mill.scalalib._
object demo extends ScalaModule with PublishModule {
  // ...
}

First you need to add a new test module, e.g. itest.

// build.sc
import $ivy.`de.tototec::de.tobiasroeser.mill.integrationtest::0.7.1`
import de.tobiasroeser.mill.integrationtest._

object demo extends ScalaModule with PublishModule {
  // ...
}

object itest extends MillIntegrationTestModule {

  def millTestVersion = "0.11.0"

  def pluginsUnderTest = Seq(demo)

}

Your test cases will be located in the source directory of the newly added itest project. The idea is that each sub-directory represents a separate mill project representing a test case.

Your project should now look similar to this:

.
+-- demo/
|   +-- src/
|
+-- itest/
    +-- src/
        +-- 01-first-test/
        |   +-- build.sc
        |   +-- src/
        |
        +-- 02-second-test/
            +-- build.sc

As the buildfiles build.sc in your test cases typically want to access the locally built plugin(s), the plugins publishes all modules referenced under pluginsUnderTest and temporaryIvyModule to a temporary ivy repository, just before the test is executed. The mill version used in the integration test then uses that temporary ivy repository.

Because you are using your locally developed plugin, instead of referring to your plugin with import $ivy.'your::plugin:version', you should use the following line, which ensures that you use the correct locally build plugins.

// build.sc
import $file.plugins

Effectively, at execution time, Mill is also loading file plugins.sc, a file which was generated just before the test started to execute. It will $ivy import all dependencies you listed in pluginsUnderTest. (If you want to find out how this works, read this)

Example for a generated plugins.sc

// Import a locally published version of the plugin under test
import $ivy.`org.example:mill-demo_2.12:0.1.0-SNAPSHOT`

Configuration and Targets

Mill 0.9.3 or newer is required. See also Version Compatibility Matrix.

The MillIntegrationTestModule trait provides the following targets:

Mandatory configuration

def millTestVersion: T[String] The mill version used for executing the test cases. Used by downloadMillTestVersion to automatically download.
def pluginsUnderTest: Seq[PublishModule] - The plugins used in the integration test. You should at least add your plugin under test here. You can also add additional libraries, e.g. those that assist you in the test result validation (e.g. a local test support project). The defined modules will be published into a temporary ivy repository before the tests are executed. In your test build.sc file, instead of the typical import $ivy. line, you should use import $file.plugins to include all plugins that are defined here.

Optional configuration

def temporaryIvyModules: Seq[PublishModule] - Additional modules you need in the temporary ivy repository, but not in the resulting mill build classpath. The defined modules will be published into a temporary ivy repository before the tests are executed. This is almost the same as pluginsUnderTest, but does not end up in the generated plugins.sc.
def sources: Sources - Locations where integration tests are located. Each integration test is a sub-directory, containing a complete test mill project.
perTestResources: Sources - Shared test resources, will be copied as-is into each test case working directory before the test is run. You can also generate these, making some test setups easier (e.g. including additional classpath resources).
def testCases: Target[Seq[PathRef]] - The directories each representing a mill test case. Derived from sources.
def testInvocations: Target[Seq[(PathRef, Seq[TestInvocation.Targets])]] - The test invocations to test the project. Defaults to run TestInvokation.Targets with the targets from and expecting successful execution. For each test case, you can define a seq of invocations.
def testTargets: Target[Seq[String]] - Deprecated: Please use testInvocations instead The targets which are called to test the project. Defaults to verify, which should implement test result validation.
def downloadMillTestVersion: T[PathRef] - Download the mill version as defined by millTestVersion. Override this, if you need to use a custom built mill version. Returns the PathRef to the mill executable (must have the executable flag).
def useCachedMillDownload: T[Boolean] - If true, the downloaded mill version used for tests will be cached to the system cache dir (e.g. ~/.cache). Default: true.
def showFailedRuns: T[Boolean] - If true, The run log of a failed test case will be shown. Default: true.
def prefetchIvyDeps: T[Agg[Dep]] - Add dependencies here, which you want to prefetch into your local coursier cache before acually running the tests. Each dependency is resolved and fetched independently, so it is possible to fetch multiple versions of the same artifact. Use this target to prepare integration test which should run offline.

Commands / Action Targets

def test(args: String*): Command[Seq[TestCase]] - Run the integration tests. The args here are the actual test cases that will run. By default this will run them all, but it’s also possible to just pass the single test name in to run that single test.
def testCached: Target[Seq[TestCase]] - Run the integration tests (same as test), but only if any input has changed since the last run.
def prepareOffline: Command[Unit] - Prepares going offline by pre-fetching all known dependencies.

How can I …

Run multiple targets in one go

Use testInvocations to configure the targets to execute.

def testInvocations = T{
  Seq(
    pathRefToTest1 -> Seq(
      TestInvocation.Targets(Seq("target1", "target2"))
    )
  )
}

Run multiple mill invocations with different or even the same targets

Use testInvocations to configure the targets to execute.

def testInvocations = T{
  Seq(
    PathRef()-> Seq(
      // first mill run
      TestInvocation.Targets(Seq("target1", "target2")),
      // second mill run
      TestInvocation.Targets(Seq("target3", "target4")),
      // third mill run with same targets
      TestInvocation.Targets(Seq("target3", "target4"))
    )
  )
}

Run a single defined integration test

Given a setup like this:

def testInvocations = T{
  Seq(
    PathRef(testBaseDir / "exampleTestDir") -> Seq(
      TestInvocation.Targets(Seq("target1")),
    )
  )
}

You can run an individual target by passing in the name to itest:

---
mill itest exampleTestDir
---

Test failing mill targets

Use testInvocations to configure the targets to execute and fail.

def testInvocations = T{
  Seq(
    pathRefToTest1 -> Seq(
      // first 2 targets that should succeed
      TestInvocation.Targets(Seq("target1", "target2")),
      // third target should fail with exit code 1
      TestInvocation.Targets(Seq("target3"), expectedExitCode = 1)
    )
  )
}

Better prints context of failed targets

Many test libraries provide nice asserting APIs which produce helpful error messages.

For example, use `munit’s Assertions when defining your test targets

// itest/src/project1/build.sc
import $ivy.`org.scalameta::munit:0.7.7`, munit.Assertions._
def verify() = T.command {
  assert(None.isDefined)
  val fixedScala = read(os.pwd / "foo" / "src" / "Fix.scala")
  val expected   = """object Fix {
                   |  def procedure(): Unit = {} xxx
                   |}
                   |""".stripMargin
  assertEquals(fixedScala, expected)
}

Properly test a mill plugin that uses a worker implementation

You probably want to load the worker in a separated classloader, hence it should not end up in mills classpath. Define the plugin module with pluginsUnderTest and the worker module with temporaryIvyModules. This will ensure that all modules will be build and published to the test ivy repository, but only those listed in pluginsUnderTest will end up in the generated plugins.sc.

def itest extends MillIntegrationTestModule {
  def pluginsUnderTest = Seq(plugin)
  def temporaryIvyModules = Seq(api, worker)
  // ...
}

Test with multiple mill versions, e.g. on a CI server

Mill hasn’t a stable API (yet) and there are no binary compatibility guarantees. So, it is a good idea to add all supported mill version to your CI setup.

The recommended way of supporting multiple mill versions is via mill’s built-in support for cross building (mill.define.Cross).

val millItestVersions = Seq("0.7.3", "0.7.2", "0.7.1", "0.7.0")

object itest extends Cross[ItestCross](millItestVersions: _*)
class ItestCross(millItestVersion: String) extends MillIntegrationTestModule {
  def millTestVersion = millItestVersion
  // correct the source path (remove the extra level for the mill version)
  override def millSourcePath = super.millSourcePath / os.up
  ..
}

Now you can run a single integration test with

mill itest[0.7.3].test

Or you can all integration test in parallel with

mill -j 0 itest[_].test

Cross-Testing a cross built mill-plugin

In case you cross build your mill plugin to support multiple API versions, you need to parametrize your plugins under test.

trait Deps {
  def millVersion = "0.7.0"
  def scalaVersion = "2.13.2"

  val millMain = ivy"com.lihaoyi::mill-main:${millVersion}"
  val millScalalib = ivy"com.lihaoyi::mill-scalalib:${millVersion}"
}
object Deps_0_7 extends Deps
object Deps_0_6 extends Deps {
  override def millVersion = "0.6.0"
  override def scalaVersion = "2.12.10"
}

// The Mill API versions you want to support
val millApiVersions: Map[String, Deps] = ListMap(
  "0.7" -> Deps_0_7,
  "0.6" -> Deps_0_6
)

// The Released Mill versions you want to use in your integration tests
val millItestVersions = Seq(
  "0.7.3", "0.7.2", "0.7.1", "0.7.0",
  "0.6.3", "0.6.2", "0.6.1", "0.6.0"
)

// Your mill plugin
object core extends Cross[CoreCross](millApiVersions.keysIterator.toSeq: _*)
class CoreCross(val millApiVersion: String) extends CrossScalaModule with PublishModule {
  def deps: Deps = millApiVersions(millApiVersion)
  override def crossScalaVersion = deps.scalaVersion
  override def compileIvyDeps = Seq(
    deps.millMain,
    deps.millScalalib
  )
  ..
}

// Your integration test for your mill plugin
object itest extends Cross[ItestCross](millItestVersions: _*)
class ItestCross(millItestVersion: String)  extends MillIntegrationTestModule {
  val millApiVersion = millItestVersion.split("[.]").take(2).mkString(".")
  override def millSourcePath: Path = super.millSourcePath / os.up
  override def millTestVersion = millItestVersion
  override def pluginsUnderTest = Seq(core(millApiVersion))
  ..
}

Have a look at the build.sc of this mill plugin to see how this is done. Here are also link to two other mill plugins that uses this technique (at the time of writing this):

Collecting integration test coverage data with Scoverage

Mill already provides the mill.contrib.scoverage.ScoverageModule as part of its contrib plugin collection. To ensure you’re using the scoverage-enhanced class files (which are configured to write coverage data into a directrory) in your integration tests, you need to make sure to use the right JAR with the enhanced class files <module>.scoverage.jar instead of the <module>.jar.

To accomplish this, you need to override the protected pluginsUnderTestDetails target and swap the binary JAR with it’s ScoverageModule version. This trick has the effect that we install the scoverage-enhanced JAR file into the test ivy repository.

If you also use temporaryIvyModules, you need to do the same for temporaryIvyModulesDetails.

Important

It’s important to only use the scoverage-enhanced classes in tests! Do not distribute them.

If you would use them outside of your test case, loading them or executing their code would fail in almost all cases.

class core extends ScalaModule with PublishModule with ScoverageModule {
  override def scoverageVersion = "1.4.11"
  ..
}

object itest extends MillIntegrationTestModule {
  override def pluginsUnderTest = Seq(core)
  override def pluginUnderTestDetails: Task.Sequence[(PathRef, (PathRef, (PathRef, (PathRef, (PathRef, Artifact)))))] =
    T.traverse(pluginsUnderTest) { p =>
      val jar = p match {
        case p: ScoverageModule => p.scoverage.jar
        case p => p.jar
      }
      jar zip (p.sourceJar zip (p.docJar zip (p.pom zip (p.ivy zip p.artifactMetadata))))
    }
  ..
}

Also, you need to make sure, that you load the required scoverage runtime library into your mill under test. You can do this by adding the following $ivy import to your build.sc in each test case.

import $ivy.`org.scoverage::scalac-scoverage-runtime:1.4.11`

Now, when you run the integration tests coverage data will be gathered and can be used to generate reports.

mill -j 0 itest.test
mill core.scoverage.htmlReport

How is `mill-integrationtest` tesing itself?

Glad you asked!

mill-integrationtest is using a previously released version of itself to test itself. This means we have three levels of mill-integrationtest:

The project itself, configured in build.sc
A previously released version of mill-integrationtest to run the integration tests, configured in the cross module itest (in top-level build.sc). The cross parameter denotes the Mill version to run the tests against.
And finally the freshly built mill-integrationtest plugin under test, used in the test cases located under itest/src.

This makes understanding the test setup and the build/test output rather hard to read, even for me.

Download

You can download binary releases from Maven Central.

Newer versions of this plugin (after version 0.3.3) have a mill platform suffix in the artifact name. If you use Mill 0.9.10 or above, you can use the double colon (::) between artifact name and version to always use the right Mill binary platform. On older Mill versions, you need to add the platform suffix manually.

Table 1. Mill Platform suffix

mill version	mill platform	suffix	example
0.10.x	0.10	`_mill0.10`	import $ivy.`de.tototec::de.tobiasroeser.mill.integrationtest::0.7.1`
0.9.3 - 0.9.x	0.9	`_mill0.9`	import $ivy.`de.tototec::de.tobiasroeser.mill.integrationtest_mill0.9:0.7.1`

License

This project is published under the Apache License, Version 2.0.

Version Compatibility Matrix

Mill is still in active development, and has no stable API yet. Hence, not all mill-integrationtest versions work with every mill version.

The following table shows a matrix of compatible mill and mill-integrationtest versions. Newer version of mill may or may not work. (Feel free to update this page via a pull request, thanks.)

Table 2. Version Compatibility Matrix

mill-integrationtest	mill
0.7.0	0.9.x, 0.10.x, 0.11.0-M8
0.6.1	0.9.3 - 0.9.x, 0.10.x
0.6.0	0.9.3 - 0.9.x, 0.10.x
0.5.1	0.9.3 - 0.9.x, 0.10.x
0.5.0	0.9.3 - 0.9.12, 0.10.0 - 0.10.1
0.4.2	0.9.3 - 0.9.12, 0.10.0 - 0.10.1
0.4.1	0.6.2 - 0.9.12
0.4.0	0.6.2 - 0.9.6, (not 0.9.7), 0.9.8 - 0.9.9
0.3.3	0.6.2 - 0.8.0
0.3.2	0.6.2 - 0.8.0
0.3.1	0.6.2 - 0.8.0
0.3.0	0.6.2 - 0.8.0
0.2.1	0.6.0 - 0.6.3
0.2.0	0.5.7
0.1.2	0.5.7
0.1.1	0.5.7
0.1.0	0.3.6 - 0.5.3

About

mill: Mill is a Scala-based open source build tool. In my opinion the best build tool for the JVM. It is fast, reliable and easy to understand.
me: I’m a professional software developer and love to write and use open source software. I’m actively developing and maintaining mill as well as several mill plugins.

If you like my work, please star it on GitHub. You can also support me via GitHub Sponsors.
Contributing: If you found a bug or have a feature request, please open a new issue on GitHub. I also accept pull requests on GitHub.

Changelog

0.7.1 - 2023-06-07

Support for Mill 0.11 API
Update Scala to 2.13.11

0.7.0 - 2023-04-27

Support for Mill 0.11.0-M8
Slight API changes to accommodate to Mill 0.11 (only return values of pluginUnderTestDetails and temoraryIvyModuleDetails)
Minor documentation and internal improvements

0.6.1 - 2022-07-11

Fixed default value for TestInvocation.Targets.noServer

0.6.0 - 2022-04-13

Support running Mill under test in server mode
mill-integrationtest is now also CI tested on Windows

0.5.1 - 2022-04-12

Fixed non-functional tests under Windows

0.5.0 - 2022-03-09

Support to specify environment variables for test runs
Support moduleDeps of tests plugins (to publish them transitively into the test repository)
Updated toolchain to use Mill 0.10.1 and newer plugins

0.4.2 - 2022-03-08

Added support for Mill 0.10
Added support for mill milestone versions
Added prefetcIvyDeps and offline support
Dependency updates
Dropped support for older Mill versions

0.4.1 - 2021-06-09

Improved output, esp. in error case
Added new perTestResources target
Work around binary compatibility issues with mill 0.9.7

0.4.0 - 2020-11-30

Added support for mill 0.9.3 while maintaining backward-compatible versions down to mill 0.6.2
Introduce a new artifact name suffix (_mill0.9 for mil 0.9.3) to support multiple mill API versions.
Various version bumps: scalatest 3.2.3, scalafmt 2.7.5, scoverage 1.4.2

0.3.3 - 2020-07-03

New option showFailedRuns to always show output of failed runs

0.3.2 - 2020-07-03

Re-use mill download cache under ~/.cache
Added integration tests
Improved output and error reporting
Integration test runs now will be written to a dedicated log file
When mill it run in debug mode (-d), the complete log of a failed run will be printed after the test summary
More documentation

0.3.1 - 2020-05-19

Fixed issues on Windows when setting script permissions

0.3.0 - 2020-05-15

Cross-publishing for Mill API 0.6.2 (Scala 2.12) and mill API 0.7.0 (Scala 2.13)
Use newer mill 0.6.2 API to publish to custom ivy repositories
Fixes Windows support
Only scan existing source dirs for test cases

0.2.1 - 2020-02-27

Bumped Mill API to 0.6.0

0.2.0 - 2020-02-27

Added support to run selective tests
Targets test and testCached no return the test result
new target testCachedArgs to control args feeded to testCachedArgs
Test executor now generated a mill script which allows you to manually invoke mill in a test destination directory
New target testInvocations providing much finer control over executed targets and their expected exit value

0.1.2 - 2020-02-18

New target temporaryIvyModulesDetails
New target testCached

0.1.1 - 2020-01-08

Version bump mill API to 0.5.7

0.1.0 - 2019-02-21

Initial public release

lefou / mill-integrationtest 0.7.1