mill-integrationtest - Integration test plugin for mill plugins

Quickstart

We assume, you have a mill plugin named mill-demo

// build.sc
import mill._, mill.scalalib._
object demo extends ScalaModule with PublishModule {
  // ...
}

Add an new test sub-project, e.g. itest.

import $ivy.`de.tototec::de.tobiasroeser.mill.integrationtest:0.3.3`
import de.tobiasroeser.mill.integrationtest._

object itest extends MillIntegrationTestModule {

  def millTestVersion = "0.6.2"

  def pluginsUnderTest = Seq(demo)

}

Your project should now look similar to this:

.
+-- demo/
|   +-- src/
|
+-- itest/
    +-- src/
        +-- 01-first-test/
        |   +-- build.sc
        |   +-- src/
        |
        +-- 02-second-test/
            +-- build.sc

As the buildfiles build.sc in your test cases typically want to access the locally built plugin(s), the plugins publishes all modules referenced under pluginsUnderTest and temporaryIvyModule to a temporary ivy repository, just before the test is executed. The mill version used in the integration test then uses that temporary ivy repository.

Instead of referring to your plugin with import $ivy.'your::plugin:version', you can use the following line instead, which ensures you will use the correct locally build plugins.

// build.sc
import $exec.plugins

Effectively, at execution time, this line gets replaced by the content of plugins.sc, a file which was generated just before the test started to execute. It will $ivy import all dependencies you listed in pluginsUnderTest.

Example for a generated plugins.sc
// Import a locally published version of the plugin under test
import $ivy.`org.example:mill-demo_2.12:0.1.0-SNAPSHOT`

Configuration and Targets

Mill 0.6.2 or newer is required. See also Version Compatibility Matrix.

The MillIntegrationTestModule trait provides the following targets:

Mandatory configuration
  • def millTestVersion: T[String] The mill version used for executing the test cases. Used by downloadMillTestVersion to automatically download.

  • def pluginsUnderTest: Seq[PublishModule] - The plugins used in the integration test. You should at least add your plugin under test here. You can also add additional libraries, e.g. those that assist you in the test result validation (e.g. a local test support project). The defined modules will be published into a temporary ivy repository before the tests are executed. In your test build.sc file, instead of the typical import $ivy. line, you should use import $exec.plugins to include all plugins that are defined here.

Optional configuration
  • def temporaryIvyModules: Seq[PublishModule] - Additional modules you need in the temporary ivy repository, but not in the resulting mill build classpath. The defined modules will be published into a temporary ivy repository before the tests are executed. This is almost the same as pluginsUnderTest, but does not end up in the generated plugins.sc.

  • def sources: Sources - Locations where integration tests are located. Each integration test is a sub-directory, containing a complete test mill project.

  • def testCases: Target[Seq[PathRef]] - The directories each representing a mill test case. Derived from sources.

  • def testInvocations: Target[Seq[(PathRef, Seq[TestInvocation.Targets])]] - The test invocations to test the project. Defaults to run TestInvokation.Targets with the targets from and expecting successful execution. For each test case, you can define a seq of invocations.

  • def testTargets: Target[Seq[String]] - Deprecated: Please use testInvocations instead The targets which are called to test the project. Defaults to verify, which should implement test result validation.

  • def downloadMillTestVersion: T[PathRef] - Download the mill version as defined by millTestVersion. Override this, if you need to use a custom built mill version. Returns the PathRef to the mill executable (must have the executable flag).

  • def useCachedMillDownload: T[Boolean] - If true, the downloaded mill version used for tests will be cached to the system cache dir (e.g. ~/.cache). Default: true.

  • def showFailedRuns: T[Boolean] - If true, The run log of a failed test case will be shown. Default: true.

Commands / Action Targets
  • def test(): Command[Seq[TestCase]] - Run the integration tests.

  • def testCached: Target[Seq[TestCase]] - Run the integration tests (same as test), but only if any input has changed since the last run.

How can I …​

Run multiple targets in one go

Use testInvocations to configure the targets to execute.

def testInvocations = T{
  Seq(
    pathRefToTest1 -> Seq(
      TestInvocation.Targets(Seq("target1", "target2"))
    )
  )
}

Run multiple mill invocations with different or even the same targets

Use testInvocations to configure the targets to execute.

def testInvocations = T{
  Seq(
    pathRefToTest1 -> Seq(
      // first mill run
      TestInvocation.Targets(Seq("target1", "target2")),
      // second mill run
      TestInvocation.Targets(Seq("target3", "target4")),
      // third mill run with same targets
      TestInvocation.Targets(Seq("target3", "target4"))
    )
  )
}

Test failing mill targets

Use testInvocations to configure the targets to execute and fail.

def testInvocations = T{
  Seq(
    pathRefToTest1 -> Seq(
      // first 2 targets that should succeed
      TestInvocation.Targets(Seq("target1", "target2")),
      // third target should fail with exit code 1
      TestInvocation.Targets(Seq("target3"), expectedExitCode = 1)
    )
  )
}

Better prints context of failed targets

Many test libraries provide nice asserting APIs which produce helpful error messages.

For example, use `munit’s Assertions when defining your test targets

// itest/src/project1/build.sc
import $ivy.`org.scalameta::munit:0.7.7`, munit.Assertions._
def verify() = T.command {
  assert(None.isDefined)
  val fixedScala = read(os.pwd / "foo" / "src" / "Fix.scala")
  val expected   = """object Fix {
                   |  def procedure(): Unit = {} xxx
                   |}
                   |""".stripMargin
  assertEquals(fixedScala, expected)
}

Properly test a mill plugin that uses a worker implementation

You probably want to load the worker in a separated classloader, hence it should not end up in mills classpath. Define the plugin module with pluginsUnderTest and the worker module with temporaryIvyModules. This will ensure that all modules will be build and published to the test ivy repository, but only those listed in pluginsUnderTest will end up in the generated plugins.sc.

def itest extends MillIntegrationTestModule {
  def pluginsUnderTest = Seq(plugin)
  def temporaryIvyModules = Seq(api, worker)
  // ...
}

Test with multiple mill versions, e.g. on a CI server

Mill hasn’t a stable API (yet) and there are no binary compatibility guarantees. So, it is a good idea to add all supported mill version to your CI setup.

The recommended way of supporting multiple mill versions is via mill’s built-in support for cross building (mill.define.Cross).

val millItestVersions = Seq("0.7.3", "0.7.2", "0.7.1", "0.7.0")

object itest extends Cross[ItestCross](millItestVersions: _*)
class ItestCross(millItestVersion: String) extends MillIntegrationTestModule {
  def millTestVersion = millItestVersion
  // correct the source path (remove the extra level for the mill version)
  override def millSourcePath = super.millSourcePath / os.up
  ..
}

Now you can run a single integration test with

mill itest[0.7.3].test

Or you can all integration test in parallel with

mill -j 0 itest[_].test

Cross-Testing a cross built mill-plugin

In case you cross build your mill plugin to support multiple API versions, you need to parametrize your plugins under test.

trait Deps {
  def millVersion = "0.7.0"
  def scalaVersion = "2.13.2"

  val millMain = ivy"com.lihaoyi::mill-main:${millVersion}"
  val millScalalib = ivy"com.lihaoyi::mill-scalalib:${millVersion}"
}
object Deps_0_7 extends Deps
object Deps_0_6 extends Deps {
  override def millVersion = "0.6.0"
  override def scalaVersion = "2.12.10"
}

// The Mill API versions you want to support
val millApiVersions: Map[String, Deps] = ListMap(
  "0.7" -> Deps_0_7,
  "0.6" -> Deps_0_6
)

// The Released Mill versions you want to use in your integration tests
val millItestVersions = Seq(
  "0.7.3", "0.7.2", "0.7.1", "0.7.0",
  "0.6.3", "0.6.2", "0.6.1", "0.6.0"
)

// Your mill plugin
object core extends Cross[CoreCross](millApiVersions.keysIterator.toSeq: _*)
class CoreCross(val millApiVersion: String) extends CrossScalaModule with PublishModule {
  def deps: Deps = millApiVersions(millApiVersion)
  override def crossScalaVersion = deps.scalaVersion
  override def compileIvyDeps = Seq(
    deps.millMain,
    deps.millScalalib
  )
  ..
}

// Your integration test for your mill plugin
object itest extends Cross[ItestCross](millItestVersions: _*)
class ItestCross(millItestVersion: String)  extends MillIntegrationTestModule {
  val millApiVersion = millItestVersion.split("[.]").take(2).mkString(".")
  override def millSourcePath: Path = super.millSourcePath / os.up
  override def millTestVersion = millItestVersion
  override def pluginsUnderTest = Seq(core(millApiVersion))
  ..
}

Have a look at the build.sc of this mill plugin to see how this is done. Here are also two other mill plugins that use this technique:

Collecting integration test coverage data with Scoverage

Mill already provides the mill.contrib.scoverage.ScoverageModule as part of its contrib plugin collection. To ensure you’re using the scoverage-enhanced class files (which are configured to write coverage data into a directrory) in your integration tests, you need to make sure to use the right JAR with the enhanced class files <module>.scoverage.jar instead of the <module>.jar.

To accomplish this, you need to override the protected pluginsUnderTestDetails target and swap the binary JAR in case it is a ScoverageModule. This trick has the effect that we install the scoverage-enhanced JAR file in the test ivy repository.

If you also use temporaryIvyModules, you need to do the same for temporaryIvyModulesDetails.

Important
It’s important to only use the scoverage-enhanced classes in tests. If you would use them outside of your test case, loading them or executing their code would fail in almost all cases.
class core extends ScalaModule with PublishModule with ScoverageModule {
  override def scoverageVersion = "1.4.1"
  ..
}

object itest extends MillIntegrationTestModule {
  override def pluginsUnderTest = Seq(core)
  override def pluginUnderTestDetails: Task.Sequence[(PathRef, (PathRef, (PathRef, (PathRef, (PathRef, Artifact)))))] =
    T.traverse(pluginsUnderTest) { p =>
      val jar = p match {
        case p: ScoverageModule => p.scoverage.jar
        case p => p.jar
      }
      jar zip (p.sourceJar zip (p.docJar zip (p.pom zip (p.ivy zip p.artifactMetadata))))
    }
  ..
}

Also, you need to make sure, that you load the required scoverage runtime library into your mill under test. You can do this by adding the following $ivy import to your build.sc in each test case.

import $ivy.`org.scoverage::scalac-scoverage-runtime:1.4.1`

Now, when you run the integration tests coverage data will be gathered and can be used to generate reports.

mill -j 0 itest.test
mill core.scoverage.htmlReport

Download

You can download binary releases from Maven Central.

License

This project is published under the Apache License, Version 2.0.

Version Compatibility Matrix

Mill is still in active development, and has no stable API yet. Hence, not all mill-integrationtest versions work with every mill version.

The following table shows a matrix of compatible mill and mill-integrationtest versions.

Table 1. Version Compatibility Matrix
mill-integrationtest mill

0.3.2

0.6.2 - 0.7.3

0.3.1

0.6.2 - 0.7.3

0.3.0

0.6.2 - 0.7.3

0.2.1

0.6.0 - 0.6.3

0.2.0

0.5.7

0.1.2

0.5.7

0.1.1

0.5.7

0.1.0

0.3.6 - 0.5.3

About

mill

Mill is a Scala-based open source build tool. In my opinion the best build tool for the JVM. It is fast, reliable and easy to understand.

me

I’m a professional software developer and love to write and use open source software. I’m actively developing and maintaining mill as well as several mill plugins.

If you like my work, please star it on GitHub. You can also support me via GitHub Sponsors.

Contributing

If you found a bug or have a feature request, please open a new issue on GitHub. I also accept pull requests on GitHub.

Changelog

main

mill-integrationtest 0.3.3 - 2020-07-03

  • New option showFailedRuns to always show output of failed runs

mill-integrationtest 0.3.2 - 2020-07-03

  • Re-use mill download cache under ~/.cache

  • Added integration tests

  • Improved output and error reporting

  • Integration test runs now will be written to a dedicated log file

  • When mill it run in debug mode (-d), the complete log of a failed run will be printed after the test summary

  • More documentation

mill-integrationtest 0.3.1 - 2020-05-19

  • Fixed issues on Windows when setting script permissions

mill-integrationtest 0.3.0 - 2020-05-15

  • Cross-publishing for Mill API 0.6.2 (Scala 2.12) and mill API 0.7.0 (Scala 2.13)

  • Use newer mill 0.6.2 API to publish to custom ivy repositories

  • Fixes Windows support

  • Only scan existing source dirs for test cases

mill-integrationtest 0.2.1 - 2020-02-27

  • Bumped Mill API to 0.6.0

mill-integrationtest 0.2.0 - 2020-02-27

  • Added support to run selective tests

  • Targets test and testCached no return the test result

  • new target testCachedArgs to control args feeded to testCachedArgs

  • Test executor now generated a mill script which allows you to manually invoke mill in a test destination directory

  • New target testInvocations providing much finer control over executed targets and their expected exit value

mill-integrationtest 0.1.2 - 2020-02-18

  • New target temporaryIvyModulesDetails

  • New target testCached

mill-integrationtest 0.1.1 - 2020-01-08

  • Version bump mill API to 0.5.7

mill-integrationtest 0.1.0 - 2019-02-21

  • Initial public release