ohze / scripted-scalatest-sbt-plugin

A SBT plugin to use ScalaTest with scripted-plugin to test your SBT plugins

Version Matrix

scripted-scalatest-sbt-plugin

CI

A SBT plugin to use Scalatest with scripted-plugin to test your SBT plugins

Traditionally, to test a SBT plugin, you had to create subprojects in /sbt-test, then in the subprojects, create SBT tasks to perform the testing, then specify the tasks to execute in a test file (see https://www.scala-sbt.org/1.x/docs/Testing-sbt-plugins.html).

This is fine when performing simple tests, but for complicated tests (see https://www.scala-sbt.org/1.x/docs/Testing-sbt-plugins.html#step+6%3A+custom+assertion), this can get messy really quickly:

  • It sucks to not be able to write tests in a BDD style (except by using comments, which feels clunky).
  • Manually writing code to print the test results to the console for each subproject is a pain.

This plugin leverages Scalatest's powerful assertion system (to automatically print useful messages on assertion failure) and its expressive DSLs.

This plugin allows you to use any of Scalatest's test Suites, including AsyncTestSuites.

Notes

  • Do not use Scalatest's ParallelTestExecution mixin with this plugin. ScriptedScalatestSuiteMixin runs sbt clean before each test, which may cause weird side effects when run in parallel.
  • When executing SBT tasks in tests, use Project.runTask(<task>, state.value) instead of <task>.value. Calling <task>.value declares it as a dependency, which executes before the tests, not when the line is called.
  • When implementing BeforeAndAfterEach's beforeEach, make sure to invoke super.beforeEach afterwards:
override protected def beforeEach(): Unit = {
  // ...
  super.beforeEach() // To be stackable, must call super.beforeEach
}
  • This SBT plugin is now tested using itself!

Usage

Require sbt 1.2.x+

Step 1. Add sbt-scripted-scalatest

project/plugins.sbt

addSbtPlugin("com.sandinh" % "sbt-scripted-scalatest" % "3.0.3")

Step 2. Add dependencies for your sbt-test's projects

See step 3: src/sbt-test

build.sbt

 lazy val `my-plugin` = project
   .enablePlugins(SbtPlugin)
   .settings(
-    scriptedLaunchOpts := { scriptedLaunchOpts.value ++
-      Seq("-Xmx1024M", "-Dplugin.version=" + version.value)
-    },
-    scriptedBufferLog := false
+    scriptedScalatestDependencies += "org.scalatest::scalatest-wordspec:3.2.10",
   )

scriptedScalatestDependencies will be used to the auto-generated plugins.sbt in all sbt-test's project:

scr/sbt-test/<test-group>/<test-name>/project/plugins.sbt

Ex, for The FunSuite style, set:

scriptedScalatestDependencies ++= Seq(
  "org.scalatest::scalatest-funsuite:3.2.10",
  "org.scalatest::scalatest-mustmatchers:3.2.10",
)

To add addSbtPlugin("com.sandinh" % "sbt-devops" % "5.0.12") into the generated project/plugins.sbt, set:

scriptedScalatestDependencies += "sbt:com.sandinh:sbt-devops:5.0.12"

Step 3. Remove src/sbt-test/*/*/{test, project/plugins.sbt}

Those files are auto-generated by sbt-scripted-scalatest
You can also add this to .gitignore:

**/sbt-test/*/*/project/build.properties
**/sbt-test/*/*/project/plugins.sbt
**/sbt-test/*/*/test

Step 4. Write your test

In sbt-test/<test-group>/<test-name>/build.sbt, create a new Scalatest Suite/Spec, mixin ScriptedScalatestSuiteMixin and pass it into scriptedScalatestSpec. When mixing in ScriptedScalatestSuiteMixin, implement sbtState as state.value.

Using SBT's Example in https://www.scala-sbt.org/1.x/docs/Testing-sbt-plugins.html#step+6%3A+custom+assertion:

import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.matchers.must.Matchers

lazy val root = (project in file("."))
  .settings(
    version := "0.1",
    scalaVersion := "3.0.2",
    assembly/ assemblyJarName := "foo.jar",
    scriptedScalatestSpec := Some(new AnyFunSuite with Matchers with ScriptedScalatestSuiteMixin {
      override val sbtState: State = state.value
      test("assembly must create a JAR that prints out 'bye'") {
        Project.runTask(assembly, sbtState)
        import scala.sys.process._
        val process = s"java -jar ${crossTarget.value / "foo.jar"}"
        process.!!.trim mustBe "bye"
      }
    },
  ),

It is possible move the Scalatest Suite/Spec into a separate .scala file in the project folder, however that may cause issues when trying to access SBT SettingKeys or declaring custom TaskKeys, therefore is currently not recommended except for extremely simple tests. A better approach would be to move all configurations related to this plugin to a new .sbt file, eg. test.sbt.

See Settings for other configurable settings.

Step 5: Use the scripted-plugin as usual

Eg. Run sbt scripted on the main project to execute all tests.

Settings

Setting Type Description
scriptedScalatestSpec Option[Suite with ScriptedScalatestSuiteMixin] Required. The Scalatest Suite/Spec. If not configured (defaults to None), no tests will be executed.
scriptedScalatestDurations Boolean Optional. If true, displays durations of tests. Defaults to true.
scriptedScalatestStacks NoStacks / ShortStacks / FullStacks Optional. The length of stack traces to display for failed tests. NoStacks will not display any stack traces. ShortStacks displays short stack traces. FullStacks displays full stack traces. Defaults to NoStacks.
scriptedScalatestStats Boolean Optional. If true, displays various statistics of tests. Defaults to true.

Tasks

Task Description
scriptedScalatest Executes all test configured in scriptedScalatestSpec. This task must be configured for scripted-plugin to run in the test script file.