WARP (Workday Automated Regression Platform) is a flexible, lightweight, (mostly) functional Scala framework for collecting performance metrics and conducting sound experimental benchmarking.
WARP features a domain specific language for describing experimental designs (conducting repeated trials for different experimental groups). Our library allows users to wrap existing tests with layers of measurement collectors that write performance metrics to a relational database. We also allow users to create arbiters to express custom failure criteria and performance requirements. One arbiter included out of the box is the RobustPcaArbiter, which uses machine learning to detect when a test is deviating from expected behavior and signal an alert.
We believe engineers should reason more scientifically about code performance, and we are excited to provide a platform that allows for easily describing an experiment, collecting benchmark results, and conducting statistical analyses to verify hypotheses about expected performance.
More detailed documentation can be found here.
Starting local services
You can start up the required services (note that the included docker-compose file is not intended to be used in production) to run tests with
$ docker-compose up -d
And test via
$ ./gradlew clean test
or to just run unit tests, run the
All port values and service version numbers are in
Code Coverage Requirements
We use scoverage and coveralls gradle plugins to track code coverage. We enforce that high coverage should be maintained. At time of writing, coverage must be at least 92% for a build to pass. If you want to test coveralls out on your fork, sign in to coveralls and get your repo token. Then you can generate the coverage reports and submit them to coveralls using
$ export COVERALLS_REPO_TOKEN=abcdefg $ ./gradlew clean reportScoverage coveralls
We use scalafix to automatically refactor code. Since we are interested in using semantic rules, we also need to use semanticdb scala compiler plugin, which harvests and dumps semantic information about the symbols and types in our program. Scalafix semantic rules depend on semanticdb compiler output. Scalafix should be run like this:
$ ./gradlew clean scalafix
We use the
maven-publish gradle plugin. https://docs.gradle.org/current/userguide/publishing_maven.html
Please use the included
publish.sh for uploading artifacts. This script handles some subtle interaction between creating repo tags and scala multiversion plugin.
./publish.sh snapshot minor local
Will increment minor version component and publish a snapshot (eg 2.3.0-SNAPSHOT) to local maven repo.
To publish to sonatype, the invocation would be something like:
./publish.sh candidate minor sonatype
To publish to sonatype, you'll need to configure your sonatype and signing credentials as project properties:
signing.keyId=BEEF signing.password=abc123 signing.secretKeyRingFile=/full/path/to/secring.gpg sonatypeUsername=jean-luc.picard sonatypePassword=makeItSoNumberOne
Signing is not required for a local publish.
We use gradle-scala-multiversion-plugin to cross-compile the project with different scala-lang major versions and publish artifacts with scala version suffixes. The versions are defined in gradle.properties, however you can also override from the command line:
$ ./gradlew -PscalaVersions=2.11.8,2.12.6 test
This plugin works by repeatedly invoking the gradle task graph with each different scala version specified. Without any version specified, gradle will use the defaultScalaVersion from gradle.properties. This means local IDE builds will use just one scala version. If you need to run with all configured scala versions, pass the project property
$ ./gradlew -PallScalaVersions test
Since some of our dependencies are not cross-compiled, currently we only build for 2.12.
We use the
nebula.release plugin to determine versions. https://github.com/nebula-plugins/nebula-release-plugin
Versions exist as git tags in the canonical fork. If you are publishing from your fork, you may need to periodically sync the tags:
git fetch upstream --tags git push origin --tags
There are 4 types of releases we support:
- candidate (rc)
- devSnapshot (includes some extra information in the version, including branch name and commit hash)
Artifacts with type
devSnapshot are published to sonatype snapshots repo, while
candidate artifacts are published to sonatype releases repo.
Please use the included
publish.sh script for publishing, as that script handles interaction between creating repo tags and scala multiversion. We don't want to create multiple tags during a release process and incorrectly publish some artifacts under the wrong version.
To enforce repeatable builds while allowing developers to use the flexibility of dynamic dependency ranges, we use the gradle's native dependency-locking mechanism. Note, gradle creates a separate lockfile per configuration.
We create/update lock files as follows:
./gradlew resolveAndLockAll --write-locks
Note that this locks all configurations in one single build execution. If you have updated
versionInfo.gradle, you probably need to recreate the lock files as well. If any configuration needs to be excluded/filtered, you can do so in the
resolveAndLockAll step. Please see https://docs.gradle.org/6.2.1/userguide/dependency_locking.html#generating_and_updating_dependency_locks for more detailed usage.
Please avoid using global locks. We have noticed behavior where this can override the version of scala-library used by the zinc compiler.
~/code/warp-core(master*) » gradle clean unitTest --refresh-dependencies * What went wrong: A problem occurred evaluating root project 'warp-core'. > com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: org/gradle/api/artifacts/VersionConstraint
You probably need a newer version of gradle. The above error has been observed with gradle 4.2.1, however version 4.4 works. The root cause appears to be a runtime incompatibility with one of the plugins.
$ docker-compose up -d ... ERROR: for mysql Cannot start service mysql: driver failed programming external connectivity on endpoint warpcore_mysql_1 (31cb1d98ed696bba0c9ef8cdee368186c52c15c2d0c0f69f1ffae0e7db406a3d): Error starting userland proxy: Bind for 0.0.0.0:3306 failed: port is already allocated ERROR: Encountered errors while bringing up the project.
Change the MySQL port (or that of the offending service) in
Note that this code has been ported from an internal repository and the public commit history does not accurately reflect authorship. See
CONTRIBUTORS.md for a full list of everyone who has contributed to this project.