Test your plugin against different versions of Gradle as part of your build. This plugin extends the power that the new GradleTestKit brings without having to actually author code. It rather allows plugins authors to write functional tests that looks like normal Gradle projects (multi-project is supported). This also makes it easier to create sample projects that can directly be used in documentation.
If you are using a Gradle version < 2.13, it will run in legacy mode and behave like the 0.5.5 version of the plugin. |
Bootstrap
GradleTest is available in the Gradle Plugins repository.
To use it, add the appropriate plugin to the plugins
block.
plugins {
id 'org.ysb33r.gradletest' version '1.0' (1)
id 'org.ysb33r.gradletest.base' version '1.0' (2)
}
1 | The common use case for compatibility testing |
2 | Use the base plugin when you do not require the default gradleTest tasks (and related source sets) |
When your setup is more complex and the plugins block does not work OR if you are using Gradle 2.0 use the following instead
buildscript {
repositories {
maven {
url "https://plugins.gradle.org/m2/"
}
}
dependencies {
classpath 'org.ysb33r.gradle:gradletest:1.0'
}
}
apply plugin: 'org.ysb33r.gradletest'
Dynamic dependencies: Hard-coding the plugin version in to the build.gradle
files of the gradleTest
test fixtures is a maintenance pain. However the plugin is injected automatically into the build via GradleTestKit
.
Testkit Mode (Gradle 2.13+)
Getting Started
In build.gradle
define the versions of Gradle that needs to be tested:
gradleTest {
versions '2.9', '3.4', '4.1'
}
Create a folder called src/gradleTest
. Beneath that add a folder - call it 'myExample` for now. Within that create a normal Gradle project i.e. drop a build.gradle
file and fill it out with the kind of syntax that a person will use to run your plugin. Finally add a custom task called runGradleTest
and make it depend on some task that your plugin provides.
For example here is one example from GradleTest’s test suite of how itself is being tested.
apply plugin : 'org.ysb33r.gradletest'
repositories {
jcenter()
}
gradleTest {
versions '2.12','2.13'
expectFailure 'expectedBuildFailure'
}
task runGradleTest( dependsOn : ['gradleTest'] )
Conventions
Filesystem
${projectDir} / src / gradleTest / project1 / build.gradle project2 / project3 /
Each directory below gradleTest
becomes a test. Tests are executed in-folder
and expects a build.gradle
file. If the latter is not found the test will not be
executed.
For testing, a folder is created for each gradle version to be tested and the projects are duplicated below the verson folder. Each version testsute is executed within a separate JVM.
${buildDir} / gradleTest / project1 / ver1 (1) / ver2 (2) / project2 / ver1 / manifest (3) / src (4) / init.gradle (5)
1 | A folder is created for each project |
2 | A folder for each Gradle version is created underneath the project folder |
3 | Classpath manifest used when running tests |
4 | Folder where generated source code ends up in |
5 | Initscript used by GradleTest |
... / project1 / .gradle (1) / build (2) / build.gradle (3) / settings.gradle (4)
1 | Project cachedir is sent here |
2 | It is recommended that the build directory not be changed and left as per default |
3 | build.gradle is required for the test to be executed. It must contain a task called runGradleTest . |
4 | If a test project does not have a settings.gradle file an empty one will
be generated in the test folder |
apply plugin : 'my.project.id' (1) task runGradleTest { dependsOn 'someTask' (2) }
1 | Use apply plugin instead as it works across a larger range of Gradle versions. (No need for buildscript block
as GradleTest will take care of injecting the plugin on the classpath). |
2 | Create a runGradleTest task and make it depend on the real task from your plugin that needs execution. |
Gradle Entities
A GradleTest
task called gradleTest
will have the following associated entities:
Entity | Type | Description |
---|---|---|
|
Configuration |
For local available dependencies. |
|
Task |
The actual |
|
SourceSet |
Manages internal generated source code |
|
Configuration |
Internal configuration for dependencies required to compile compatibility tests. |
|
Configuration |
Internal configuration for dependencies required execute tests (different classpath to those passed to
actual |
|
Task |
Generates test code that will be compiled and used to actually executed |
|
Task |
Compiles Groovy code generated by |
|
Task |
Associated |
|
Task |
Creates a classpath manifest file that will be used during test execution to load the correct plugin and transitive dependencies |
Temporarily overriding Gradle versions
It is very convenient to sometimes to runs the test with a subset of the versions specified in the configuration. This is achievable by passing a system property to Gradle
-DgradleTest.versions=2.2,2.3 (1) (2)
1 | If more than one GradleTest task is defined, replace gradleTest with the name of the appropriate GradleTest task. |
2 | List the versions in a commma-separated list. No validation is done on the string. If it leads to invalid Gradle versions the build will fail. |
Task dependencies
gradleTest
is linked into the check
lifecycle task.
HTML Reports
HTML reports will appear as folder named after the GradleTest
instance below the ${reporting.baseDir}
folder.
Practically in most cases this will be build/reports/gradleTest
.
Add additional compatibility test sets
Use the additionalGradleTestSet
project extension. Specify the basename and all of
the appropriate configurations and tasks will be added.
additionalGradleTestSet 'nonLegacyOnly' (1)
nonLegacyOnlyGradleTest { (2)
versions '4.0', '4.1', '4.2'
versions '3.5', '3.4.1', '3.3'
versions '3.2', '3.1', '3.0' (3)
versions '2.14.1'
}
nonLegacyOnlyGradleTestGenerator { (4)
pluginJarDirectory = gradleTestGenerator.pluginJarDirectory
}
additionalGradleTestSet 'kotlin'
kotlinGradleTest {
versions '4.0', '4.1', '4.2'
}
kotlinGradleTestGenerator {
pluginJarDirectory = gradleTestGenerator.pluginJarDirectory
}
1 | Supply basename to additionalGradleTestSet |
2 | This will result in a test task in the format ${baseName}GradleTest . Sources are expected to be layed out below
the src/${baseName}GradleTest folder in a similar fashion to gradleTest |
3 | Configure test task as per normal. |
4 | Other tasks in the groups are all named using the same convention and can also just be configured as per normal. |
More feedback
Running lots of compatibility tests can take some time. If you need more visual feedback
regarding progress try adding something like this to the gradleTest
configuration.
gradleTest{
beforeTest {
println " ${it.name}" (1)
}
}
1 | Print the name of the current compatibility test that is runnign |
Kotlin DSL
With the growing use of Kotlin-based build scripts, it is also important to test compatibility of a plugin for this context. Turn Kotlin-testing on by setting the following:
gradleTest {
kotlinDsl = true
}
Now drop a build.gradle.kts
file alongside your build.gradle
build script. If the gradle version under test is 4.0 or later, GradleTest will also run a test using the Kotlin build script.
Command-line Awareness
GradleTest is aware of certain command-line parameters and will pass them on to running tests.
-
--rerun-tasks
- All tests will be re-run even if they were previsouly successful. -
--offline
- No dependency checks will be performed and if plugins implements any offline behaviour this will automatically be enabled.
Advanced Features
Caching Dependencies
Although gradle tests can download their own dependencies, this might consume unnecessary
bandwidth and waste a lot of testing time. In order to combat this,
any dependencies listed under gradleTest
configuration will be downloaded and
made available to the running gradle tests.
dependencies {
gradleTest 'commons-cli:commons-cli:1.2'
}
These dependencies then appear as a flatDir
repository in the gradle test.
NOTE: It is not necessary to add your plugin to the dependencies. The output of the jar
task
is automatically added to the gradleTest
configuration.
buildscript {
dependencies {
classpath ':gnumake:1.0.1' (1)
}
}
dependencies {
compile ':commons-cli:1.2' (2)
}
1 | It is completely possible to add it to the buildscript for loading
plugins |
2 | Load up any dependencies a per normal |
NOTE: This repository is injected into the test using an internal initialisation script.
Overriding internal dependencies
GradleTest uses Spock Framework, JUnit & Apache Commons IO underneath. If, for some reason you need to override the
versions of these dependencies it can be done in the gradleTestCompile
configuration. If you created a new GradleTest
task called foobar
then the appropriate configuration will be called foobarCompile
.
Base plugin
If the convention of adding a gradleTest
with it’s default settings are not suitable for your
environment, then the base plugin can be considered. This will only add the
additionalGradleTestSet
method.
The GradleTest Base Plugin is available in the Gradle Plugins repository.
To use it, add the following snippet into your build script.
plugins {
id 'org.ysb33r.gradletest.base' version '1.0'
}
When your setup is more complex and the plugins block does not work OR if you are using Gradle 2.0 use the following instead
buildscript {
repositories {
maven {
url "https://plugins.gradle.org/m2/"
}
}
dependencies {
classpath 'org.ysb33r.gradle:gradletest:1.0'
}
}
apply plugin: 'org.ysb33r.gradletest.base'
Known Limitations (Gradle 2.13+)
-
Not designed to test with Gradle < 2.0. If the community requires this functionality an effort will be made to see if it is possible. (It will actually try to runs tests against a Gradle version < 2.0, if configured in
gradleTest
by YMMV). -
Runs in legacy mode when the project is built with Gradle 2.0 - 2.12.
Upgrading from v0.5.x
%%.VERSION%%
is no longer supported when building with Gradle 2.13+ and builds will fail with
Could not find foo:bar:%%VERSION%%`
error if encountered.
It will still be supported if using Gradle 2.12 or earlier as legacy mode will be invoked.
Even under legacy mode it’s use is no longer required. Simply remove the classpath, and if
you don’t anything extra in the buildscript
block, the whole of the buildscript
block.
For instance the following block can be complety removed:
buildscript {
dependencies {
classpath 'com.github.jruby-gradle:jruby-gradle-plugin:%%VERSION%%'
}
}
Compatibility with Other Plugins
License plugin
The License plugin is a really useful plugin to help with
maintenance of license headers in source code. One standard feature of this plugin is to scan all source sets
for license governance. Unfortunately this detects the source sets that are added by GradleTest
itself which has
its own internally generated license headers.
There is an open issue against this,
but for now the best workaround is to exclude the test code generated by GradleTest
.
license {
exclude '**/*CompatibilitySpec.groovy'
}
CodeNarc plugin
When adding GradleTest to a project that also has codenarc configured, the tests generated by GradleTest will be scanned by CodeNarc.
codenarcGradleTest.enabled = false
Alternative Approaches to GradleTest
Gradle TestKit
GradleTestKit
is the core API that ships with Gradle since Gradle 2.6. which is meant for plugin developers to test compatibility across Gradle versions. Easrly versions could only tests against the same version of Gradle that was running, but it has been enhanced over different Gradle releases. Today it is powereful allowing multiple versions to be tested and a number of different environments to be set up.
It is also a core compoenent of GradleTest
when the version of Gradle is 2.13 or later. (When the current build version is older GradleTest
falls back to legacy mode, simply because the GradleTestKit
API either is non-existent or not mature enough.
GradleTestKit
works well for integration testing when writing plugins if a select small group of Gradle versions are used or even if it only targets the current Gradle version. In this way one can quickly determine issues in a plugin which cannot be discovered via normal testing. GradleTest
used GradleTestKit
directly for its integration tests for exactly this reason.
Testing multiple versions of Gradle with GradleTestKit soon bcomes a burden for developers as the API differs and classpath setups can be tricky. Older versions of Gradle give the most issues. In addition GradleTestKit
is essentially a library it focuses on the developer rather than the build script user.
In this regard GradleTest
steps in as it allow tests to be written as if a normal buildscript would be written. it also manages differences (or non-existance) in API from Gradle 2.0 onwards in a completely transparent way to the plugin author.
Overall the approach should be to write integration tests with GradleTestKit
using a limited or even just one Gradle version. This is then complimented by some GradleTest
scripts to test a wider range of Gradle versions.
Stutter
Stutter is another library by Andy Oberstar to aid with plugin development. It focuses on the cases where you might have to run the same tests against all supported versions, but don’t want to deal with making each test loop over those versions (or @Unroll
with a where
block in Spock). Stutter
generates one task per supported version for you and runs the full set of tests against it.
It definitely removes some of the complexity of setting up the GradleTestKit
environment, but beware that it works from Gradle 2.13 onwards. The latter is not a restriction, but essentialy the same reason why GradleTest
reverts to legacy mode if the build version of Gradle is older than 2.13: a lack of maturity in the GradleTestKit
API in those releases.
It adds a compatTest
source set, which will not clash with the gradleTest
source set from GradleTest
.
Stutter
is essentialy developer-centric whereas GradleTest
is script author-centric. This distinction is important as it clarifies different levels of validation:
-
If you want to create good samples and make sure they build, GradleTest is a clear win. Not having to write the extra test code (or touch the TestKit API) is a great benefit.
-
If you need to validate what the build did, you are back in
GradleTestKit
land and thenStutter
will make it easier for you.
It can be an easy way to manage some more complex plugin integration tests. There is no reason to not use it alongside GradleTest
as part of plugin development.
GradleTest Legacy Mode (Gradle 2.0-2.12)
Global Configuration
gradleLocations {
searchGradleUserHome = true (1)
includeGradleHome = true (2)
searchGvm = true (3)
download = true (4)
downloadToGradleUserHome = false (5)
useGradleSite = true (6)
uri (7)
search (8)
}
1 | Search Gradle home for Gradle installations. |
2 | Include the current GRADLE_HOME folder. |
3 | Search the GVM folder for Gradle installations |
4 | Download uninstalled version if online. These distributions will be
placed in the appropriate location in gradle.gradleHome . |
5 | Place downloaded distributions in gradle.gradleUserHomeDir in the same way
Gradle wrapper will do. |
6 | Download distribution from the global Gradle distribution site. |
7 | List additional URLs to search for Gradle distributions. |
8 | Search these additional folders for Gradle installations. Search will be performed
both as if it a cache-style folder (aka gradleUserHome ) or an installation-style folder
(aka GVM_HOME ). |
If the above is used with Gradle 2.13+ it will be accepted and ignored. Deprecation warnings will also be printed. |
Dynamic dependencies
Hard-coding the plugin version in to the build.gradle
files of the gradleTest
test fixtures is a maintenance pain.
In previous versions of the plugin one had to dp to write something like
buildscript {
dependencies {
classpath ':gnumake:%%VERSION%%'
}
}
and the plugin would substitute the %%VERSION%%
token with the version of your project. This is no longer necessary
with this release even when running in legacy mode. The plugin will inject the path to the plugin automatically
eliminating the need to use a block like the above at all.
Task dependencies
The gradleTest
task is not linked to any other tasks. Run this as
explicit task on the command-line or add your own task dependencies in your
gradle script. The reason for this is that in legacy mode it can be quite a time-consuming testset to run. The typical
case will be that the tests are only run close to release time. If you prefer you might want to set
install.dependsOn gradleTest
rather than check.dependsOn gradleTest
or build.dependsOn gradleTest
.
Test directory layout
For testing, a folder is created for each gradle version to be tested and the projects are duplicated below the verson folder. Each version’s testsuite is executed within a separate JVM.
${buildDir} / gradleTest / ver1 / project1 (1) / project2 / project3 / ver2 / project1 / project2 / project3 / init20.gradle (2) / repo (3) / home (4) / classes / gradleTest (5) / gradleDist (6)
1 | Projects are duplicated for each Gradle version. See below for a more detailed layout |
2 | This is a generated initscript used to start all tests. |
3 | This is flat repository that can be utilised by all tests. |
4 | Gradle home directory for all the tests |
5 | Temporary classes for bootstrapping the tests are kept here. |
6 | Distributions are downloaded here if necessary |
... / project1 / .gradle (1) / src (2) / build (3) / build.gradle (4) / settings.gradle (5)
1 | Project cachedir is sent here |
2 | If the test project has any folders they will be copied here |
3 | It is recommended that the build directory not be changed and left as per default |
4 | build.gradle is required for the test to be executed. It must contain a task called runGradleTest . |
5 | If a test project does not have a settings.gradle file an empty one will
be generated in the test folder |
Adding additional GradleTest tasks
It is possible to add additional test tasks beyond gradleTest
, by doing
configurations {
furtherTest
}
task furtherTest( type : org.ysb33r.gradle.gradletest.legacy20.GradleTest ) {
versions '2.2'
}
Test files should be placed under src/furtherTest
using the same layout as described earlier. Dependencies should be
listed under furtherTest
configuration.
Global configuration is still read from gradleLocations
project extension.
Known Limitations (Gradle 2.0-2.12)
-
The plugin assumes that no Gradle distributions in
gradle,gradleUserHomeDir
orGVM_HOME
will be removed whilst it is running. -
The source sets for the Gradle tests cannot be renamed or added to. The subdirectory name is fixed to the task name.
-
No nice HTML report (https://github.com/ysb33r/gradleTest/issues/2)
-
No graceful failure as for
test
task. Currently throws aTaskExecutionException
at the end, which is ugly. (https://github.com/ysb33r/gradleTest/issues/1) -
No running counter of tests run and test failures (as for
test
task). (https://github.com/ysb33r/gradleTest/issues/3) -
All test output is going to stdout instead of being captured and added to test report. (https://github.com/ysb33r/gradleTest/issues/4)
-
--no-daemon
is set, as we don’t want to clash with existing running daemons. -
Init scripts can no longer be passed. (This will probably be addressed in another release).
-
Does not run test in parallel, even though it theoretically could
Unresolved directive in product-documentation.adoc - include::parts/gradle-runner.adoc[]