Test your plugin against different versions of Gradle as part of your build. This plugin extends the power that the new GradleTestKit brings without having to actually author code. It rather allows plugins authors to write functional tests that looks like normal Gradle projects (multi-project is supported). This also makes it easier to create sample projects that can directly be used in documentation.

If you are using a Gradle version < 2.13, it will run in legacy mode and behave like the 0.5.5 version of the plugin.


GradleTest is available in the Gradle Plugins repository. To use it, add the appropriate plugin to the plugins block.

plugins {
  id 'org.ysb33r.gradletest' version '1.1' (1)
  id 'org.ysb33r.gradletest.base' version '1.1' (2)
  id 'org.ysb33r.gradlerunner' version '1.1' (3)
1 The common use case for compatibility testing
2 Use the base plugin when you do not require the default gradleTest tasks (and related source sets)
3 Use the gradlerunner to easily execute independent steps of GradleRunner.

When your setup is more complex and the plugins block does not work OR if you are using Gradle 2.0 use the following instead

buildscript {
  repositories {
    maven {
      url "https://plugins.gradle.org/m2/"

  dependencies {
    classpath 'org.ysb33r.gradle:gradletest:1.1'

apply plugin: 'org.ysb33r.gradletest'

Dynamic dependencies: Hard-coding the plugin version in to the build.gradle files of the gradleTest test fixtures is a maintenance pain. However the plugin is injected automatically into the build via GradleTestKit.

Testkit Mode (Gradle 2.13+)

Getting Started

In build.gradle define the versions of Gradle that needs to be tested:

gradleTest {
   versions '2.9', '3.4', '4.1'

Create a folder called src/gradleTest. Beneath that add a folder - call it 'myExample` for now. Within that create a normal Gradle project i.e. drop a build.gradle file and fill it out with the kind of syntax that a person will use to run your plugin. Finally add a custom task called runGradleTest and make it depend on some task that your plugin provides.

For example here is one example from GradleTest’s test suite of how itself is being tested.

apply plugin : 'org.ysb33r.gradletest'

repositories {

gradleTest {
    versions '2.12','2.13'
    expectFailure 'expectedBuildFailure'

task runGradleTest( dependsOn : ['gradleTest'] )



Source directory layout
${projectDir} / src / gradleTest /  project1  / build.gradle
                                    project2  /
                                    project3  /

Each directory below gradleTest becomes a test. Tests are executed in-folder and expects a build.gradle file. If the latter is not found the test will not be executed.

For testing, a folder is created for each gradle version to be tested and the projects are duplicated below the verson folder. Each version testsute is executed within a separate JVM.

Test directory layout
${buildDir} / gradleTest / project1  / ver1   (1)
                                     / ver2   (2)
                         / project2  / ver1
                         / manifest  (3)
                         / src       (4)
                         / init.gradle (5)
1 A folder is created for each project
2 A folder for each Gradle version is created underneath the project folder
3 Classpath manifest used when running tests
4 Folder where generated source code ends up in
5 Initscript used by GradleTest
Test layout for one test
... / project1 / .gradle         (1)
               / build           (2)
               / build.gradle    (3)
               / settings.gradle (4)
1 Project cachedir is sent here
2 It is recommended that the build directory not be changed and left as per default
3 build.gradle is required for the test to be executed. It must contain a task called runGradleTest.
4 If a test project does not have a settings.gradle file an empty one will be generated in the test folder
Setting up your test project’s build.gradle
apply plugin : 'my.project.id'   (1)

task runGradleTest {
  dependsOn 'someTask'   (2)
1 Use apply plugin instead as it works across a larger range of Gradle versions. (No need for buildscript block as GradleTest will take care of injecting the plugin on the classpath).
2 Create a runGradleTest task and make it depend on the real task from your plugin that needs execution.

Gradle Entities

A GradleTest task called gradleTest will have the following associated entities:

Entity Type Description



For local available dependencies.



The actual GradleTest task type (extending Test task type).



Manages internal generated source code



Internal configuration for dependencies required to compile compatibility tests.



Internal configuration for dependencies required execute tests (different classpath to those passed to actual GradleRunner invocation).



Generates test code that will be compiled and used to actually executed GradleTestKit-based code



Compiles Groovy code generated by gradleTestGenerator



Associated classes task for compileGradleTestGroovy



Creates a classpath manifest file that will be used during test execution to load the correct plugin and transitive dependencies

Temporarily overriding Gradle versions

It is very convenient to sometimes to runs the test with a subset of the versions specified in the configuration. This is achievable by passing a system property to Gradle

-DgradleTest.versions=2.2,2.3 (1) (2)
1 If more than one GradleTest task is defined, replace gradleTest with the name of the appropriate GradleTest task.
2 List the versions in a commma-separated list. No validation is done on the string. If it leads to invalid Gradle versions the build will fail.

Task dependencies

gradleTest is linked into the check lifecycle task.

HTML Reports

HTML reports will appear as folder named after the GradleTest instance below the ${reporting.baseDir} folder. Practically in most cases this will be build/reports/gradleTest.

Add additional compatibility test sets

Use the additionalGradleTestSet project extension. Specify the basename and all of the appropriate configurations and tasks will be added.

    additionalGradleTestSet 'nonLegacyOnly' (1)

    nonLegacyOnlyGradleTest { (2)
        versions '4.0', '4.1', '4.2'
	    versions '3.5', '3.4.1', '3.3'
        versions '3.2', '3.1', '3.0' (3)
        versions '2.14.1'

    nonLegacyOnlyGradleTestGenerator { (4)
        pluginJarDirectory = gradleTestGenerator.pluginJarDirectory
    additionalGradleTestSet 'kotlin'

    kotlinGradleTest {
        versions '4.0', '4.1', '4.2'

    kotlinGradleTestGenerator {
        pluginJarDirectory = gradleTestGenerator.pluginJarDirectory
1 Supply basename to additionalGradleTestSet
2 This will result in a test task in the format ${baseName}GradleTest. Sources are expected to be layed out below the src/${baseName}GradleTest folder in a similar fashion to gradleTest
3 Configure test task as per normal.
4 Other tasks in the groups are all named using the same convention and can also just be configured as per normal.

More feedback

Running lots of compatibility tests can take some time. If you need more visual feedback regarding progress try adding something like this to the gradleTest configuration.

  beforeTest {
    println "  ${it.name}" (1)
1 Print the name of the current compatibility test that is runnign

Kotlin DSL

With the growing use of Kotlin-based build scripts, it is also important to test compatibility of a plugin for this context. Turn Kotlin-testing on by setting the following:

gradleTest {
  kotlinDsl = true

Now drop a build.gradle.kts file alongside your build.gradle build script. If the gradle version under test is 4.0 or later, GradleTest will also run a test using the Kotlin build script.

Command-line Awareness

GradleTest is aware of certain command-line parameters and will pass them on to running tests.

  • --rerun-tasks - All tests will be re-run even if they were previsouly successful.

  • --offline - No dependency checks will be performed and if plugins implements any offline behaviour this will automatically be enabled.

Advanced Features

Caching Dependencies

Although gradle tests can download their own dependencies, this might consume unnecessary bandwidth and waste a lot of testing time. In order to combat this, any dependencies listed under gradleTest configuration will be downloaded and made available to the running gradle tests.

Define dependencies in build.gradle
dependencies {
  gradleTest 'commons-cli:commons-cli:1.2'

These dependencies then appear as a flatDir repository in the gradle test.

NOTE: It is not necessary to add your plugin to the dependencies. The output of the jar task is automatically added to the gradleTest configuration.

Configure test build.gradle for dependency
buildscript {
  dependencies {
    classpath ':gnumake:1.0.1' (1)

dependencies {
  compile ':commons-cli:1.2' (2)
1 It is completely possible to add it to the buildscript for loading plugins
2 Load up any dependencies a per normal

NOTE: This repository is injected into the test using an internal initialisation script.

Overriding internal dependencies

GradleTest uses Spock Framework, JUnit & Apache Commons IO underneath. If, for some reason you need to override the versions of these dependencies it can be done in the gradleTestCompile configuration. If you created a new GradleTest task called foobar then the appropriate configuration will be called foobarCompile.

Base plugin

If the convention of adding a gradleTest with it’s default settings are not suitable for your environment, then the base plugin can be considered. This will only add the additionalGradleTestSet method. The GradleTest Base Plugin is available in the Gradle Plugins repository. To use it, add the following snippet into your build script.

plugins {
  id 'org.ysb33r.gradletest.base' version '1.1'

When your setup is more complex and the plugins block does not work OR if you are using Gradle 2.0 use the following instead

buildscript {
  repositories {
    maven {
      url "https://plugins.gradle.org/m2/"

  dependencies {
    classpath 'org.ysb33r.gradle:gradletest:1.1'

apply plugin: 'org.ysb33r.gradletest.base'

Known Limitations (Gradle 2.13+)

  • Not designed to test with Gradle < 2.0. If the community requires this functionality an effort will be made to see if it is possible. (It will actually try to runs tests against a Gradle version < 2.0, if configured in gradleTest by YMMV).

  • Runs in legacy mode when the project is built with Gradle 2.0 - 2.12.

Upgrading from v0.5.x

%%.VERSION%% is no longer supported when building with Gradle 2.13+ and builds will fail with Could not find foo:bar:%%VERSION%%` error if encountered. It will still be supported if using Gradle 2.12 or earlier as legacy mode will be invoked. Even under legacy mode it’s use is no longer required. Simply remove the classpath, and if you don’t anything extra in the buildscript block, the whole of the buildscript block.

For instance the following block can be complety removed:

buildscript {
    dependencies {
        classpath 'com.github.jruby-gradle:jruby-gradle-plugin:%%VERSION%%'

Compatibility with Other Plugins

License plugin

The License plugin is a really useful plugin to help with maintenance of license headers in source code. One standard feature of this plugin is to scan all source sets for license governance. Unfortunately this detects the source sets that are added by GradleTest itself which has its own internally generated license headers.

There is an open issue against this, but for now the best workaround is to exclude the test code generated by GradleTest.

Configuring licence plugin for exclusion
license {
  exclude '**/*CompatibilitySpec.groovy'

CodeNarc plugin

When adding GradleTest to a project that also has codenarc configured, the tests generated by GradleTest will be scanned by CodeNarc.

Configuring CodeNarc to ignore GradleTest
codenarcGradleTest.enabled = false

Alternative Approaches to GradleTest

Gradle TestKit

GradleTestKit is the core API that ships with Gradle since Gradle 2.6. which is meant for plugin developers to test compatibility across Gradle versions. Easrly versions could only tests against the same version of Gradle that was running, but it has been enhanced over different Gradle releases. Today it is powereful allowing multiple versions to be tested and a number of different environments to be set up.

It is also a core compoenent of GradleTest when the version of Gradle is 2.13 or later. (When the current build version is older GradleTest falls back to legacy mode, simply because the GradleTestKit API either is non-existent or not mature enough.

GradleTestKit works well for integration testing when writing plugins if a select small group of Gradle versions are used or even if it only targets the current Gradle version. In this way one can quickly determine issues in a plugin which cannot be discovered via normal testing. GradleTest used GradleTestKit directly for its integration tests for exactly this reason.

Testing multiple versions of Gradle with GradleTestKit soon bcomes a burden for developers as the API differs and classpath setups can be tricky. Older versions of Gradle give the most issues. In addition GradleTestKit is essentially a library it focuses on the developer rather than the build script user.

In this regard GradleTest steps in as it allow tests to be written as if a normal buildscript would be written. it also manages differences (or non-existance) in API from Gradle 2.0 onwards in a completely transparent way to the plugin author.

Overall the approach should be to write integration tests with GradleTestKit using a limited or even just one Gradle version. This is then complimented by some GradleTest scripts to test a wider range of Gradle versions.


Stutter is another library by Andy Oberstar to aid with plugin development. It focuses on the cases where you might have to run the same tests against all supported versions, but don’t want to deal with making each test loop over those versions (or @Unroll with a where block in Spock). Stutter generates one task per supported version for you and runs the full set of tests against it.

It definitely removes some of the complexity of setting up the GradleTestKit environment, but beware that it works from Gradle 2.13 onwards. The latter is not a restriction, but essentialy the same reason why GradleTest reverts to legacy mode if the build version of Gradle is older than 2.13: a lack of maturity in the GradleTestKit API in those releases.

It adds a compatTest source set, which will not clash with the gradleTest source set from GradleTest.

Stutter is essentialy developer-centric whereas GradleTest is script author-centric. This distinction is important as it clarifies different levels of validation:

  • If you want to create good samples and make sure they build, GradleTest is a clear win. Not having to write the extra test code (or touch the TestKit API) is a great benefit.

  • If you need to validate what the build did, you are back in GradleTestKit land and then Stutter will make it easier for you.

It can be an easy way to manage some more complex plugin integration tests. There is no reason to not use it alongside GradleTest as part of plugin development.

GradleTest Legacy Mode (Gradle 2.0-2.12)

If you are building your plugin project with any Gradle version 2.0 - 2.12 it automatically invoked legacy mode and behaves like the v0.5.5 version of this plugin.

Legacy mode will be removed in version 2.0 of the plugin.

Global Configuration

gradleLocations {
  searchGradleUserHome = true      (1)
  includeGradleHome = true         (2)
  searchGvm = true                 (3)
  download = true                  (4)
  downloadToGradleUserHome = false (5)
  useGradleSite = true             (6)
  uri                              (7)
  search                           (8)
1 Search Gradle home for Gradle installations.
2 Include the current GRADLE_HOME folder.
3 Search the GVM folder for Gradle installations
4 Download uninstalled version if online. These distributions will be placed in the appropriate location in gradle.gradleHome.
5 Place downloaded distributions in gradle.gradleUserHomeDir in the same way Gradle wrapper will do.
6 Download distribution from the global Gradle distribution site.
7 List additional URLs to search for Gradle distributions.
8 Search these additional folders for Gradle installations. Search will be performed both as if it a cache-style folder (aka gradleUserHome) or an installation-style folder (aka GVM_HOME).
If the above is used with Gradle 2.13+ it will be accepted and ignored. Deprecation warnings will also be printed.

Dynamic dependencies

Hard-coding the plugin version in to the build.gradle files of the gradleTest test fixtures is a maintenance pain. In previous versions of the plugin one had to dp to write something like

buildscript {
  dependencies {
    classpath ':gnumake:%%VERSION%%'

and the plugin would substitute the %%VERSION%% token with the version of your project. This is no longer necessary with this release even when running in legacy mode. The plugin will inject the path to the plugin automatically eliminating the need to use a block like the above at all.

Task dependencies

The gradleTest task is not linked to any other tasks. Run this as explicit task on the command-line or add your own task dependencies in your gradle script. The reason for this is that in legacy mode it can be quite a time-consuming testset to run. The typical case will be that the tests are only run close to release time. If you prefer you might want to set install.dependsOn gradleTest rather than check.dependsOn gradleTest or build.dependsOn gradleTest.

Test directory layout

For testing, a folder is created for each gradle version to be tested and the projects are duplicated below the verson folder. Each version’s testsuite is executed within a separate JVM.

Test directory layout
${buildDir} / gradleTest / ver1 / project1 (1)
                                / project2
                                / project3
                         / ver2 / project1
                                / project2
                                / project3
                         / init20.gradle   (2)
                         / repo            (3)
                         / home            (4)
            / classes / gradleTest         (5)
            / gradleDist                   (6)
1 Projects are duplicated for each Gradle version. See below for a more detailed layout
2 This is a generated initscript used to start all tests.
3 This is flat repository that can be utilised by all tests.
4 Gradle home directory for all the tests
5 Temporary classes for bootstrapping the tests are kept here.
6 Distributions are downloaded here if necessary
Test layout for one test
... / project1 / .gradle         (1)
               / src             (2)
               / build           (3)
               / build.gradle    (4)
               / settings.gradle (5)
1 Project cachedir is sent here
2 If the test project has any folders they will be copied here
3 It is recommended that the build directory not be changed and left as per default
4 build.gradle is required for the test to be executed. It must contain a task called runGradleTest.
5 If a test project does not have a settings.gradle file an empty one will be generated in the test folder

Adding additional GradleTest tasks

It is possible to add additional test tasks beyond gradleTest, by doing

configurations {

task furtherTest( type : org.ysb33r.gradle.gradletest.legacy20.GradleTest ) {
  versions '2.2'

Test files should be placed under src/furtherTest using the same layout as described earlier. Dependencies should be listed under furtherTest configuration.

Global configuration is still read from gradleLocations project extension.

Known Limitations (Gradle 2.0-2.12)

  • The plugin assumes that no Gradle distributions in gradle,gradleUserHomeDir or GVM_HOME will be removed whilst it is running.

  • The source sets for the Gradle tests cannot be renamed or added to. The subdirectory name is fixed to the task name.

  • No nice HTML report (https://github.com/ysb33r/gradleTest/issues/2)

  • No graceful failure as for test task. Currently throws a TaskExecutionException at the end, which is ugly. (https://github.com/ysb33r/gradleTest/issues/1)

  • No running counter of tests run and test failures (as for test task). (https://github.com/ysb33r/gradleTest/issues/3)

  • All test output is going to stdout instead of being captured and added to test report. (https://github.com/ysb33r/gradleTest/issues/4)

  • --no-daemon is set, as we don’t want to clash with existing running daemons.

  • Init scripts can no longer be passed. (This will probably be addressed in another release).

  • Does not run test in parallel, even though it theoretically could

Unresolved directive in product-documentation.adoc - include::parts/gradle-runner.adoc[]