GradleTest : A Gradle Compatibility Testing Plugin

Test your plugin against different versions of Gradle as part of your build. This plugin extends the power that the new GradleTestKit brings without having to actually author code. It rather allows plugins authors to write functional tests that looks like normal Gradle projects (multi-project is supported). This also makes it easier to create sample projects that can directly be used in documentation.

If you are using Gradle 6.0, you will see warnings. These are resolved in version 3.0 of this plugin.

If you are using a Gradle version < 4.1, you will need to use any 2.x version of this plugin.

If you are using a Gradle version < 3.0, you will need to use version 1.1 of this plugin.

Bootstrap

GradleTest is available in the Gradle Plugins repository. To use it, add the appropriate plugin to the plugins block.

plugins {
  id 'org.ysb33r.gradletest' version '3.0.0-alpha.2' (1)
  id 'org.ysb33r.gradletest.base' version '3.0.0-alpha.2' (2)

1	The common use case for compatibility testing
2	Use the base plugin when you do not require the default `gradleTest` tasks (and related source sets)

When your setup is more complex and the plugins block does not work, use the following instead

buildscript {
  repositories {
    maven {
      url "https://plugins.gradle.org/m2/"
    }
  }

  dependencies {
    classpath 'org.ysb33r.gradle:gradletest:3.0.0-alpha.2'
  }
}

apply plugin: 'org.ysb33r.gradletest'

Support this project

If this project is useful to you, you may want to consider donating to the associated Patreon account

Getting Started

In build.gradle define the versions of Gradle that needs to be tested:

gradleTest {
   versions '4.0', '5.4', '6.1'
}

Create a folder called src/gradleTest. Beneath that add a folder - call it 'myExample` for now. Within that create a normal Gradle project i.e. drop a build.gradle file and fill it out with the kind of syntax that a person will use to run your plugin. Finally add a custom task called runGradleTest and make it depend on some task that your plugin provides.

For example here is one example from GradleTest’s test suite of how itself is being tested.

plugins {
    id 'org.ysb33r.gradletest'
}

apply from: project.properties.OFFLINE_REPO_DSL

gradleTest {
    versions '6.7','4.1'
    expectFailure ~/expectedBuildFailure/
}

task runGradleTest( dependsOn : ['gradleTest'] ) {
}

Conventions

Filesystem

Source directory layout

${projectDir} / src / gradleTest /  project1  / build.gradle
                                              / testTwo.gradle
                                    project2  /
                                    project3  /

Each directory below gradleTest becomes a test. Tests are executed in-folder. The test folder can have more than one build file in it. Each build file will be considered an individual test that is executed independently of the others.

With multiple build files, its important to remember that GradleTest will reuse the existing ${buildDir}. So in the example above, if build.gradle happens to be executed first, then when testTwo.gradle is executed, it will reuse the build folder that was created by build.gradle.

If this is not desired, then add the clean task to all your build execution scripts like this:

Cleaning before multi-test

apply plugin : 'my.project.id'

task runGradleTest {
  dependsOn 'clean', 'someTask'
}

For testing, a folder is created for each gradle version to be tested and the projects are duplicated below the verson folder. Each version testsute is executed within a separate JVM.

Test directory layout

${buildDir} / gradleTest / project1  / ver1   (1)
                                     / ver2   (2)
                         / project2  / ver1
                         / manifest  (3)
                         / src       (4)
                         / init.gradle (5)

1	A folder is created for each project
2	A folder for each Gradle version is created underneath the project folder
3	Classpath manifest used when running tests
4	Folder where generated source code ends up in
5	Initscript used by GradleTest

Test layout for one test

... / project1 / .gradle         (1)
               / build           (2)
               / build.gradle    (3)
               / settings.gradle (4)

1	Project cachedir is sent here
2	It is recommended that the build directory not be changed and left as per default
3	`build.gradle` is required for the test to be executed. It must contain a task called `runGradleTest`.
4	If a test project does not have a `settings.gradle file` an empty one will be generated in the test folder

Setting up your test project’s build.gradle

apply plugin : 'my.project.id'   (1)

task runGradleTest {
  dependsOn 'someTask'   (2)
}

1	Use `apply plugin` instead as it works across a larger range of Gradle versions. (No need for `buildscript` block as GradleTest will take care of injecting the plugin on the classpath).
2	Create a `runGradleTest` task and make it depend on the real task from your plugin that needs execution.

Gradle Entities

A GradleTest task called gradleTest will have the following associated entities:

Entity Type Description

Entity	Type	Description
`gradleTest`	Configuration	For making additional dependencies available to the manifest that will be passed to the underlying `GradleRunner` instance.
`gradleTest`	Task	The actual `GradleTest` task type (extending `Test` task type).
`gradleTest`	SourceSet	Manages internal generated source code
`gradleTestCompile`	Configuration	Internal configuration for dependencies required to compile compatibility tests.
`gradleTestRuntime`	Configuration	Internal configuration for dependencies required execute tests (different classpath to those passed to actual `GradleRunner` invocation).
`gradleTestGenerator`	Task	Generates test code that will be compiled and used to actually executed `GradleTestKit`-based code
`compileGradleTestGroovy`	Task	Compiles Groovy code generated by `gradleTestGenerator`
`gradleTestClasses`	Task	Associated `classes` task for `compileGradleTestGroovy`
`gradleTestClasspathManifest`	Task	Creates a classpath manifest file that will be used during test execution to load the correct plugin and transitive dependencies

gradleTest

Configuration

For making additional dependencies available to the manifest that will be passed to the underlying GradleRunner instance.

gradleTest

Task

The actual GradleTest task type (extending Test task type).

gradleTest

SourceSet

Manages internal generated source code

gradleTestCompile

Configuration

Internal configuration for dependencies required to compile compatibility tests.

gradleTestRuntime

Configuration

Internal configuration for dependencies required execute tests (different classpath to those passed to actual GradleRunner invocation).

gradleTestGenerator

Task

Generates test code that will be compiled and used to actually executed GradleTestKit-based code

compileGradleTestGroovy

Task

Compiles Groovy code generated by gradleTestGenerator

gradleTestClasses

Task

Associated classes task for compileGradleTestGroovy

gradleTestClasspathManifest

Task

Creates a classpath manifest file that will be used during test execution to load the correct plugin and transitive dependencies

Task dependencies

gradleTest is linked into the check lifecycle task.

HTML reports

HTML reports will appear as folder named after the GradleTest instance below the ${reporting.baseDir} folder. Practically in most cases this will be build/reports/gradleTest.

Generated source

Tests are generated into .generated-src/gradleTestPlugin directory. Add .generated-src to your .gitignore file. If you are using IntelliJ, mark .generated-src/gradlteTestPlugin/gradleTest/src as a test source, so that you can debug dreictly from within the IDE.

Command-line Awareness

Gradle standard options

GradleTest is aware of certain command-line parameters and will pass them on to running tests.

--rerun-tasks - All tests will be re-run even if they were previsouly successful.
--offline - No dependency checks will be performed and if plugins implements any offline behaviour this will automatically be enabled.

Temporarily overriding Gradle versions

It is very convenient to sometimes to runs the test with a subset of the versions specified in the configuration. This is achievable by passing a system property to Gradle

-DgradleTest.versions=3.5,4.4.1 (1) (2)

1	If more than one GradleTest task is defined, replace `gradleTest` with the name of the appropriate GradleTest task.
2	List the versions in a commma-separated list. No validation is done on the string. If it leads to invalid Gradle versions the build will fail.

Selecting what tests to run

GradleTest now purely uses the --tests command-line parameter, and no longer the -DgradleTest.include and -DgradleTest.exclude system properties. It slightly manipulates this for you as you don;t have to remember the name of the generated class file, you simply have to provide the name or parital name of the folder below your src/gradleTest directory.

So take this folder structure:

src/gradleTest/mytest
    build.gradle
src/gradleTest/theirtest
    build.gradle

It are these resulting files that you filter what tests you want to run. The first way is to specify a single file

./gradlew gradleTest --tests mytest

If you do

./gradlew gradleTest --tests '*test'

it will run all tests where the folder names contain test.

Beyond the Basics

Base plugin

If the convention of adding a gradleTest with it’s default settings are not suitable for your environment, then the base plugin can be considered. This will only add the additionalGradleTestSet method. The GradleTest Base Plugin is available in the Gradle Plugins repository. To use it, add the following snippet into your build script.

plugins {
  id 'org.ysb33r.gradletest.base' version '3.0.0-alpha.2'
}

When your setup is more complex and the plugins block does not work use the following instead

buildscript {
  repositories {
    maven {
      url "https://plugins.gradle.org/m2/"
    }
  }

  dependencies {
    classpath 'org.ysb33r.gradle:gradletest:3.0.0-alpha.2'
  }
}

apply plugin: 'org.ysb33r.gradletest.base'

Add additional compatibility test sets

Use the additionalGradleTestSet project extension. Specify the basename and all of the appropriate configurations and tasks will be added.

gradleTestSets.add 'kotlin' (1)

kotlinGradleTest { (2)
    versions '4.10.2' (3)
    versions '5.0', '5.4.1', '5.6.4'
    versions '6.0.1', '6.5.1', '6.9'
    versions '7.0'
}

kotlinGradleTestGenerator { (4)
    pluginJarDirectory = gradleTestGenerator.pluginJarDirectory
}

1	Supply basename to `additionalGradleTestSet`
2	This will result in a test task in the format `${baseName}GradleTest`. Sources are expected to be layed out below the `src/${baseName}GradleTest` folder in a similar fashion to `gradleTest`
3	Configure test task as per normal.
4	Other tasks in the groups are all named using the same convention and can also just be configured as per normal.

Provide more feedback during test execution

Running lots of compatibility tests can take some time. If you need more visual feedback regarding progress try adding something like this to the gradleTest configuration.

gradleTest{
  beforeTest {
    println "  ${it.name}" (1)
  }
}

1	Print the name of the current compatibility test that is runnign

Kotlin DSL

With the growing use of Kotlin-based build scripts, it is also important to test compatibility of a plugin for this context. Turn Kotlin-testing on by setting the following:

gradleTest {
  kotlinDsl = true
}

Now drop a build.gradle.kts file alongside your build.gradle build script. If the Gradle version under test is 4.0 or later, GradleTest will also run a test using the Kotlin build script.

Clean Gradle Project Cache

Every test run has an individual Gradle project cache. This is cleared by default. if you want to maintain this between runs, set cleanCahcne = false

Advanced Features

Caching Dependencies

Although gradle tests can download their own dependencies, this might consume unnecessary bandwidth and waste a lot of testing time. In order to combat this, any dependencies listed under gradleTest configuration will be downloaded and made available to the running gradle tests.

Define dependencies in build.gradle

dependencies {
  gradleTest 'commons-cli:commons-cli:1.2'
}

These dependencies then appear as a flatDir repository in the gradle test.

NOTE: It is not necessary to add your plugin to the dependencies. The output of the jar task is automatically added to the gradleTest configuration.

Configure test build.gradle for dependency

buildscript {
  dependencies {
    classpath ':gnumake:1.0.1' (1)
  }
}

dependencies {
  compile ':commons-cli:1.2' (2)
}

1	It is completely possible to add it to the `buildscript` for loading plugins
2	Load up any dependencies a per normal

NOTE: This repository is injected into the test using an internal initialisation script.

Overriding internal dependencies

GradleTest uses Spock Framework, JUnit & Apache Commons IO underneath. If, for some reason you need to override the versions of these dependencies it can be done in the gradleTestCompile configuration. If you created a new GradleTest task called foobar then the appropriate configuration will be called foobarCompile.

Changing the Gradle distribution name

Enterprise environments may have their own custom Gradle versions, which are not named in exactly the same way as the standard Gradle versions. In such cases it is possible to override the naming pattern by using gradleDistributionFilenamePattern DSL keyword.

gradleTest {
  gradleDistributionFilenamePattern '/acme-gradle-@version@-bin.zip' (1)
}

1	Change file naming pattern. `@version@` will be substituted with the appropriate Gradle version at test time.

Analysing Testkit data

By default any any data created by TestKit is discarded at the end of the test. Sometimes non-trivial plugins needs a detailed diagnosis and access to the TestKit data can be very useful. Data can be shared across a full test run or be preserved on a per test basis.

gradleTest {
   testKitStrategy = discardData (1)
   testKitStrategy = directoryPerGroup (2)
   testKitStrategy = directoryPerTest (3)
}

1	Do not keep the data. This is the default.
2	Share the data across the whole testset. Data will appear in `${buildDir}/gradleTest/testkit-data`.
3	Keep the data for each test separate. This will result ina whole lot more disk space and test will probably run slower, but this setting allows for a very detailed analysis. Data will appear in `${buildDir}/gradleTest/${testName}/${gradleVersion}/testkit-data'.

Upgrading From Older Releases

v2.0

Sources are now genreated into the ./.generated-src/gradleTestPlugin directory instead of build/gradleTest/src. This si so that IntelliJ can correctly identify the test source code.

v1.x

Any version of Gradle 2.x listed in gradleTest.versions will be ignored during testing. A warning message will be printed in this case.

v0.5.x

%%.VERSION%% is no longer supported when building with Gradle 2.13+ and builds will fail with Could not find foo:bar:%%VERSION%%` error if encountered. Simply remove the classpath, and if you don’t anything extra in the buildscript block, the whole of the buildscript block.

For instance the following block can be completely removed:

buildscript {
    dependencies {
        classpath 'com.github.jruby-gradle:jruby-gradle-plugin:%%VERSION%%'
    }
}

Compatibility with Other Plugins

License plugin

The License plugin is a really useful plugin to help with maintenance of license headers in source code. One standard feature of this plugin is to scan all source sets for license governance. Unfortunately this detects the source sets that are added by GradleTest itself which has its own internally generated license headers.

There is an open issue against this, but for now the best workaround is to exclude the test code generated by GradleTest.

Configuring licence plugin for exclusion

license {
  exclude '**/*.dsl.groovySpec.groovy'
  exclude '**/*.dsl.kotlinSpec.groovy'
}

CodeNarc plugin

When adding GradleTest to a project that also has codenarc configured, the tests generated by GradleTest will be scanned by CodeNarc.

Configuring CodeNarc to ignore GradleTest

codenarcGradleTest.enabled = false

Jacoco plugin

When the Jacoco plugin is added, coverage will be added from all GradleTest tasks. The debug flag on GradleTest tasks are also set to`true` on order to enable this.

Alternative Approaches to GradleTest

Gradle TestKit

GradleTestKit is the core API that ships with Gradle since Gradle 2.6. which is meant for plugin developers to test compatibility across Gradle versions. Easrly versions could only tests against the same version of Gradle that was running, but it has been enhanced over different Gradle releases. Today it is powereful allowing multiple versions to be tested and a number of different environments to be set up.

It is also a core compoenent of GradleTest when the version of Gradle is 2.13 or later. (When the current build version is older GradleTest falls back to legacy mode, simply because the GradleTestKit API either is non-existent or not mature enough.

GradleTestKit works well for integration testing when writing plugins if a select small group of Gradle versions are used or even if it only targets the current Gradle version. In this way one can quickly determine issues in a plugin which cannot be discovered via normal testing. GradleTest used GradleTestKit directly for its integration tests for exactly this reason.

Testing multiple versions of Gradle with GradleTestKit soon bcomes a burden for developers as the API differs and classpath setups can be tricky. Older versions of Gradle give the most issues. In addition GradleTestKit is essentially a library it focuses on the developer rather than the build script user.

In this regard GradleTest steps in as it allow tests to be written as if a normal buildscript would be written. it also manages differences (or non-existance) in API from Gradle 2.0 onwards in a completely transparent way to the plugin author.

Overall the approach should be to write integration tests with GradleTestKit using a limited or even just one Gradle version. This is then complimented by some GradleTest scripts to test a wider range of Gradle versions.

Stutter

Stutter is another library by Andy Oberstar to aid with plugin development. It focuses on the cases where you might have to run the same tests against all supported versions, but don’t want to deal with making each test loop over those versions (or @Unroll with a where block in Spock). Stutter generates one task per supported version for you and runs the full set of tests against it. It definitely removes some of the complexity of setting up the GradleTestKit environment.

It adds a compatTest source set, which will not clash with the gradleTest source set from GradleTest.

Stutter is essentialy developer-centric whereas GradleTest is script author-centric. This distinction is important as it clarifies different levels of validation:

If you want to create good samples and make sure they build, GradleTest is a clear win. Not having to write the extra test code (or touch the TestKit API) is a great benefit.
If you need to validate what the build did, you are back in GradleTestKit land and then Stutter will make it easier for you.

It can be an easy way to manage some more complex plugin integration tests. There is no reason to not use it alongside GradleTest as part of plugin development.

Awesomeness

This plugin is so awesome, it applies to itself and then runs a collection of tests.

In v0.5.5 this was handled through the self-referencing plugin recipe - See gradle/self-reference.gradle in an old codebase.

Since 1.0, the complexities required more than the above could handle, and the same is now achieved via a GradleBuild task. See gradle/compatibility-tests.gradle in an up to date codebase on how this is done.