This is an example of how to manage building projects inside monorepo with Gradle as build tool and one of the following services as CI tool:
When I push some changes to monorepository I want to
There is only one main job called build started automatically on every push. This job is responsible for triggering another jobs for each affected project in order with respecting project dependencies.
Build job is running until all triggered jobs are finished.
Build job is successful only when there were no failed jobs (even when there were no jobs).
There is file
tools/ci/projects.txt which contains lines with glob patterns pointing to root directories of all supported projects.
Jobs are defined in default location depending on which CI tool you are using.
Currently there is a convention used for mapping project to CI job. Job name is resolved from project's directory path as last path component.
e.g. project under directory
apps/serveris built by job
Dependencies are based on Gradle's composite build feature. To define dependency between projects use
includeBuild function in project build script (usually in
If you are not using Gradle or you want to add some additional dependency you can specify it in dependency file. Default location is in each project directory on path
.ci/dependencies.txt. Location can be changed by setting environment variable
CI_DEPENDENCIES_FILE. File should contain lines with paths to other projects (relative paths to monorepo root, e.g. libs/common).
To respect dependencies between projects jobs are triggered in multiple rounds. For each round one or more jobs are triggered and only when all jobs are successfully finished next round is processed. Even if there is only one failed job all next rounds are skipped and whole build is failed.
Commit message can contain some special words which if found can modify default building behavior.
Whole logic is implemented as bunch of Bash scripts under directory
tools/ci. Every script can be run directly and it should output some documentation when it requires some parameters.
Implementation is split to core logic and plugins for different CI tools. Core logic can be found under
tools/ci/core and plugins are under
Main script is
tools/ci/core/build.sh and it is only thing started from build job.
There is tool called jq used for JSON parsing.
You need to care only when you want run it locally because jq is usually part of default images used in CI tools.
It is possible to run
tools/ci/core/build.sh locally but there is need to provide few environment variables depending on CI tool used.
CI_TOOL=circleci \ CIRCLE_API_USER_TOKEN=XXX \ CIRCLE_PROJECT_USERNAME=zladovan \ CIRCLE_PROJECT_REPONAME=monorepo \ CIRCLE_BRANCH=master \ CIRCLE_SHA1=$(git rev-parse HEAD) \ tools/ci/core/build.sh
Note that this command could trigger some jobs in circleci
CI_TOOL=bitbucket \ BITBUCKET_USER=zladovan \ BITBUCKET_PASSWORD=xxx \ BITBUCKET_REPO_FULL_NAME=zladovan/monorepo \ BITBUCKET_BRANCH=master \ BITBUCKET_COMMIT=$(git rev-parse HEAD) \ tools/ci/core/build.sh
Note that this command could trigger some jobs in bitbucket
CI_TOOL=travis \ TRAVIS_TOKEN=xxx \ TRAVIS_REPO_SLUG=zladovan/monorepo \ TRAVIS_BRANCH=master \ TRAVIS_COMMIT=$(git rev-parse HEAD) \ tools/ci/core/build.sh
Note that this command could trigger some jobs in travis
CI_TOOL=github \ GITHUB_REPOSITORY=zladovan/monorepo \ GITHUB_TOKEN=xxx \ GITHUB_REF=refs/head/master \ GITHUB_SHA=$(git rev-parse HEAD) \ tools/ci/core/build.sh
Note that this command could trigger some jobs in github
Folder structure used in this repository is only an example of how it can look like. It is possible to use any structure, there is only need to use different patterns in
apps/ └── stand-alone runnable and deployable applications libs/ └── reusable libraries (used in apps dependencies) tools/gradle-plugins/ └── reusable gradle logic (used in apps and libs builds) tools/ci └── ci scripts