Concourse-Up has been replaced with Control Tower. First-time users should deploy using
control-tower and raise issues under that project.
A tool for easily deploying Concourse in a single command.
$ AWS_ACCESS_KEY_ID=<access-key-id> \ AWS_SECRET_ACCESS_KEY=<secret-access-key> \ concourse-up deploy <your-project-name>
$ GOOGLE_APPLICATION_CREDENTIALS=<path/to/googlecreds.json> \ concourse-up deploy --iaas gcp <your-project-name>
The goal of Concourse-Up is to be the world's easiest way to deploy and operate Concourse CI in production.
In just one command you can deploy a new Concourse environment for your team, on either AWS or GCP. Your Concourse-Up deployment will upgrade itself and self-heal, restoring the underlying VMs if needed. Using the same command-line tool you can do things like manage DNS, scale your environment, or manage firewall policy. CredHub is provided for secrets management and Grafana for viewing your Concourse metrics.
You can keep up to date on Concourse-Up announcements by reading the EngineerBetter Blog
|Concourse IP whitelisting||+||+|
|Custom tagging||BOSH only||BOSH only|
|Custom TLS certificates||+||+|
|Database vertical scaling||+||+|
|Interruptable worker support||+||+|
|Retrieving deployment information||+||+|
|Retrieving deployment information as shell exports||+||+|
|Retrieving deployment information in JSON||+||+|
|Retrieving director NATS cert expiration||+||+|
|Rotating director NATS cert||+||+|
|Web server vertical scaling||+||+|
|Worker horizontal scaling||+||+|
|Worker type selection||+||N/A|
|Worker vertical scaling||+||+|
GOOGLE_APPLICATION_CREDENTIALS_CONTENTSset to the path to a GCP credentials json file
Download the latest release and install it into your
--region valueAWS or GCP region (default: "eu-west-1" on AWS and "europe-west1" on GCP) [$AWS_REGION]
--namespace valueAny valid string that provides a meaningful namespace of the deployment - Used as part of the configuration bucket name [$NAMESPACE].
Note that if namespace has been provided in the initial
deployit will be required for any subsequent
concourse-upcalls against the same deployment.
The default IAAS for Concourse-Up is AWS. To choose a different IAAS use the
--iaas flag. For every IAAS provider apart from AWS this flag is required for all commands.
Supported IAAS values: AWS, GCP
--iaas value(optional) IAAS, can be AWS or GCP (default: "AWS") [$IAAS]
Deploy a new Concourse with:
concourse-up deploy <your-project-name>
$ concourse-up deploy ci ... DEPLOY SUCCESSFUL. Log in with: fly --target ci login --insecure --concourse-url https://10.0.0.0 --username --password Metrics available at https://10.0.0.0:3000 using the same username and password Log into credhub with: eval "$(concourse-up info ci --env)"
A new deploy from scratch takes approximately 20 minutes.
All flags are optional. Configuration settings provided via flags will persist in later deployments unless explicitly overriden.
--domain value Domain to use as endpoint for Concourse web interface (eg: ci.myproject.com) [$DOMAIN]
$ concourse-up deploy --domain chimichanga.engineerbetter.com chimichanga
In the example above
concourse-up will search for a hosted zone that matches
engineerbetter.com and add a record to the longest match (
chimichanga.engineerbetter.com in this example).
--tls-cert value TLS cert to use with Concourse endpoint [$TLS_CERT]
--tls-key value TLS private key to use with Concourse endpoint [$TLS_KEY]
concourse-up will generate a self-signed cert using the given domain. If you'd like to provide your own certificate instead, pass the cert and private key as strings using the
--tls-key flags respectively. eg:
$ concourse-up deploy \ --domain chimichanga.engineerbetter.com \ --tls-cert "$(cat chimichanga.engineerbetter.com.crt)" \ --tls-key "$(cat chimichanga.engineerbetter.com.key)" \ chimichanga
--workers value Number of Concourse worker instances to deploy (default: 1) [$WORKERS]
--worker-type Specify a worker type for aws (m5 or m4) (default: "m4") [$WORKER_TYPE] (see comparison table below). Note: this is an AWS-specific option
AWS does not offer m5 instances in all regions, and even for regions that do offer m5 instances, not all zones within that region may offer them. To complicate matters further, each AWS account is assigned AWS zones at random - for instance,
eu-west-1afor one account may be the same as
eu-west-1bin another account. If m5s are available in your chosen region but not the zone Concourse-Up has chosen, create a new deployment, this time specifying another
--worker-size value Size of Concourse workers. Can be medium, large, xlarge, 2xlarge, 4xlarge, 10xlarge, 12xlarge, 16xlarge or 24xlarge depending on the worker-type (see above) (default: "xlarge") [$WORKER_SIZE]
|--worker-size||AWS m4 Instance type||AWS m5 Instance type*||GCP Instance type|
* m5 instances not available in all regions and all zones. See
--worker-type for more info.
--web-size value Size of Concourse web node. Can be small, medium, large, xlarge, 2xlarge (default: "small") [$WEB_SIZE]
|--web-size||AWS Instance type||GCP Instance type|
--db-size value Size of Concourse Postgres instance. Can be small, medium, large, xlarge, 2xlarge, or 4xlarge (default: "small") [$DB_SIZE]
Note that when changing the database size on an existing concourse-up deployment, the SQL instance will scaled by terraform resulting in approximately 3 minutes of downtime.
The following table shows the allowed database sizes and the corresponding AWS RDS & CloudSQL instance types
|--db-size||AWS Instance type||GCP Instance type|
--allow-ips value Comma separated list of IP addresses or CIDR ranges to allow access to (default: "0.0.0.0/0") [$ALLOW_IPS]
allow-ipsgoverns what can access Concourse but not what can access the control plane (i.e. the BOSH director).
--github-auth-client-id value Client ID for a github OAuth application - Used for Github Auth [$GITHUB_AUTH_CLIENT_ID]
--github-auth-client-secret value Client Secret for a github OAuth application - Used for Github Auth [$GITHUB_AUTH_CLIENT_SECRET]
--add-tag key=value Add a tag to the VMs that form your
concourse-up deployment. Can be used multiple times in a single
--spot=value Use spot instances for workers. Can be true/false. Default is true.
Concourse Up uses spot instances for workers as a cost saving measure. Users requiring lower risk may switch this feature off by setting --spot=false.
--preemptible=value Use preemptible instances for workers. Can be true/false. Default is true.
Be aware the preemptible instances will go down at least once every 24 hours so deployments with only one worker will experience downtime with this feature enabled. BOSH will ressurect falled workers automatically.
preemptible are interchangeable so if either of them is set to false then interruptible instances will not be used regardless of your IaaS. i.e:
# Results in an AWS deployment using non-spot workers concourse-up deploy --spot=true --preemptible=false <your-project-name> # Results in an AWS deployment using non-spot workers concourse-up deploy --preemptible=false <your-project-name> # Results in a GCP deployment using non-preemptible workers concourse-up deploy --iaas gcp --spot=false <your-project-name>
--zone Specify an availability zone [$ZONE] (cannot be changed after the initial deployment)
If any of the following 5 flags is set, all the required ones from this group need to be set
--vpc-network-range value Customise the VPC network CIDR to deploy into (required for AWS) [$VPC_NETWORK_RANGE]
--public-subnet-range value Customise public network CIDR (if IAAS is AWS must be within --vpc-network-range) (required) [$PUBLIC_SUBNET_RANGE]
--private-subnet-range value Customise private network CIDR (if IAAS is AWS must be within --vpc-network-range) (required) [$PRIVATE_SUBNET_RANGE]
--rds-subnet-range1 value Customise first rds network CIDR (must be within --vpc-network-range) (required for AWS) [$RDS_SUBNET_RANGE1]
--rds-subnet-range2 value Customise second rds network CIDR (must be within --vpc-network-range) (required for AWS) [$RDS_SUBNET_RANGE2]
All the ranges above should be in the CIDR format of IPv4/Mask. The sizes can vary as long as
vpc-network-rangeis big enough to contain all others (in case IAAS is AWS). The smallest CIDR for
privatesubnets is a /28. The smallest CIDR for
rds2subnets is a /29
To fetch information about your
$ concourse-up info --json <your-project-name>
To load credentials into your environment from your
$ eval "$(concourse-up info --env <your-project-name>)"
To check the expiry of the BOSH Director's NATS CA certificate:
$ concourse-up info --cert-expiry <your-project-name>
Warning: if your deployment is approaching a year old, it may stop working due to expired certificates. For information please see this issue https://github.com/EngineerBetter/concourse-up/issues/81.
All flags are optional
--json Output as json [$JSON]
--env Output environment variables
--cert-expiry Output the expiry of the BOSH director's NATS certificate
To destroy your Concourse:
$ concourse-up destroy <your-project-name>
Handles maintenance operations in concourse-up
All flags are optional
--renew-nats-cert Rotate the NATS certificate on the director
Note that the NATS certificate is hardcoded to expire after 1 year. This command follows the istructions on bosh.io to rotate this certificate. This operation will cause downtime on your Concourse as it performs multiple full recreates.
--stage value Specify a specific stage at which to start the NATS certificate renewal process. If not specified, the stage will be determined automatically. See the following table for details.
|0||Adding new CA (create-env)|
|1||Recreating VMs for the first time (recreate)|
|2||Removing old CA (create-env)|
|3||Recreating VMs for the second time (recreate)|
|4||Cleaning up director-creds.yml|
When Concourse-up deploys Concourse, it now adds a pipeline to the new Concourse called
concourse-up-self-update. This pipeline continuously monitors our Github repo for new releases and updates Concourse in place whenever a new version of Concourse-up comes out.
This pipeline is paused by default, so just unpause it in the UI to enable the feature.
Patch releases of
concourse-up are compiled, tested and released automatically whenever a new stemcell or component release appears on bosh.io.
To upgrade your Concourse, grab the latest release and run
concourse-up deploy <your-project-name> again.
Concourse-up now automatically deploys Influxdb, Riemann, and Grafana on the web node. You can access Grafana on port 3000 of your regular concourse URL using the same username and password as your Concourse admin user. We put in a default dashboard that tracks
Concourse-up deploys the credhub service alongside Concourse and configures Concourse to use it. More detail on how credhub integrates with Concourse can be found here. You can log into credhub by running
$ eval "$(concourse-up info --env --region $region $deployment)".
Concourse-up normally allows incoming traffic from any address to reach your web node. You can use the
--allow-ips flag to add firewall rules to prevent this.
For example to deploy Concourse-up and only allow traffic from your local machine, you could use the command
concourse-up deploy --allow-ips $(dig +short myip.opendns.com @resolver1.opendns.com).
--allow-ips takes a comma seperated list of IP addresses or CIDR ranges.
concourse-up deploys to the AWS eu-west-1 (Ireland) region or the GCP europe-west1 (Belgium) region, and uses spot instances for large and xlarge Concourse VMs. The estimated monthly cost is as follows:
|gp2 storage||20GB (bosh, web)||2||4.40|
|gp2 storage||200GB (worker)||1||22.00|
|disk storage||20GB (bosh, web) + 200GB (worker)||-||40.80|
concourse-up first creates an S3 or GCS bucket to store its own configuration and saves a
config.json file there.
It then uses Terraform to deploy the following infrastructure:
Once the terraform step is complete,
concourse-up deploys a BOSH director on an t2.small/n1-standard-1 instance, and then uses that to deploy a Concourse with the following settings:
If you'd like to run concourse-up with it's own IAM account, create a user with the following permissions:
A IAM Primitive role of
roles/owner for the target GCP Project is required
CI Pipeline (deployed with Concourse Up!)
To build and test you'll need:
concourse-up uses golang compile-time variables to set the release versions it uses. To build locally use the
build_local.sh script, rather than running
You will also need to clone
concourse-up-ops to the same level as
concourse-up to get the manifest and ops files necessary for building. Check the latest release of
concourse-up for the appropriate tag of
Tests use the Ginkgo Go testing framework. The tests require you to have set up AWS authentication locally.
Install ginkgo and run the tests with:
go get github.com/onsi/ginkgo/ginkgo ginkgo -r
$ go get github.com/onsi/ginkgo/ginkgo $ ginkgo -r
Go linting, shell linting, and unit tests can be run together in the same docker image CI uses with
./run_tests_local.sh. This should be done before committing or raising a PR.
The pipeline listens for new patch or minor versions of
ops/versions.json coming from the
concourse-up-ops repo. In order to pick up a new major version first make sure it exists in the repo then modify
tag_filter: X.*.* in the
concourse-up-ops resource where
X is the major version you want to pin to.