8 Pro Tips for Using Concourse CI with Cloud Foundry

by Alexander Lomov and Volha KurylionakMay 11, 2016

Learn how Concourse CI differs from other similar products, how it works, and how it can help you to "nitrocharge" your own projects.

Table of Contents

Why Concourse CI is different

concourse-ci-architecture-features-and-usage-logo

There are at least a dozen continuous integration solutions out there, but the majority of them are challenging to learn, scale, and debug. Most importantly, few of them were designed with pipelines in mind. Not able to find a tool that could handle something as massive and complex as Cloud Foundry, Pivotal decided to write their own. This resulted in Concourse CI, a user-friendly continuous delivery technology that focuses on pipelines, is easy to scale, runs anywhere, and works with anything.

Here are some of the features that really make Concourse CI stand out:

Pipelines are first-class citizens. Concourse CI is the first open-source tool built around pipelines, such as Atlassian Bamboo. Unlike it, Concourse has no proprietary license and does not lock you on any technologies.
Containers and microservices. Everything runs in ephemeral, reproducible containers. Horizontal scaling is a wheeze. Each part of the pipeline can be scaled independently.
Everything as Code. Deployments, builds, and anything can be automated, scripted, and then recreated without wizards or manual configuration.
Infrastructure-agnostic. Concourse CI is deployed with BOSH and can run on any cloud with BOSH support (AWS, GCE, vSphere, etc.), as well as on bare metal.
Written in Golang. It is possible to test software on various platforms, running workers on Linux, Windows, or MacOS (like it is done for Fly CLI).
Built for Cloud Foundry with love. Concourse CI was created to streamline development of the Cloud Foundry PaaS and can be readily integrated with BOSH.
Other perks. Concourse is packed with features that facilitate building and debugging CD pipelines (e.g., pipeline visualization, user-friendly text formatting, etc.).

For a detailed introduction with demos, watch this talk by Alex Suraci, one of Concourse CI’s main contributors, from the Cloud Foundry Summit 2015.

Main concepts and architecture

Concourse CI strives for simplicity in everything. There are only three core concepts:

Resources. Anything one can publish to or pull from and anything that can be versioned (e.g., GitHub, AWS S3 buckets, Docker images, BOSH deployments / releases / stemcells, etc.). Each resource is a container with scripts that can interact with external objects, such as S3.
Tasks. Instructions on how something should be run (container properties, dependencies, etc.)
Jobs. These combine resources and tasks, which results in something being done.

Resources, tasks, and jobs make up pipelines where each part is relatively independent and can be reused, even for another project. Visualization makes it easy to follow the stages that the artifacts go through inside the pipeline.

A distinct feature is that both resources and jobs are run in containers. Resources may be written in any language, which makes it simpler to integrate Concourse CI with other tools. As a bonus, you can quickly “SSH” into any container for debugging with the Fly CLI command. This beauty is brought to Concourse by Garden, a reimplementation of Cloud Foundry’s container manager—Warden—in Golang.

The architecture of Concourse CI is very straightforward, with cool aircraft-inspired naming. At the heart of it, lies Air Traffic Control (ATC), which provides a REST API and web GUI, processes all logic, and manages all workflows. It works closely with Transport Security Agency (TSA), which exposes an interface to workers, allowing them to register with ATC.

The Concourse CI architecture

Workers consist of several components, the main one being Garden. To provide Garden containers with a remote layered file system, the baggageclaim service is used. Workers are also serviced by groundcrew, which registers them with ATC at certain intervals, advertises supported resource types, and cleans up ephemeral containers on shutdown.

A detailed description of Concourse CI’s architecture can be found in the official documentation.

Getting started with Concourse CI

Concourse CI can be deployed with BOSH, via standalone binaries, or in a Vagrant box. Each way produces a fully functional deployment, the only difference being scalability and how fast you can get it working. Using a Vagrant box and the concourse-lite project (similar to bosh-lite), you can get Concourse CI on AWS or a local machine in minutes. Updates for Vagrant come very frequently, thanks to the Concourse CI team. For more on how to use Concourse, check out this tutorial.

Pro tips

Once you have a working Concourse CI deployment, it is trivial to build continuous delivery for projects of any size. Still, there are some things that you should keep in mind to avoid running into issues down the road. Here is a selection of tips based on our experience of working with various Concourse CI clusters. They will help to get your continuous delivery right from the first try:

Follow best practices. Every week, a question about best practices is raised in the Concourse Slack channel, and Alex Suraci answers it, using examples from pipelines for Concourse CI. This is where you can learn that the common practice is to have a ci folder in the project repo. The ci repo should contain folders called tasks, docker, and pipelines for tasks, Docker files, and pipelines respectively. If there is only one pipeline, it is better to save it to the pipeline.yml file directly under the ci folder. It is also preferable to have a separate directory for scripts and assets. The screenshot below shows an example of the approximate structure, which you can generate, for instance, with our concourse-helpers gem, which also creates your pipeline to test the framework you are using.
For more ideas on how to arrange continuous integration pipelines, read the sources of Diego, buildpacks, and Garden Linux.
Use CPU-intensive VMs for Concourse CI workers. Since running a lot of containers on one VM is a CPU-intensive task, it is a good idea to have fewer workers and a larger number of CPUs per machine (see this Slack discussion). If you work with physical machines, it is best to have a single worker per machine.
Use notifications for dependency updates. Keeping up with changes to dependencies is vital for any cloud project. For example, the Cloud Foundry Buildpacks Slack channel uses this Slack bot to post notifications every time a buildpack dependency is updated. Another example is Altoros’s Jumpbox Ansible role, which is updated automatically whenever a dependency gets updated.
Don’t forget about security. Concourse CI’s TSA components address most security threats. TSA even allows users to connect ATC to workers via the Internet. The only thing to care about is authentication. Currently, there are two main types of it: basic HTTP and GitHub-based.
Expose the Concourse CI cluster. In most cases, this will require a load balancer, like HAProxy. Alternatively, you can install this router-registrar project with Concourse CI deployed with BOSH to conveniently expose the cluster through Cloud Foundry.
Code your logic. Concourse works as a service and can be customized or extended with other services, such as the go-concourse client for Golang.
Save your logs. Getting a log aggregator to store and organize logs is common advice for anyone maintaining a service that generates a lot of activity. There are many tools that can help with that. We would recommend the ELK stack BOSH release.
Collaborate. Remember that you can always find engineers ready to help and answer questions in the Concourse Slack channel.

Summary

Concourse CI is awesome for Cloud Foundry or any large cloud-native project that uses microservices. It is written entirely in Go and is available as standalone binaries that run on any platform / architecture with Go support: Windows, Linux, or OS X, as well as IBM POWER8, ARM, etc. Developing resources in Go means we can reduce pipeline execution time, using lightweight images for containers (e.g., Alpine or BusyBox). Alternatively, apps and dependencies can be packaged as Docker images. Last, but not least, Concourse CI supports different types of authentication, including Vault.

Concourse CI is already employed by dozens of companies, Altoros among them. Our team is also working on some new functionality that will make Concourse CI even better. We will keep you updated on the progress.