How to Add BOSH Support to a Custom Cloud (Part 1): BOSH Components

by Alexander LomovApril 16, 2015

With a step-by-step instructions, learn how to establish relations between BOSH components, enable MicroBOSH on OpenStack, etc.

Part 1 | Part 2

The Cloud Foundry PaaS was designed to provide cross-cloud portability and compatibility. BOSH is the official orchestration and deployment tool for CF that makes these features possible. Currently, there is a set of cloud platforms that are able to work with BOSH, but it can be extended to work with clouds that are not on this list.

In this blog series, we’ll go through all the steps necessary to add BOSH support for a new cloud: from CPI implementation to generating a stemcell.

Table of Contents

The process in a nutshell

In the recent years, the BOSH team have been working on an external CPI mechanism that will make adding BOSH support to new IaaS systems quicker. Still, implementing the full stack of solutions to accomplish this is rather difficult. The BOSH project is evolving very rapidly, so it is hard to find up-to-date information about it. There is also no explicit description of the full process for creating a BOSH CPI in available publications.

Enabling BOSH support for a new cloud is a large topic, so we’ve divided it into three parts:

Understanding BOSH Essentials. This part covers some necessary background information on BOSH, as well as the requirements to your cloud infrastructure. You can skip it, if you already understand BOSH releases, CPI, stemcells, and MicroBOSH.
Creating an External BOSH CPI. This blog post will describe how to implement a BOSH CPI in the most modern way and say a few words about the new way to bootstrap your cloud with MicroBOSH using the bosh-init (bosh-micro-cli) project.
Building a Stemcell for a New IaaS. Currently, building stemcells for new infrastructures is not an easy task due to lots of reasons. This post will help you to cope with the issues that you are likely to encounter.

So, let’s get to know the BOSH components better. Here, we’re using the BOSH version 1.2922.0.

BOSH components and relations between them

If you are a beginner, we strongly recommend that you look through this documentation page and get familiar with the BOSH components before you start.

A BOSH release is a package that defines how software is compiled and how it should be run. So, it determines how sources are compiled, how templates are rendered, etc.

A job is an execution unit for BOSH. In fact, a job is a virtual machine where software from BOSH releases is executed. Jobs can contain content from several different releases.

BOSH CPI stands for BOSH Cloud Provider Interface. It is a programmatic wrapper between BOSH and the API of a specific IaaS. A BOSH CPI implements a list of methods defined by the cloud class, such as create_vm, create_disk, attach_disk, etc. Originally, the CPI was implemented as a Ruby gem and a built-in BOSH project, but an external CPI mechanism has recently become available.

A stemcell is another cloud-specific BOSH component. In BOSH terminology, a stemcell is a cloud- or hypervisor-specific image that contains a pre-installed BOSH agent with associated packages, folder structure, groups, and users. These support the operation of BOSH inside an instance. In fact, every instance performing a BOSH job is created from a stemcell image. We will take a look at stemcell internals in the upcoming articles.

MicroBOSH is an official way to bootstrap a cloud with a BOSH installation limited to one instance. The term MicroBOSH is often used to refer to a single-instance deployment of BOSH. The main idea behind deploying MicroBOSH is to use BOSH itself to install BOSH components (more detail will be provided below).

The following diagram shows the relations between all these parts.

So, if you decided to run BOSH on a new cloud, you will need to implement two things: a BOSH CPI and a stemcell builder. The CPI uses stemcells in its methods, such as create_stemcell and create_vm. The stemcell contains a MicroBOSH release with a CPI for the target cloud (the reason is described below). This cross-dependency causes lots of issues during development. In our upcoming posts, we are going to explain how to solve these issues.

Understanding MicroBOSH

Since MicroBOSH is used to install BOSH, you need to know how MicroBOSH works before you start implementing the CPI and building a stemcell for your cloud.

The standard MicroBOSH deployer is implemented as a BOSH CLI plugin. You can read about BOSH CLI plugins and how they work in our previous blog post. Almost the entire process of deployment is declared in the create method of the InstanceManager class. This process requires a MicroBOSH release to be built into the stemcell.

Let’s see how MicroBOSH works in case of OpenStack:

First of all, you need to specify the stemcell and the manifest—the way it is described in documentation.
When a user runs bosh micro deploy, this triggers the create_deployment method of InstanceManager and then the create method is called.
Inside the create method, the create_stemcell CPI method is executed:
- It takes an image file as a parameter and unpacks it to the root.img file located in a temp folder.
- Using the glance service of the fog library, it creates an image from the root.img file.
- After the image has been created, the image ID is returned to BOSH.
Then, the create_vm CPI is called. This method creates a VM using the image ID that was returned after the create_stemcell invocation.
After this step, BOSH waits for the agent to be run on the VM.
At this stage, the BOSH agent on the new VM gets a request to build a MicroBOSH release that is stored in the local file system of the VM. Instructions (or specs) for building a MicroBOSH release are stored in a stemcell file. They are passed to the agent by the BOSH Micro CLI plugin.
After the agent is done, MicroBOSH waits for the BOSH Director to finish his jobs, and the deployment is over.

The concept of MicroBOSH deployment is not so straightforward, but it’s not too hard to see how it works. Anyway, things are going to change soon with the new bosh-micro-cli tool that will be released in near future. You can find more about bosh-micro-cli in this conversation.

Issues to solve beforehand

After you get an understanding of BOSH basics, you’ll probably want to take a look at your infrastructure and check if it has all the necessary features. We suppose, your cloud is ready to perform such basic functions as creating and running VMs. What else will we need? Dealing with infrastructures of our customers, we have met several typical issues related to clouds that want to add BOSH support.

First of all, some clouds have difficulties with creating new custom images. By “custom image,” we mean a template for a bootable system disk that will become a VM. This is how we create stemcells. The stemcell is generated with a special builder, which is a part of the BOSH project. This is done to ensure that BOSH components will work in an environment with specific binary libraries. The way the stemcell builder is implemented implies that your cloud can use a file as a “custom image.” In case of OpenStack, this is the .qcow2 file format built using the qemu-img tool. vSphere uses the vmdk files built with the ovftool utility. If you can’t use the same workflow, you will need to implement “lightweight” stemcells. We’ll provide more details on this solution in the upcoming posts.

Second, BOSH saves data to persistent disks. This means the data is retained after an instance has been rebooted. It is possible to make BOSH work with clouds without persistent disks, but such a deployment will be unstable. For instance, you can take a look at this Twitter discussion about BOSH support for DigitalOcean.

There are also some optional features, e.g., sometimes it is useful for BOSH to write data to a metadata service. A metadata service helps BOSH to optimize the working process and use tags and names for instances. Such services are implemented in the most popular infrastructures, e.g., AWS, OpenStack, and GCE. This is an optional requirement for BOSH deployment. In fact, using a metadata service for sensitive data in public clouds can be dangerous. You can avoid it altogether by relying on a BOSH registry.

Blob storage is another a nice-to-have feature for your cloud. This is where BOSH stores compiled packages used by jobs. The AWS CPI uses the S3 service as blob storage, while Swift is used in case of the OpenStack CPI. This doesn’t mean you cannot run BOSH without blob store support. There are different substitutions for it, e.g., the BOSH project has a simple built-in blob store server.

Next: an external BOSH CPI

By now, you should be familiar with the basic BOSH terminology and how BOSH interacts with different IaaS systems. In the upcoming posts, we will go through the steps for creating an external BOSH CPI module and building a stemcell for a new cloud.