Porting Cloud Foundry to IBM POWER Processors

by Alexander LomovAugust 14, 2015

Featuring a demo of a CF app on the ppc64le architecture, this blog post explains how to tweak build configuration, compile Ruby gems, modify buildpacks, etc.

IBM POWER microprocessors are extensively used to run mission-critical workloads for the world’s largest financial companies, but they are little known inside the Cloud Foundry community. IBM and Altoros are working to change that by bringing Cloud Foundry to the POWER8 architecture. This post sheds some light on the current progress and describes the issues that we had to overcome. It also features a short demo of a Cloud Foundry app running on ppc64le architecture.

Table of Contents

What is Power Architecture?

Power Architecture is an RISC (Reduced Instruction Set Computing) microprocessor architecture for a line of high-performance servers by IBM. The Power platform delivers superior security, workload density, and sustained utilization compared to commodity x86 systems. According to results shared at the STAC Summit on June 4, 2015, an IBM POWER8-based server delivered more than twice the performance of a best-in-class x86 counterpart when running a set of standard financial industry benchmarks.

POWER8 systems can run up to eight threads on a single core. Their memory bandwidth provides almost three times the speed of a typical x86 processor. They also feature a large internal L3 cache. All of these enable POWER8 to deliver dramatically higher performance for data and compute intensive applications. The Power Systems line of servers is also expanding to include smaller scale-out systems, which provide the same performance and security advantages as the large systems, but at a smaller scale.

Despite these strengths, the platform hasn’t received much attention from the Cloud Foundry community. So, at the moment IBM and Altoros are working to enable CF on POWER8, a new server-oriented processor architecture. The work is still in progress, but we have already achieved some tangible results, which are described below.

What has been done so far

At this moment, we have developed an automated way to create stemcells, build BOSH and CF releases, and, finally, install BOSH and CF over OpenStack with Ubuntu VMs on IBM Power Systems. All this work was done inside of Oregon State University’s OpenStack environment based on POWER.

The project is implemented as a set of Ansible playbooks that can be used to build BOSH releases with binaries and create a BOSH OpenStack Ubuntu 14.04 stemcell that supports the Power architecture. You can find it in our GitHub organization. To help you easily deploy MicroBOSH and CF, we also created a project with a BOSH workspace that contains templates for manifests and instructions on how to use it.

These two projects provide proof of concept for running Cloud Foundry on top of IBM Power servers and may be interesting to those who utilize BOSH and CF on a daily basis. They help to quickly install all the software necessary to run a jumpbox. There are some similar projects, such as the BOSH release for a jumpbox, but we believe that it is easier to run Ansible playbooks on a clean system. Below, I describe some of the issues that we encountered when porting CF to IBM’s Power architecture, as well as the workarounds that we found for them.

Issues and solutions

Changing build configuration

A common issue encountered when trying to get BOSH and CF clusters to run under Ubuntu on Power is related building dependent open-source projects. Many OSS packages (e.g., SQLite) include a script file called config.guess, which is part of the standard GNU Build System. This file is used to determine the platform and architecture type of the build. Since Ubuntu Little Endian (LE) is a new platform on Power, many dated config.guess files are unable to correctly determine the platform type. This results in failed builds with the following error:

configure: error: cannot guess build type; you must specify one.

One possible workaround is to specify the build platform (ppc64le) in the configure command line with the –build parameter, e.g.:

./configure --build=ppc64le

Another way is to update the config.guess file to the latest version, like this:

curl "http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.guess;hb=HEAD" > config.guess

You can find the updated script that performs these operations in an Ansible playbooks role called binary builder. We are already working to ensure that an updated config.guess file is used for as many source code packages utilized by BOSH as possible (see the pull requests here and here).

Golang support

Most Cloud Foundry components use Go as the implementation language. Currently, the native Go compiler from Google doesn’t support the Power architecture. There are plans for such support in future releases, but the transition to a new version of Go may take some time, like it was with the transition from Go 1.3 to Go 1.4.

We found that the GCC frontend for the Go language satisfies all the requirements and gccgo supports versions 1.3 and 1.4 of the Go spec. Still, an issue surfaced when gccgo was used to run some of the BOSH components, e.g., BOSH Agent and bosh-init.

The order in which the parameters of a function return statement are evaluated is not fully defined in the Go specification. This can result in unexpected behavior when a function returns a variable whose value is calculated through a closure in function execution that follows this variable in the return statement. This issue is described in more detail here. To solve this, we had to update function calls in the BOSH Agent and bosh-init components in a few places.

One more feature that we have discovered while using the GCC Go frontend is that it does not compile binaries statically by default. The resulting binary depends on the libgo.so library. This behaviour can be easily changed by specifying a couple of additional build options (-gccgoflags -static-libgo) for the Go compiler.

Ruby gems compilation

Lots of BOSH and CF components are written in Ruby and take advantage of Ruby gems. These include BOSH Director, CF Cloud Controller, DEAs, Warden, etc. However, Ruby gems with native extensions may not install easily under Ubuntu on Power. Many of the root causes are similar to the issue described in the “Changing Build Configuration” section. The latest versions of many Ruby gems have already been fixed and CF BOSH releases have been updated to include the latest gems, but there are still some that may not install smoothly.

A good example is the nokogiri gem that uses outdated versions of the libxml and libxslt libraries. The libraries also have an incorrect build configuration (an outdated config.guess file). During nokogiri installation, libxml and libxslt are fetched and built, which causes an error.

There are a couple of solutions for this. It is possible to put correct versions of these libraries inside a gem file. After some experiments, we created a script that can build this kind of nokogiri .gem file. Another way is to make bundler use system libraries when installing Ruby gems. Since the BOSH stemcell already includes libxml and libxslt, we decided to simply link with the system libraries. As a result, we did not have to update any of the Ruby gems.

Generating rootfs

All Cloud Foundry apps run within Warden containers inside Droplet Execution Agent (DEA) virtual machines. The root file system employed for Warden containers under Ubuntu on Power needs to be different from the root filesystem for Ubuntu on Intel x86.

The root file system is packaged in a tarball in the Cloud Foundry BOSH release. So, we had to create a container root filesystem for Ubuntu on Power with this script. To run applications under the Ubuntu on Power root file system, we added a new stack (trustyppc64le), pointing to this new rootfs, to the Cloud Controller and DEA components.

The stemcell builder

Stemcells are basically images used by BOSH to run jobs. A stemcell contains an operating system with all the necessary software, as well as an embedded BOSH Agent that allows BOSH to control VMs created based on the stemcell image. From the very beginning, we decided to support bosh-init as a MicroBOSH deployment method. This was done not only because bosh-init is likely to be the future standard. In fact, it also eases stemcell builder development and increases its speed.

The old way, implemented as a BOSH CLI plug-in, implied that stemcells have to include a built-in MicroBOSH release. This resulted in an additional special stage in the process of building a MicroBOSH release by the stemcell generator. So we patched the existing stemcell builder. At the time this was written, the latest version that our stemcell builder supported was 3033.

Buildpacks

Cloud Foundry apps require buildpacks to work under different runtimes. At the moment, Cloud Foundry buildpacks (e.g., the Node.js buildpack) only include executables for x86 systems. So, we are in the process of modifying the buildpacks to include executables for Ubuntu LE on Power.

The first buildpack that we have updated was for Ruby. BOSH buildpacks use the buildpack-packager gem to fetch dependencies. Standard buildpacks—e.g., for Ruby, Node.js, and Go—use scripts from the compile-extensions project to select, fetch, and cache binaries for the required stack.

Our branch of the Ruby buildpack contains scripts for both building and uploading the required binaries to S3. At the same time, we are going to add Power support to binary-builder, a project that is used by the community to build binaries. At this point, we can demonstrate apps running with custom buildpacks for Ruby, Node.js, or Golang deployment.

BOSH Agent

Once started, BOSH Agent begins reading settings and handling such resources as disks and networks. You can learn more about this process from this blog post. On OpenStack, when a VM doesn’t have an ephemeral disk, BOSH Agent makes partitions for a data drive and swap disk on the root device. In its current implementation, BOSH Agent expects the root filesystem to be mounted to the first partition (e.g., /dev/vda1). Power systems need to have a PReP boot partition as the first partition and this means that the root file system has to be mounted as the second partition (e.g., /dev/vda2).

To fix this and enable BOSH Agent to work with OpenStack flavors that only have a root disk, we needed to patch BOSH Agent. These changes have not been merged to the BOSH repository yet, but we are moving in that direction.

Demo: running and scaling CF apps on IBM Power

This video demonstrates a development milestone that has been reached, with a Ruby app pushed and scaled in Cloud Foundry running on IBM’s POWER architecture.

What’s next?

Although Power Systems are only starting to gain popularity, they can be a good choice for running large Cloud Foundry clusters and/or heavily loaded applications. Our next steps will include merging the necessary changes to the repositories of the Cloud Foundry community and setting up a test environment for the CI process. If you have any questions, you are welcome to leave a comment below.