Are Diego and Docker Really Good Friends?
Support for Docker is one of the main advantages of Cloud Foundry Diego, but how far does this compatibility go? Is it possible to push an arbitrary image from the Docker Hub to Diego and, if not, what are the constraints? What’s going to change in the future? Finally, why should anyone want to use Diego with Docker at all? In this post, we’ll provide answers for these questions.
Technical prerequisites
We’re going to examine the features of the latest (as of the time this post was written) Cloud Foundry release (v207) and the corresponding version of Diego (release v0.1099.0). The functionality that we will describe has been tested on AWS deployments of Cloud Foundry and Diego.
Diego has to be configured to use the Docker back end (the diego_docker manifest property has to be set to true). In order to push Docker applications, the Diego plug-in for the CF CLI has to be installed.
How Diego supports Docker
When we talk about Docker support, we mean that some Docker image, which Docker can create a container from and run processes in, is consumed by Diego to create a container, in which the same processes can run seamlessly.
To provide such support, Diego uses Garden-Linux that constructs Linux containers and mounts Docker images to them. While not directly using Docker to create containers, Diego employs some of the libraries Docker is built with. With them in place, Garden-Linux extracts the necessary metadata (mainly the commands to be launched) from a Docker image, mounts all the image layers to the container it has created, and executes the starting commands along with the Diego health-check script when the container is launched.
You can find more details about the Docker support in Diego, as well as some notes about the compatibility issues and work in progress in this GitHub repo.
The main incompatibilities
The two main pitfalls that may prevent some Docker images from running on Diego are the health-check script indispensably checking the 8080 port and container processes running as the VCAP user with limited permissions and ownership rights.
This is not a big deal if you run a regular user application that normally doesn’t need to have root privileges or use files in system directories that may require some permissions. In that case, you can usually configure the port to fit the external requirements. The “port issue” does not apply to the apps that do not need routes either. Issues arise if you are working with applications that are actually services: web servers, databases, message queues, etc.
Available workarounds
The simplest solution for the issues described above is customizing your Docker images, so that they match Diego’s constraints.
Consider the official Redis image as an example. If we look through its Dockerfile and the starting script, we’ll see that the redis user is created and then used for starting the service. This won’t work in Diego, because the vcap user doesn’t have the privilege to set the user ID. What we can do here is run Redis as the vcap user.
Still, we may need to change user permissions and/or ownership rights. The latter is impossible right now, but there is a workaround to resolve the former. Permissions can be set when a Docker image is created. If we set the proper bits for the other users, they will be applied to the vcap user later when the command will be executed.
RUN chmod 777 /data
Running as the vcap user with proper permissions granted should be enough for most services (Redis, MongoDB, etc.), because they rarely rely on ownership of files. In case of the Redis image, though, it is sufficient to simply remove the corresponding user-manipulation directives.
And what about the port? That’s easy. In case of Redis, we just need to provide the --port argument to the redis-server command.
The modified Redis image with the fixes described above can be found in the Altoros’s GitHub repo. Redis can be installed from our repo on Docker Hub. The installation process is as simple as shown below.
cf docker-push redis altoros/redis
Turning a Docker app into a CF service
We have deployed Redis from the Docker image in almost the same way any regular app can be deployed. Why may we need such a service and how can we use it?
A use case we can see here is a service in a private PaaS that can be easily created either by an operator or by a developer to be consumed by a set of apps deployed to this platform. There are three main reasons why you may want this kind of service:
- the simplicity of service deployment
- the convenience of further service development (you can debug it locally with Docker)
- the advantages provided by Diego, including, but not limited to, health checking, monitoring, log aggregation, and scaling (in case of stateless HTTP services)
So, how do you set it up? One thing Cloud Foundry helps us to do is connect services with apps via user-provided service instances. We can create an instance with the CF CLI, giving it the necessary credentials.
cf create-user-provided-service redis -p ‘{“host”:”<host>”,”port”:”<port>”,”db”:”<db>”}’
How do we know the exact host and port? Since every Diego application is actually a so-called Long Running Process (LRP), we can query the vital information via the Receptor HTTP API.
curl http://receptor.<CF_DOMAIN>/v1/actual_lrps
As a response, we will receive an array of JSON objects, describing the currently running LRPs. The object we need can be found by the process_guid field that is the corresponding app’s GUID (cf app redis --guid). The pieces of information we are looking for are the address field that corresponds to the host parameter and the container_port field that corresponds to the service port. The db parameter is just an index of the database that Redis will use.
There is one more step here: You have to configure a Cloud Foundry security group to allow incoming TCP traffic for the service. For more detailed instructions, refer to the README file of cf-mysql-release.
Some important things to note here:
- The application route is not of any use to Redis, because Router in Cloud Foundry has no support for TCP traffic. Sure, if you deploy the HTTP service, the route just works, and there is no need to investigate the host and port.
- It is up to you which credentials to provide. This depends on the applications that consume the service (e.g., in case of Redis, you can additionally set the password or omit the
dbparameter, so that it defaults to “0”). - Each time you start an app, Diego assigns a new port to it. After a restart, the service needs to be updated (
cf update-user-provided-service) and its consumers need to rebind it and restage (yeah, that’s lame). - A user-provided service instance exists only inside the space it was created in.
- There is currently no way to get a user-provided service instance with tags. Therefore, your application needs to search for the service by name. This results in some constraints related to binding multiple instances of the same service to multiple instances of a single application.
- Containers that require persistent data is one of the anti-goals of the Diego team. Therefore, this approach is not suitable for databases and other services that are used for persistent data storage.
We’ve modified a popular cf-redis-example-app from Pivotal to work with the user-provided Redis instance. You can get it from this GitHub repo. Note that if you want to push the app to the Diego back end, the external network access has to be explicitly enabled to consume the Redis service.
Summing it up (for Ops and Devs)
To conclude, below is what the above-described process looks like from an operator’s and developer’s perspectives.
The operator’s way:
cf docker-push redis altoros/redis
cf app redis --guid # we’ll use this in the next command
curl http://receptor.<CF_DOMAIN>/v1/actual_lrps/ # search for the host and port
cf cups redis -p ‘{“host”:”<HOST>”,”port”:”<PORT>”,”db”:”0”}’
# create a security group to allow the incoming tcp traffic
The developer’s way:
git clone https://github.com/Altoros/cf-redis-example-app.git cd cf-redis-example-app cf push cf bind-service redis-example-app redis cf restage redis-example-app # try it: curl -X PUT -d “data=value” <APP_ROUTE>/testkey
The service developer’s way:
git clone https://github.com/Altoros/redis.git cd redis/3.0 # do some modifications docker build -t altoros/redis . docker run altoros/redis # local debugging docker push altoros/redis cf restage redis
Future plans for Diego, Docker, and Lattice
Diego is currently in active development. Docker support is being improved, as well. Soon, it will be possible to run processes as an arbitrary user.
As we talk about Diego and Docker, we can’t simply skip Lattice—a project that is now a better friend of Docker than Cloud Foundry Diego. Lattice is a platform that offers a cluster scheduler, HTTP load balancing, log aggregation, and health management for containerized workloads. Its container runtime is based on Diego, but some additional steps have been made towards Docker support. In the documentation, there are numerous examples of official Docker images that can be run on Lattice. In addition, Lattice can be configured to communicate with a private Docker registry.
For more details about the future of Diego and Docker, read the Google doc shared by the Diego development team.
Further reading
