PaaS Comparison 2014: Cloud Foundry vs. Other Platforms
Each of the PaaS systems is evaluated for its ability to support different programming models, languages, integration services, data stores, SLAs, etc. This session demonstrates the pros and cons of the six platforms to help you make the choice.
No one likes unforseen issues popping out. This session reveals what to expect of the platforms under consideration while logging, monitoring, automating, scaling, etc.
Before choosing a PaaS, you need know how it behaves under various workloads. This talk features the results of the behavior under different types of load, as well as the list of tools and services supported.
One of the most interesting presentations at this year’s CF Summit was a comparison of Cloud Foundry with other PaaS systems. It was delivered by Michael Maximilien of IBM and James Bayer of Pivotal. The overview featured OSS Cloud Foundry, Microsoft Azure, Google App Engine, AWS+ (EC2, Beanstalk, and services), Heroku, and OpenShift. This blog post highlights the main points of this presentation.
Background
IBM started working on a PaaS comparison back in 2010 to evaluate different platforms for the company. The research presented at the CF Summit was an updated and revised version with 50 criteria divided into 8 groups, such as:
- workloads and tooling
- integration services and SLAs
- programming models and data stores
- management, etc.
The solutions were scored from 0 to 3 in accordance to what extent they support each of the features. The presentation featured scoring tables with the results of the comparison, as well as comments on them.
Dr. Max (left) and James Bayer (right) at Cloud Foundry Summit 2014According to the speakers, this research is not exactly scientific, since there was no way to measure some of the features and often what was advertised did not match the reality. The comparison also has a certain bias to it, since it was made by Pivotal and IBM, members of the Cloud Foundry ecosystem. Still, it provides a framework for other PaaS benchmarks.
(Last year, Altoros did a similar evaluation of Cloud Foundry and OpenShift, but on a smaller scale.)
Comparing the features
Michael Maximilien led the first part of the session, giving insights into how the platforms compared against each other in terms of their features.
Open source PaaS systems, such as OpenShift and Cloud Foundry, are generally stronger when it comes to tooling. They are also better at providing programming models and frameworks. Today, Cloud Foundry is the leader in this area—together with Heroku, which supports multiple frameworks and languages.
Source
Source
Pros and cons
The second part of the session was dedicated to quality of experience. James Bayer described what it feels like to use each of the platforms from a developer’s point of view. According to him, 60–70% of all applications running on a PaaS are Java-based. So, the main scenario was to push, scale, update, etc. an app written in this language to see what it feels like to work with a platform in day-to-day situations.
1. AWS Elastic Beanstalk provides great integration with Amazon’s services along with outstanding out-of-the-box monitoring features. It is really easy to start using Beanstalk, but this platform has some shortcomings. For example, it turned out to be by far the slowest—it took James Bayer 15 minutes to spin up his app. Beanstalk treats different languages/frameworks very differently, you are often forced to learn many AWS concepts, and the extensibility is low.
Source2. Google App Engine is deeply integrated with Google’s services ans APIs, like Beanstalk is with AWS. It comes with nice load balancing, Cloud SQL integration, events, and runtimes. The downsides include lots of Java limitations—e.g., they want you to use their Google SQL driver, you have to insert the DB connection into the code manually, and if you want to customize a container, you won’t be able to debug it.
3. When speaking about Azure Websites, James mentioned that he did not have the time to see all the features, so this evaluation is only partial. Still, he noted the great job Microsoft had done on the web experience. Azure provides lots of SCM integration options and there are many ways you can move your app to the platform. However, they put you on Windows right away, although it is advertised as a general PaaS.
Source4. OpenShift is the platform most frequently compared to Cloud Foundry—both are open source, both were launched at approximately the same time. Of all the platforms evaluated, OpenShift has the best container SSH and port forwarding—e.g., you can go directly to the container and see the problems, forward things to your VM with one command, etc. Unfortunately, many things turned out to be outdated: you have to choose which cartridge to use from a lot of options, and scaling can only happen one instance at a time. In addition, you can see the traffic from all the containers on your host—i.e., other people may see yours, as well.
5. Heroku seems to provide the most “polished” user experience. In addition, they have some features that other platforms don’t: run command, pipelines, stats in logs, a good marketplace, etc. The drawbacks include poor support for Java. You have to use Git, and team support is not so great compared to other platforms.
Source6. The most notable thing mentioned about Cloud Foundry was that it actually has the best Java support—with Maven, Gradle, Eclipse, and many builpacks to choose from. All dependencies and JDK are updated almost every week. Cloud Foundry also “scales and responds fast.” It has Loggregator, introduced about 6 months ago, that logs both app and audit events. Using services is extremely easy—you don’t have to insert anything into your code, etc. However, there is still no shell access to containers, so you can’t see what’s going on. Beanstalk and Heroku have much better app metrics. Finally, zero-downtime updates on Cloud Foundry are not yet a “irst-class” feature—it still requires scripting.
Bottom line and next steps
James and Michael ended their talk by saying it was hard to be objective when comparing the platforms and then shared some future plans. In particular, they wanted the community to be involved in the research to create a PaaS benchmark that will use standard sample apps and scenarios close to real life. Although the comparison did not include any pricing data, it was very thorough and, absolutely, unique.
What else could be there?
In their presentation, James Bayer and Dr. Max clearly state they’re in a quandary in trying to compare several unlike products. They make a game try, though, coming up with some rankings and making detailed "pro" and "con" statements about each.
However, coming from Pivotal and IBM, the two reviewers are going to have a freely admitted, in-built bias toward Cloud Foundry. As a Silver member of the Cloud Foundry Foundation, we at Altoros will have the same bias.
So, the question that needs to be asked is, "Do any of these other PaaS offerings exceed Cloud Foundry to the extent that you should adopt their attached infrastructure?" The real issue is whether you must be a Microsoft house for the project at hand, or whether you’re tied to Red Hat or locked in to AWS or Google.
If you are, then you may not have a choice other than to go with a PaaS other than Cloud Foundry. Further, if you’ve gone with (or are going with) a commercial version of Cloud Foundry, you might have locked yourself in—to some degree—as well.
What we would like to see, for instance, is a comparison of proprietary Cloud Foundry offerings (including a TCO analysis) vs. its open-source version. Or, trying the DIY approach vs. relying on managed PaaS services.
That would be of great interest to developers and architects alike.
Further reading
Want details? Watch the video!
Table of contents
|
Related slides
About the experts
