Warner Music Builds a Software Factory with Cloud Foundry
Entertainment: Cloud Foundry is essential in bringing the "factory" paradigm to software development as a music company becomes, in essence, a software company. Warner Music Group uses and extends Cloud Foundry as an essential platform in all of its operations, internal and external.
Much closer integration with and faster response to line of business needs. Every application and service within Warner Music Group is put out with Cloud Foundry.
1) A complete organizational re-think is required to achieve dramatic change on this scale.
2) Deploying across multiple data centers stresses the CAP theorem. WMG must emphasize availability over consistency, as it does not exist in a nanosecond environment.
3) WMG decided not to run terabytes of data into a single Cassandra cluster with 30-50 apps that might hit this cluster. Instead, a single Cassandra cluster was deployed for each app stream--that may have multiple services running within it.
Cassandra
It serves global employees and customers, with a specific Cassandra cluster for each application stream.
Warner Music Group comprises an array of businesses aimed at helping artists achieve long-term creative and financial success while providing consumers with the highest-quality music content available. Warner Music has 5,000 employees and more than $3 billion in annual revenue, making it the third-largest music company in the world.
It dates to 1958 with the creation of the Warner Bros. records out of Warner Bros. Pictures by Jack Warner. The original Warner brothers opened their first movie theater in 1903.
Note: This use case is based on presentations by , ex-Lead of Infrastructure Engineering at WMG (presented at the CF Summit 2014) and by , former EVP and CTO at WMG (recorded in 2013).
A complete organizational re-think
Warner Music Group (WMG) is the world’s third-largest music company and has 5,000 employees with about $3 billion in annual revenue. The company comprises an array of businesses aimed at helping artists achieve long-term creative and financial success while providing consumers with the highest-quality music content available. WMG must serve its own developers, other employees, and customers in all regions of the world.
By late 2013, WMG was well into what it calls its “journey to Cloud Foundry.” Representing what company EVP and CTO Jonathan Murray called a “long-term commitment,” WMG had a vision to build a “composable enterprise” and “software factory” that are dynamic, resilient, and innovative.
A few overarching essentials for the journey appear in the following slide:
The company had 300 existing applications that Murray says “we’re not going to migrate, but replace,” using a strategy and techniques made possible with Cloud Foundry. “Backward compatibility is no longer an approach than an enterprise can afford. Building on the past is no longer something an enterprise can afford,” he stated.
Complexity builds over time
The enterprise IT journey WMG has undertaken aims to break up and remove the cement of the past, brought on by legacy technologies that are simply built on top of one another.
“Development and deployment speed is limited because complexity has built up over generations of IT,” Murray said. “New tech built on top of old simply adds complexity.” In contrast, the CF-centered strategy aims “to create organizational models that had the adaptability we need for today’s global environment,” according to him.
Another drawback of grappling with generations of legacy technology is that much of the IT spend is for systems integration work rather than delivering value to the organization. So WMG is now working to develop a common service network that will deliver such value, rather than pouring more cement into the organization.
Creating a factory
The Factory Platform Strategy has four keys:
- Decouple from infrastructure
- Make data a service
- Decompose applications
- Automate everything
On the first point, Murray said he doesn’t want to care about servers, VMs, instances. operating systems, hypervisors, etc. “What I care about is delivering applications to the business,” he said.
With data, the key is to avoid putting a standard DBMS in the middle of the architecture. Instead, the software factory approach puts transaction processing into the service, and business logic semantics also go into the data services layer.
Multi-datacenter architecture
WMG has extended Cloud Foundry into its multi-datacenter environment, and also created a unique cloud console for its developers and non-developers alike.
Part of its approach lies with using the Cloud Foundry User Account and Authentication (UAA) server to provide an interface to applications running on top of Cloud Foundry. It also runs a second UAA as a Cloud Foundry application itself, managed by BOSH, where it can design and deliver its modifications. It has integrated UAA with Microsoft Active Directory, as the company’s IT originally came from a traditional Microsoft-centric enterprise. It does so with as much care as possible.
“If we were going to upset our own industry and users by re-writing the entire stack of how we run our business, we wanted to make this a pleasant situation,” according to Brian McClain, Lead of Infrastructure Engineering at WMG. In that process, it removed a previous integration with Postscript and migrated to the Apache Cassandra open-source distributed database management system.
WMG had already been running Cassandra within its enterprise IT department. Today, “Cassandra is our main data store. Using globally distributed clusters allows multiple instances to run and serve requests,” according to McClain. “We have our core Cassandra cluster that serves our basic needs of authentication and authorization globally distributed along with our other Cassandra data centers. running multiple instances of UAA,” according to McClain. “We also keep a networking separation between our public-facing applications and our internal-facing services.”
A high-level view of WMG’s architecture is shown in the graphic below. Individual Cloud Foundry deployments are highlighted in blue. As McClain notes, “these are not pieces of a cluster but full Cloud Foundry per each one of those individual boxes. All of our data services run on that layer.”
WMG also made a big bet with Cassandra for its data persistence layer. Each of these data centers in the case of network partition can run completely independently of one another. There is data replication to each Cassandra node, one copy of that data in each data center. If a data center network happens to go down, WMG’s IT department can quickly send all requests to a data center that’s running correctly.
McClain acknowledges that within the CAP theorem WMG is trading off some consistency for availability. “But we’re not dealing with (for example) nanosecond updates. Cassandra serves us in a way that meets the requirements we do have.”
A big lesson learned is not to try to run terabytes of data into a single Cassandra cluster with the 30 to 50 apps that WMG might have all hitting this cluster. WMG has instead deployed a single Cassandra cluster for each application stream. Each stream may have multiple services running within it.
Cloud console
Another significant aspect of WMG’s use of Cloud Foundry was the development of a “cloud console” for both “developers and non-developers alike,” says McClain. Applications and services get different Cloud Foundry clusters, and can be spun up to meet specific needs. “Whether its public vs. private vs. internal vs. hosted by us for customers–or any other scenario–our approach gives us flexibility without putting lots of logic into load balancing and the writing layer,” McClain notes.
Delivering Cloud Foundry in this manner has helped WMG’s developers to become much more productive. They’ve increased the velocity with respect to how quickly they can turn out services and apps. WMG can also on-board new Cloud Foundry developers quickly by use of the console, and also look at more business-related metrics through its interface layer.
“We don’t put out an app or a service that’s not based on Cloud Foundry today.” —Brian McClain, WMG
A few screenshots from the console appear below. ‘You’ll see a lot of familiar concepts if you’ve done consoles before,” says McClain. “You can dig into variables. We’ll continue to build this out.”
This second view shows the Ruby control. There are others for Java, Node.js, Go, etc.
Simplicity rules
WMG also avoids a traditional master data management (MDM) approach. It sees no reason to stick MDM “into the middle of the architecture,” according to Murray. Instead, “simplicity rules the day as we again put semantics into the data services layer.”
Apps are not only decomposed, but “we also don’t want applications building duplicate services,” Murray said. There should be “an ease of service discovery” using a tool such as the Swagger API framework. “Developers needs easy access to service APIs, to JSON scripts, and they should be able to test service calls in a very dynamic and robust environment,” he said.
With automation, the questions for WMG are:
- How far can the organization continue to push things?
- How radically can we use IT to transform the business?
Centering around the Cloud Foundry platform, “we built a new organization, a software business,” according to Murray. “We have product management, engineering processes, and we think of WMG’s business as the clients for our software company. We are able to interact with line-of-business leaders with a very agile software development process.”
Related videos
Here is Brian McClain’s presentation at the Cloud Foundry Summit 2014 in San Francisco.
In another session (recorded in 2013), Jonathan Murray, former EVP and CTO at WMG, tells how Warner is building an enterprise-grade PaaS based on CF—”from zero to a software factory.”
Related slides
This is the SlideShare version of Jonathan Murray’s presentation at the Cloud Foundry Conference 2013:
And here is Brian McClain’s presentation at Cloud Foundry Summit 2014:
Related reading
