Increasing Performance of a 100-Node Hadoop Cluster: The Story of Joyent

by Alex KhizhniakAugust 30, 2013

Joyent now produces disk I/O response times 3x faster than similar solutions from other IaaS providers.

Table of Contents

Hadoop in the cloud

Previously, companies used to run Hadoop on their own premises. However, a possibility to use resources on demand and pay for the capacities that are actually used seemed rather attractive to many customers. So, recently, Joyent—a leading IaaS provider—partnered with Hortonworks and rolled out a cloud-based Hadoop offering.

When preparing to deliver its Hadoop clusters, Joyent turned to Altoros to get independent evaluation of their system. The results showed that virtual Hadoop clusters can provide bare-metal performance with nearly a three times faster disk input/output speed compared to a similarly sized infrastructure from other vendors.

Ben Wen (Joyent) and Renat Khasanshyn (Altoros) at NoSQL Now! 2013

At NoSQL Now! 2013, Altoros and Joyent conducted a workshop based on this real-life case study. The participants learned how our company helped Joyent to provide the highest-performance Hadoop cluster on the market.

See the slides below to learn about the tools used for performance evaluation of different Hadoop clusters (50–250 nodes) and what kind of initiatives were implemented to achieve this significant improvements. Or, read about the technical details here.

Want details? View the slides!

Alex Khizhniak is Director of Technical Content Strategy at Altoros and a cofounder of a local Java User Group. Managing distributed teams since 2004, he has gained experience as a journalist, an editor-in-chief, a technical writer, a technology evangelist, a project manager, and a product owner. Alex is obsessed with AI/ML, data science, data integration, ETL/DWH, data quality, databases (SQL/NoSQL), big data, IoT, and BI. The articles and industry reports he created or helped to publish reached out to 3,000,000+ tech-savvy readers. Some of the pieces were covered on TechRepublic, ebizQ, NetworkWorld, CIO.com, etc. Find him on Twitter at @alxkh.