10 Key Criteria for Evaluating API Automation Tools

by Alex KhizhniakFebruary 10, 2015

APIs are crucial for any microservices-based architecture—as a means of communication between its components.

Developing an API from scratch is not a trivial task. You need to provide meta data, write server code, create a DB structure, develop new methods for new objects, etc. Luckily, many of these tasks can be automated. In this blog post, we investigate what are the key criteria for selecting API automation tools, which can be especially useful when creating microservices.

Table of Contents

API automation: research background

Dozens of tools have been created over the years, covering various aspects of API development and supporting different technology stacks. By using one or a combination of such tools, you can shorten development cycles and make it easier to maintain the system in the future. The only question is which one to choose.

Recently, our solution architects did some research into API frameworks that can be used for distributed cloud-based systems. The tools were evaluated across 19 technical criteria we considered to be important. Our goal was to find an API tool that would automate the entire development process and significantly shorten the release cycle. The parameters were mostly focused on building APIs for a Java project based on the Cloud Foundry PaaS.

Key selection criteria

The following table demonstrates what parameters and why our team considered to be important.

Criteria	Why
1. License type / open source	For the sake of customization, we needed open source tools.
2. API code generation	The main goal of this research was to find the tools that could automate the process of designing APIs.
3. Request-response model and endpoint	Data formats and types of protocols to work with (e.g., JSON, REST) were also under consideration.
4. Programming languages/platforms	Our project was Java-based, so all the tools under consideration had to support it.
5. Availability as a service	This would give us an ability to test the products and start using them.
6. Stateless process	For simpler implementation of a cluster and high level of scalability.
7. Authentication and authorization	Support for popular standards (such as OAuth2) would enable rapid integration into existing systems.
8. Logging and monitoring	Would be nice to have these for debugging.
9. Scalability	The chosen solutions should run on multiple instances.
10. Data storage	The tools should support NoSQL databases, such as Cassandra.

We started the evaluation with Apache UserGrid, WSO2 API Manager, Cumulocity, MuleSoft API Gateway, and StrongLoop Server. These five tools satisfied most of our initial criteria. Obviously, the list of the parameters for evaluation may vary—depending on a particucal use case.

In the end, our selection narrowed down to Apache UserGrid and WSO2 API Manager. If you’d like to know which of the options was selected and why, read the full study. There, we provided a scoring framework with all the 19 parameters important for evaluation. The results of the research were compiled in a comparative table, followed by deployment guides for the both tools.