Among the various jobs of a Web Agency there is often the realization of APIs. In short, APIs are services that an application makes available to other applications to view and modify data. Exactly as it is possible to visualize and modify data through Web pages, it is possible to perform the same operations through APIs. Imagine for example you want to edit your account name: you could achieve that through a web page, or also by submitting an API request to a specific service. Now, what if you want to edit 1000 names? Doing this through web pages, might be tedious and boring. This can be automated by using APIs and writing a small piece of code that automatically invokes the API service 1000 times and updates all the names.
In this post I will explain how we design and implement REST APIs at Renuo that are reliable, performant, easy to maintain, extensible and monitorable.
Documentation
When defining APIs for your clients, we always document them via swagger and provide not only the swagger file, but also an instance of Swagger UI, where the clients can easily read the documentation and test the services.API documentation must be easy to understand for both humans and machines. Good documentation should always be up-to-date, and contain many examples to make it easy for customers to use. Providing a testing environment is essential for our clients to develop and «hook into» our system quickly. Finally, providing examples in real programming languages will give an additional boost to productivity and make our API documentation truly complete.
At Renuo we use the OpenAPI 3 standard, and we always install an instance of SwaggerUI to be able to read and access the documentation. Both for «internal» APIs, i.e. in use only by us, and for «external» APIs, i.e. publicly exposed.
Reliability
The reliability of the API can be evaluated on different aspects. First of all the speed: how long do your APIs take to provide a response? Reliable APIs must be fast.
At Renuo we rely on Ruby On Rails: the reliability of APIs developed with this framework is, to date, undeniable. Many companies in the market use Rails to build their APIs and serve up to 2.8 billion requests per day. We know that if Ruby On Rails is able to satisfy the needs of GitHub, Shopify and Netflix, it will also hold up to our customers' needs.Besides speed, a second parameter is definitely the uptime of your API. This is not a parameter related to the application, but to the infrastructure in use. At Renuo we rely on Heroku for international customers and local providers for Swiss customers. The uptime is calculated every month by uptimerobot that indicates and reports problems. A final parameter on which to evaluate the reliability, is certainly the quantity and quality of errors. We distinguish two types of errors: managed and unmanaged.
Managed errors, i.e. those that normally return an HTTP code between 300 and 499, are not considered in the «reliability» parameter: these errors, as we said, are managed and therefore expected. One of the most well known errors is the 404, which shows up when a page or a file can not be found.
Unmanaged errors, usually with HTTP code 500, are the ones that contribute to the reliability of a system. A flawlessly implemented API will never return errors with code 500. Such errors are unexpected situations that are not handled properly.
When an API returns a 500 error, it reduces its reliability. In these cases, it's really important:
do not communicate any details about the error to the client. Since the error was unmanaged, you are not in control of what data might be exposed, including customers’ sensible information
maintain the same design, i.e., do not return HTML or other data structures;
send immediately to a logging system the details of what happened;
have a process within the development team to identify, analyze, and resolve the problem as quickly as possible;
communicate with the client that the problem has been corrected.
To be able to communicate with the client, it is necessary to have a correct management of the API-Key, as we will see later.
Design
API design is very important and must be set up right from the start. Changing the design of your APIs can be very expensive and your clients may be forced to make heavy and expensive changes if the design changes significantly between version 1 and version 2. It is therefore very important to choose the right design from scratch.
At Renuo, we always start with JSONAPI Standard. We don't think it is the best standard in most cases but it is standard.
Relying on a standard for the definition and design of your API means cutting out all the discussion parts. This results in significant time savings in both the initial definition part and the subsequent implementation.
Since you rely on a standard, in fact, there are already many libraries, for the most common languages, able to implement and communicate through JSONAPI.
Maintenance and versioning
How to correctly maintain and version your APIs? On the internet you can find many techniques to version REST APIs. At Renuo we rely on a custom header called API-Version. This parameter is optional and by default set to 1. We use a semantic versioning system in our API development. This means that our API has a three-number format: 1.0.2.
Since patches and minor releases are always backward compatible, it is not possible for the client to choose: the client is only given the possibility to change the major version and thus to adapt to breaking changes.
For API versioning it is also really important to keep a log of which version your clients are currently using. This way you can contact clients that are using older versions and invite them to migrate to newer versions of the system.
It is very important that the API development team is always very careful with API changes. It is the team's responsibility to correctly distinguish between patches, minor and major changes, and to correctly release these changes.
Api keys
We have already talked about clients, communication with clients and monitoring. At Renuo we always set up an Api-Key for each client from the beginning. Api-Keys are used to uniquely identify the consumer of our API. So we keep, on database, a list of our Api Clients with the following data:
API-key: encrypted key, used to access the services
Name: a unique name to identify the client
Email: the email address of the person to contact
API-Version: the default API-Version in use if not specified in the header
Allowlist: a regular expression to define which APIs are accessible
Denylist: a regular expression to define which APIs are not accessible.
A client can use an API if:
the key is correct;
the version is correct;
the API path is included in the allowlist;
the API path is not included in the denylist.
A logging system also records all calls made by each client. This way we always know:
how many calls are made on each version of the API and by whom;
how many unhandled errors occurred and who was affected by them;
average response times for each client;
and much more...
Monitoring
The last aspect to talk about is API monitoring. Along with the Api-Key system mentioned above, we always keep a log of the API requests made and record them:
API-Key used
API-Version used
API request: path + method
Body of the request
Body of the answer
Returned HTTP code
Request duration
Moreover, for each error occurring within the application, we send a signal to an error monitoring and management system: Sentry.
This way we have complete control of our API and all the data we need to be able to analyze possible problems.
But no tool is sufficient if there isn't a well-defined and efficient process for monitoring. The weak link in the chain, and not only in API development but in software development in general, is that in the face of the best possible monitoring systems, there is not always a well-defined process of accountability within teams.
It is common for errors and monitoring to be simply ignored, as there is no defined chain of responsibility.
At Renuo, in addition to the best software that money can buy ;) , we have daily monitoring and dispatch of the errors that occurred the previous day. Not a single error is ignored.
Final considerations
Defining and exposing APIs is easy, but doing it right isn't obvious. In our experience, what matters most are the points listed in this article.
If your API design and monitoring is well structured, your customers will be grateful. Also, maintaining these best practices when developing internal APIs is definitely helpful.
On a technical level, many of the tools and best practices we use have been made Open Source. Among the latest libraries we've released is the Rails API Logger, to help with the monitoring side of things.
