Creating great APIs is hard. Getting the right level of abstraction and building something that is pleasant for clients to integrate with is a huge challenge. This post delves into this topic in more detail.
But first an anecdote…
Product Owner: “I need you to integrate with this third-party API as part of the current feature. It provides most of the functionality that we require in this area, and will avoid us having to write loads of it ourselves. Here’s the link to their documentation.”
Engineer: *opens documentation link and browses through the cleanly presented documents*
Product Owner: “So, I need a ballpark figure of how many sprints you think this might take?”
Engineer: “If I pick this up, I can probably have it done for you in about three days.”
Product Owner: “How are you going to manage that, these things usually take weeks to integrate with?”
Engineer: “Well, this is a great API. Every endpoint is simple and easy to understand. All the data is structured and named using obvious domain terminology that I already know. They’ve even supplied an OpenAPI description and schemas for everything. I can just chuck this at a code generator and avoid having to write any any boilerplate stuff at all.”
Product Owner: “What about building the actual workflow we need, its quite bespoke. Won’t we have to go back the the API provider and get them to add all of our custom needs into their implementation?”
Engineer: “I don’t think so. Each of their endpoints looks like it does just one thing and they look to be designed to be composed together in different combinations. I think we can achieve our workflow byt combining these four together with just a bit of data transformation between each call plus a bit of simple error handling. This is going to be so much more fun than the stuff we normally have to deal with…”
Introduction
This post explores what I believe to be the current best practices for designing and building APIs that are simple and pleasant for clients to integrate with.
There is an optional Part 1 post that looks at the things that make APIs complex and difficult to use, the issues this causes and why APIs might be built in this way. If you want the full background on this topic then feel free to read that post first. However, the content below can be read standalone if you only want the best practice recommendations.
The contents of this post focus particularly on APIs for invoking web services over HTTP. However, most of the content covered also applies to non web service APIs, such as via queues, events, or even software language APIs for modules or libraries.
Simplicity - The Prime Requirement
The primary thing that makes an API easy and pleasurable to integrate with is simplicity. This doesn’t mean that the API has to be trivial, or that it will lack the power or functionality that clients need to complete their task: it just requires that the amount of complexity in the API is minimised, cleanly abstracted and that everything is organised to maximise understandability.
Let’s consider some of the main things that embody this simplicity principle…
Obvious behaviour
First off, our APIs should behave in an obvious way. This is primarily achieved by having API endpoints that do just one thing, and which are clearly named to describe exactly what that thing is.
Consider, for example, the difference between /services/tkt-engine and
/ticketing/bookReservedTickets or POST (create) /ticketing/bookings?reservation=12345. The first endpoint
has a cryptic name and likely handles a number of different functions and it’s not clear
which will be invoked without looking into the details of the request data.
The other two options are different approaches to URL naming patterns (rpc vs REST), but both
are clear that they will call into some ticketing function and book reserved tickets.
As well as clear naming, it is also important to consider the behaviour that the endpoint implements. In our above example, the endpoint for booking reserved tickets should do just that. It shouldn’t, for example, also involve taking payment. That should be a separate API endpoint (maybe even on a different service) that can be invoked either before or after the booking depending on specific workflow agreements.
Simple domain based requests and responses
Where an endpoint takes a request document (or parameters) and returns a response document, these should be as simple as possible. This doesn’t necessarily mean minimal or overly simplistic, but as simple as can be defined for the task that is to be carried out.
Absolutely they should have all fields named using domain terminology and have meaningful domain values: no cryptically named fields like “RT_BOOK_AMT_2”; and no weird encoded values like “01T7”. Where there are a range of possible field values these should be encoded into an enumerated type with meaningful names from the language of the domain.
Additionally, the requests and responses shouldn’t contain any unused or default value fields - save those for the implementation details. Only fields that the client can actually manipulate should be present. Minimise the use of optional or null-able content unless this concept is explicitly part of the business domain. Avoid future-proofing with additional fields that aren’t currently needed.
Finally, aim to avoid any fields that introduce conditional behaviour into the endpoint. Boolean flag fields or enumerated type fields that subtly change behaviour of an endpoint based on their value should be avoided in preference to separate endpoints for each variation.
Limited scope of required understanding
As well as being as simple as possible, an API should minimise the scope of knowledge required in order to understand and integrate with it. Clients should be able to complete their entire implementation using just the API contracts, documentation and examples supplied with the API.
Where a service is implemented utilising one or more dependent services, it is vital that clients are not required to also have detailed knowledge about how those dependencies work or structure their data.
Details of how an API is implemented should never leak through the API itself. For example, a service that makes use of an underlying database for storing and retrieving data should never expose any database details, table structures, SQL statements or database error codes. It shouldn’t be necessary for a client to understand specifics of an implementation technology or architecture just to use an API.
This also extends to other concerns like reliability and scalability, which should not manifest themselves in the way the API is structured or how it must be used. Concerns that cannot be excluded, such as security, should be a separate element of the API and not entangled with the actual functionality.
Try to avoid wrapping complex workflows behind a single API endpoint. If a client has to refer to a complex set of flow charts to understand what the API will do then its scope is way too big. Better to have multiple smaller endpoints and examples of how to compose them together to complete a specific workflow task. This is much more understandable for the team carrying out the integration.
Simple error status and logic
APIs are always going to fail from time-to-time. As we build more complex distributed systems and depend on increasing numbers of third-party services, then occasional failure is inevitable. Clients integrating with our APIs should be spared from having to deal with complex error scenarios as much as possible.
Ideally any API should just use the idiomatic error handling mechanisms of its underlying protocols. For example, utilising standard HTTP response codes that developers are familiar with and know how to interpret. This also aids with understanding. That said, don’t misuse these well understood codes for other purposes - that’s just confusing.
When additional error statuses are required they should form part of the domain model that the API exposes. Avoid reimplementing or duplicating the idiomatic error patterns inherent in the underlying protocols. Make sure that error codes and messages have actual meaning in the domain language of the service’s bounded context.
The API shouldn’t make it necessary for client implementations to develop complex error handling logic. How to handle any error scenario should be obvious and simple to resolve. Creating lots of conditional error states that involve checking multiple field values is an absolute no-go when the goal is a great client integration experience.
Finally, the service implementation should undertake as much recovery logic as possible rather than expecting the API client to take care of this. When that’s not possible then specific recovery endpoints should be exposed by the API to simplify the client’s task of reversing any partially completed tasks and getting things back to a stable state.
Consistency
The final element that makes things obvious and understandable is consistency. Achieving this across every API endpoint that a service exposes goes a huge way to creating the great integration experiences that we want clients to enjoy.
Consistency starts with naming: sticking to the ubiquitous language of the bounded context; common patterns with how each API is named; correct uses of singular and plural terms; and not using different names for the same things in different places.
It then continues with consistent domain abstractions and models across across each of the API endpoints. As an example, some data element that is returned from from one endpoint and then passed in to another should have the same structure, fields, types and values.
This also extends to consistency in behaviour. Each endpoint should support a similar amount of functionality. They should fail in a similar manner and have similar strategies for handling and recovering from errors.
Learning the details of one endpoint in the API should confer understanding of all the others. Each additional endpoint that a client integrates with should become easier.
API contract with clear documentation
It should go without saying that APIs should always be provided with quality documentation. These should describe all the endpoints, how to use them, what models they take as input and what they return as output. Each of the possible error scenarios should be described, including how to respond and resolve.
If the advice from the previous sections in this post have been followed, then it should be pretty easy to create good documentation. Simple endpoints with obvious behaviour are easier to describe than big, complex ones. Consistency across the API endpoints means documents can be smaller and more concise.
Avoid meaningless descriptions in documentation. A field called “date” with the description “The date” is pointless. Much better is something like “The date and time that the transaction completed processing. In ISO 8601 format, always at UTC offset.”
The very best APIs come supplied with excellent API contracts, such as OpenAPI definitions. These usually consist of automatically generated documentation, often with built-in support for executing test calls to each of the API endpoints. Request and response documents are defined by schemas that form part of the documentation.
In a perfect world it should be possible for clients integrating with the API to just feed the API contracts into generative tools. These create data models and declarative client implementations. The client’s job then just becomes a task of composing the different endpoint calls together; passing data between them, enriching or transforming along the way.
Observability and traceability
The final part of the client integration experience is ensuring that they have a way to observe what’s happening with the API calls that they make. This involves both observability of individual requests and the ability to trace the flow through multiple requests that make up a workflow.
Simple, fast running synchronous API calls may not need much in the way of observability beyond a status code in the response. Those supporting more complex, long-running, or asynchronous operations should provide a way for clients to observe the state of the processing being carried out by the API. This might be by way of informational callbacks, or additional status APIs that the client can call.
Traceability is a vital component as systems grow larger and have many different layers of services connected through various different APIs. Endpoints should always provide the capability for some form of correlation id to be passed in and returned with each response. This should also be linked to any observability features. Wherever supported, the API implementations should pass any traceability data onwards to any dependent services that it calls.
Linking everything up with consistent correlation ids is one of the best ways to make systems operationally easy to manage. It also greatly simplifies the process of providing customer support to clients when things go awry.
Common Misconceptions
We’ve explored the key concepts around the simplicity that helps create good APIs. However, there are three common misconceptions that are often linked to simplicity. Unfortunately when they are applied, the end result is usually the opposite of simplicity: the creation of unintended complexity. It’s important that we cover these ‘anti’ best practices in a little more detail.
Writing the minimum amount of code
As a general rule, writing less code is good: it eases maintenance and often reduces the number of defects in a piece of software. Keeping things lean is an important aim, but often developers go too far in the goals of writing as little code as possible and avoiding duplication.
Each API endpoint should be kept isolated from others as much as possible. Think of each one as its own little module. Try to avoid coupling them together with common base classes or shared data models. Some common functionality (such as Authentication) can be extracted into utility functions but these should be used in a composition only manner.
There’s really no problem with duplicating some code across different API endpoints. Chances are that each endpoint will grow and evolve independently, so things that are common between then now may not be in the future. New versions of an endpoint are often better implemented from a separate copy of the original, rather than adding loads of conditional flags or version numbers to existing, stable code.
Keeping endpoints separate from each other may result in more code initially. Over the lifetime of APIs that want to constantly evolve and change, however, the costs and complexities of maintenance are significantly reduced.
Wanting a one-stop API that does everything
There is sometimes a perception that giving clients a small number of feature rich endpoints is actually easier for them to work with. The argument is that they then have less integration code to write and test. This is wrong.
As we’ve already discussed above, keeping API endpoints simple, obvious and consistent makes them much easier to understand and work with. In the modern world we can generate domain models from API contracts, utilise powerful serialisation libraries and build declarative HTTP clients. It’s so easy now to build API integrations that anything that degrades understandability is the ultimate penalty to productivity.
API elements that introduce the need for conditional logic (flag fields, optionals etc.) means more testing to ensure all the different paths are handled correctly. Generating different client integrations that avoid conditionals is much easier and safer and requires much less testing.
Searching for reuse
Finally, there’s the desire to seek out reuse. This is often closely related to trying to reduce the amount of code that needs to be written and maintained.
There’s nothing wrong with limited reuse, such as for common functionality (back to Authentication again). It just has to be carefully identified and applied. In particular, forcing reuse where things just look similar but actually aren’t is a perfect recipe for a future maintenance nightmare!
The same goes for the API design. Don’t try to force reuse onto the client through shared models with different field variations or combinations of optional values. If integrators want to go down the path of reusability let them discover and evolve this themselves. Never compromise the core simplicity principles for some future reuse that may never be possible or required.
It’s worth noting that by building lots of small, single function API endpoints, we naturally get a level of reuse by being able to combine them in different ways to deliver new functional workflows. There’s no reason to force this.
Domain Abstractions and Models Are Key
So, is there one particular approach that can help us achieve many of the simplicity goals that we have described above? Fortunately there is: using a domain model. We can apply many of the principles of Domain-Driven Design as part of creating this abstraction over our domain.
The first part of this is identifying the bounded context that the service and its API lives within. This then helps identify the ubiquitous language that is used to describe the concepts within this bounded context. We can then use these to build an abstraction and domain model that our API will be based around.
When creating the ubiquitous language and domain model, always do this from the perspective of the client of the API. Better to name and structure things in a way that makes sense to them rather than using internal names or models that are not meaningful outside of the implementation detail.
Using a domain model in this way goes a long way to delivering the understandability and consistency that form a key part of our simplicity goal. Once an integration team understands the domain model then they likely have enough understanding around the behaviour of all of the API endpoints and the requests and responses that are used.
The use of a domain model within a bounded context also helps ensure that we avoid leaking implementation details through the API: this should be impossible if the API endpoints only communicate using their own domain model concepts and terminology.
An additional significant advantage of using a good abstraction of the domain is that it forces consideration around not exposing details and models from dependent services over the API. There may be some dependent services that fall within the same bounded context as our API, but far more likely is that they will be from a different bounded context. When this latter case applies, there is no option other than to map models and behaviour of those dependencies into the domain model of our service and API. How this mapping is achieved is well covered via the DDD concept of an anti-corruption layer, using common patterns such as facades and adapters.
It’s also much easier to provide quality documentation, as this just needs to describe behaviour and models in the domain language that clients likely already comprehend.
Applying CUPID Principles to APIs
We now have all of our best practices for designing and building our APIs, and a suggested approach of building them around domain abstractions and models. But, is there a way that we can evaluate our API design to ensure it complies with these practices?
Fortunately the CUPID properties can be used for exactly this purpose. Let’s look at these five properties, and the common questions we can ask about them, to evaluate our API design…
Composable - plays well with others
- Do all the endpoints use models from the same domain and bounded context?
- Can we take the output of one API endpoint and easily feed it as input into the next one in the workflow?
- Can we achieve this with just a small amount of transformation and enrichment?
- Do all the API endpoints have a consistent error and recovery mechanism, so that clients need only handle these in one place?
Unix Like - does one thing well
- Does each API endpoint implement just a single behaviour?
- Can we reuse endpoints in different combinations to achieve multiple workflow variations?
- Is each API endpoint simple and easy to understand?
- Can we describe what it does in a single sentence?
- Does it avoid mixing other concerns and instead keeps them separate?
Predictable - does what you expect
- Does the API endpoint do exactly what you would expect given its name?
- Do its inputs and outputs avoid using flags and type fields that vary its behaviour?
- Does it avoid existing state impacting how the API endpoint behaves?
- Is the behaviour idempotent whenever possible?
- Is the error handling and recovery obvious and consistent?
- If the API fails or errors, does it do so in an expected way?
Idiomatic - feels natural
- Will a client of the API be able to integrate with it using their standard tools and libraries?
- Does it work in the way they would expect?
- Are all the standard features of any underlying protocols being used correctly?
- Does the API avoid any special features that are not usual for APIs of this type?
Domain-based - models the problem domain in language and structure
- Does the API naming come from the domain language?
- Are the request and response models taken from the domain?
- Is the behaviour implemented taken from the domain?
- Does it avoid exposing any structures or values that exist in a different bounded context?
- Are any error codes or states also taken from the domain?
Summary
This post has looked in some detail about the best practices for designing and building APIs that are simple and easy for clients to integrate with. We’ve looked at some common misconceptions to avoid, and the essential technique of domain modelling to build our APIs around. Finally we’ve explored how we can use the CUPID properties as a way to evaluate our APIs.
By following these best practices you will create APIs that are easy to understand and work with; responsive to change; as simple as possible; and, most importantly, something that your clients will find pleasurable to integrate with.
comments powered by Disqus