How to Deliver High-quality and High-consistency Restful Apis and Supporting Products at Scale

Summary

In recent years, Internet technology has developed rapidly, and the API ecosystem has expanded rapidly. From RPC to REST to GraphQL, different styles and specifications, to a certain extent, are indeed helping developers better develop and use APIs in projects. , but there has not been very substantial progress in how to deliver high-quality, high-consistency APIs (especially RESTful APIs) and supporting products. The author of this article will combine his own development experience and large-scale cloud vendors. Open API design experience, through reading RESTful related papers, exploring typical RESTful style API specifications (Google Cloud and Github's RESTful style API specifications), as well as collaborative development and delivery of RESTful APIs with high consistency in multiple teams The difficulties faced, through the API delivery process, RESTful API style analysis, specification customization, tool construction, etc., to analyze how to deliver high-quality, high-consistency RESTful API and supporting products on a large scale, and the author will Combine with the work experience of Tencent Cloud API team, and taking Alibaba Cloud Serverless API as an example, the research, and discussion on the design of high-quality and high-consistency API specifications and the construction of API publishing platform are carried out.
This article consists of three main chapters, as well as a preface and a summary. The three main chapters are:
Chapter 1: What is RESTful:What is RESTful: This chapter will learn and understand RESTful through research on RESTful papers, typical RESTful-style platforms (Google Cloud, Github), and further exploration of OpenAPI3.0, Swagger, and GraphQL. And discuss "how to standardize a set of RESTful APIs";
Chapter 2: Team Collaboration and High Quality/Consistency: This chapter will focus on building the connection point between internal teams and external users through team collaboration and challenges faced when releasing APIs in a unified manner. to explore and discuss;
Chapter 2: Team Collaboration and High Quality/Consistency: This chapter will focus on building the connection point between internal teams and external users through team collaboration and challenges faced when releasing APIs in a unified manner. to explore and discuss;
Chapter 3: Large-scale API Delivery and Supporting Product Solutions: This chapter will go through the process of large-scale API delivery, RESTful-style specifications, open RESTful-style API specifications/design guidelines, and the construction of an internal unified open API publishing platform We will discuss how to deliver high-quality, high-consistency RESTful APIs on a large scale based on our experience in designing projects such as Explorer and internal tools in Tencent Cloud API team, as well as Alibaba Cloud Serverless-related open APIs as examples. Comprehensive exploration of supporting products;
1 Introduction
This chapter will go through the process of large-scale API delivery, RESTful-style specifications, open RESTful-style API specifications/design guidelines, and the construction of an internal unified open API publishing platform We will discuss how to deliver high-quality, high-consistency RESTful APIs on a large scale based on our experience in designing projects such as Explorer and internal tools in Tencent Cloud API team, as well as Alibaba Cloud Serverless-related open APIs as examples. A comprehensive exploration of supporting products;
If the API is the interface definition for the interaction between modules or subsystems, then the quality of the API design will directly affect the development complexity when the system is connected, as well as the subsequent operation and maintenance complexity ; Similarly, a complete set of standardized The scientific and open API will also have a greater impact on the user's mind, the overall brand image of the product, and the user's sense of trust to a certain extent. In the process of large-scale open API design and external release, there are usually two major challenges:
For large-scale open API products jointly maintained by multiple teams, it is very challenging to achieve high quality and high consistency;
Different from internal APIs, when APIs of cloud products and open APIs of payment products are exposed to the outside world, more efforts should be devoted not only to performance , stability , and security , but also to standardization , science , and overall Sexually enhanced construction; because such APIs usually have the following characteristics:
System-level docking : Once exposed, it is difficult to cancel/change, because it is likely to involve the stability of external user systems;
Huge number of APIs : It usually involves multiple products, so the number of APIs is relatively large, and may even exceed thousands; such a large number of open APIs enables users to quickly discover , understand , use , and debug becomes a very challenging thing;
High consistency and interoperability convenience : usually maintained by multiple business lines; its style and product interoperability between multiple teams need to be unified and guaranteed (including but not limited to a unified SDK , a unified signature method , The same parameters are named uniformly , the style is uniform , etc.);
Stability and security are the lifeline : because it is usually directly related to the user's system, if a problem occurs, it may directly affect the user's system, so the security and stability of this part are like the lifeline, which is very important. There must be a clear protection;
In view of the above challenges and the characteristics of large-scale cloud products /payment APIs, the following points need to be paid attention to when delivering large-scale, high-quality, high-consistency RESTful APIs:
Rules for internal teams:

Perfect open API design guidelines: for example, API specifications based on RESTful style, etc., which will include naming specifications, data structure specifications, error message specifications, etc.;

Scientific open API release process: Once the API is external, it may involve system-level docking, so it is difficult to withdraw and change. When the API is released, there must be a complete process, including but not limited to API entry, pre-release, self-testing, testing, Review, release, and other processes;

Tools/capabilities provided to internal teams:

Complete business abstraction capabilities: such as upper-layer abstraction capabilities such as frequency limitation, unified authentication, and unified disaster tolerance ;

Unified open API publishing tools: a complete internal toolchain for API entry, testing, and publishing, assisting internal teams to publish open APIs more simply, conveniently, and scientifically;

Unified open API release review and inspection work: Provide complete open API pre-launch inspection, testing, and other related solutions;

Products delivered to external users:

Documents in a unified style: It is necessary to provide users with unified and standardized open API interface documents, interface change records, upgrade plans, etc.;

Unified style supporting products: It is necessary to provide users with a unified SDK, API Explorer and other related supporting products;
2 What is RESTful
With the rapid development of Internet technology, although RESTful-style open APIs have become the "de facto standard" of the industry, in fact RESTful describes an architectural style network system, and there is no clear standard, as Wikipedia and Baidu Encyclopedia say Same thing: RESTful is more of a design style. This design style was defined and described by Dr. Roy Fielding, one of the main authors of the HTTP specification, in his doctoral dissertation "Architectural Styles and the Design of Network-based Software Architectures". Among the three current mainstream Web service interaction schemes, REST is simpler and clearer than SOAP (Simple Object Access protocol, Simple Object Access Protocol) and XML-RPC. Whether it is URL processing or Payload encoding, REST is all about It tends to be designed and implemented in a simpler and lighter way. There are also many typical and excellent RESTful API interfaces, such as Gihub, Rapidapi, Google, etc.
2.1 RESTful style and specification
Through the reading and analysis of Roy Fielding's doctoral thesis "Architectural Styles and the Design of Network-based Software Architectures", especially the part of the study on RESTful in Chapter 5, we can see that this paper is only a few styles of RESTful constraints, and no norm-setting was carried out. Therefore, whether in Baidu Encyclopedia or Wikipedia, the related entries of RESTful are very clearly explained: RESTful is a style, not a specification.
In Chapter 5 of Roy Fielding's doctoral dissertation (Representational State Transfer (REST)), Roy Fielding argues that REST provides a set of architectural constraints that, when applied as a whole, emphasize the scalability of component interactions and the generality of interfaces properties, independent deployment of components, and intermediate components used to reduce interaction delays, enhance security, and encapsulate legacy systems. At the same time, this set of architectural constraints is also summarized into six core parts:
client-server

no status

cacheable

layered system

Code on Demand (optional)

unified interface
2.1.1 Common RESTful style specifications
Although the RESTful style is accepted by more people and applied to their own business, because RESTful is not a specification more, because it does not impose strong constraints on the specific expression of the API, the actual production It is difficult to implement RESTful APIs with "high quality and high consistency". At this time, the open API specification/design guide supporting RESTful is particularly important. At present, most developers see the general understanding of the RESTful style including the following parts:
URI specification
Do not use uppercase, lowercase uniformly;
Replace underscores ( _ ) with dashes ( - );
The vocabulary in the URI should use nouns, try not to use verbs; Nouns denote collections of resources, use the plural;
The hierarchy is expressed in the URL, which is used for object navigation according to the entity association relationship, generally according to the id navigation;
Version
The API and version are expressed in the URL, for example:
api.example.com/v1/
www.example.com/api/v1/
ask
GET: safe and idempotent, get representation;
POST: Insecure and not idempotent, use the instance number managed by the server (automatically generated) to create resources, create sub-resources, and partially update resources. If they have not been modified, the resources will not be updated (optimistic lock); PUT: Unsafe but idempotent, create a resource with the instance number managed by the client, update the resource by replacement, and update the resource if it is not modified (optimistic lock);
DELETE: Unsafe but idempotent, deletes resources; PATCH: Update resources on the server (client provides changed properties); status code Use status codes to indicate service status. Common conventions are (only examples are given): 200 (OK): if the existing resource is changed; 404 (not found): The resource does not exist; 503 (Service Unavailable): The service is currently unable to process the request; server response Do not return plain text, it should be a JSON object because this can return standard structured data; When an error occurs, do not return the 200 status code. There is an inappropriate way to return the 200 status code even if an error occurs, and put the error information in the data body; Provide a link, the user of the API may not know how the URL is designed. One solution is to provide relevant links in the response to facilitate the next steps. In this way, users can discover other URLs as long as they remember one URL;
2.1.2 OpenAPI3.0 and Swagger
When it comes to RESTful-style open API specifications, OpenAPI3.0 and Swagger cannot be mentioned. In the official Repo of OpenAPI3.0, we can see this description: The OpenAPI Specification is a community-driven open specification within the OpenAPI Initiative, a Linux Foundation Collaborative Project. this specification is not intended to cover every possible style of HTTP APIs but does include support for REST APIs. Therefore, we can think that RESTful is a style, OpenAPI 3.0 is its latest specification, and Swagger is a related tool for this specification.
The OpenAPI Specification (OAS) defines a standard, language-agnostic RESTful API interface specification that allows both developers and operating systems to view and understand the functionality of service without access to source code, documentation, or network traffic inspection (either It is convenient for humans to learn and read, and it is also convenient for machines to read). When OAS is properly defined, developers can understand and interact with remote services with minimal implementation logic.

In the specification, you can see that many things are clearly stated and defined, such as:

Regarding the version of OpenAPI: The Open API specification uses a version number that conforms to the Semantic Versioning 2.0.0 (server) specification;

Format : A document that conforms to the Open API specification is a self-contained JSON object that can be written in JSON or YAML format;

Document structure: An OpenAPI document can be a single file or can be split into multiple files, and the connected part is determined by the user;

Data type: The original data type in OAS is based on the type supported by JSON Schema Specification Wright Draft 00 ;

Rich text formatting: description fields throughout the specification are marked as supporting CommonMark markdown formatting;

Relative reference of URL : Unless explicitly specified, all attribute values of URL type can be relative addresses, as defined in RFC3986, with Server Object as Base URI;
In addition, the structure and the objects in the structure (such as Server, Paths, etc.) are specified and defined in detail.
2.1.3 Summary
When we carefully read Roy Fielding's doctoral dissertation "Architectural Styles and the Design of Network-based Software Architectures", we will gradually find that RESTful is more often not a standard, nor a technology, service, or component, but a kind of Style, the idea behind RESTful is to use the existing features and capabilities of the Web, and better use some of the guidelines and constraints in existing Web standards.
As he said in the paper: "The purpose of my writing this article is to understand and evaluate the architecture design of network-based application software under the premise of conforming to the architecture principles, and to obtain a powerful, high-performance, An architecture suitable for communication. REST refers to a set of architectural constraints and principles. If an architecture conforms to the constraints and principles of REST, we call it a RESTful architecture.”
With the development of time, RESTful has gradually produced some specifications on top of the style. There are general specifications, universal specifications, and specifications customized by different organizations and different manufacturers according to their own business requirements, but no matter How we can be sure: To have a RESTful style, you must have a complete set of specifications. Especially under the premise of multi-team collaborative development and a large number of interfaces, this specification is obviously more important. If RESTful and specification are compared to human society, I think that RESTful is a moral quality, and OpenAPI is a legal requirement; the quality is uneven. When the time comes, it is up to the law to make the final statute .
2.2 Typical API platform/ecology and RESTful exploration
At present, there are many websites or platforms applying the RESTful style. Typical examples include:

Issue of a repo on Github: https://github.com/OAI/OpenAPI-Specification/issues/2674

Alibaba Cloud Function Compute console: https://fc.console.aliyun.com/fc/service/cn-hangzhou/anycodes-tiku/function/server/overview

Knowing questions and answers: https://www.zhihu.com/question/24795126/answer/41691845
In addition to these very classic websites adopting the RESTful style, many personal website projects are also adopting the RESTful style, such as:

My personal blog, a list of articles under a tag: http://bluo.cn/tags/cold-start
A small program of mine gets the comment content of a topic: https://server.anycodes-tiku.1583208943291465.cn-hangzhou.fc.devsapp.net/api/v1/comments/questions/{question id}
Of course, some open-source frameworks themselves are RESTful by default, for example:

The Django framework in the Python web framework, when the article is processed in the background :

https://www.example.com/admin/article/articlemodel/3/change/
Through these typical RESTful-style websites exemplified above, it is not difficult to find one thing: they seem to be the same, but they are different . E.g:

In the question/answer section of a Github repo's issue and Zhihu, you can see that the former issue is plural, and the latter's question and answer are singular;

In the Python web framework, the background update of the Django framework, you can see that it includes a verb change;

It is not difficult to find one thing at this time, whether it is a RESTful style API designed by individuals or some classic RESTful API cases, they are partially customized specifications based on RESTful constraints, and this customized specification It is often highly compatible with its own business, that is, it will be customized according to its own business characteristics.
2.2.1. Github case
When it comes to RESTful APIs, we have to say GIthub. For the Github we have seen so far, whether it is the path of the website or the open API, it can be considered as a very standard RESTful style, and it can even be considered as almost all of it. The interfaces follow the OpenAPI specification, which can also be confirmed in the Github documentation:
OpenAPI is a standard specification for describing REST APIs. The OpenAPI description allows humans and computers to discover the functionality of an API without first reading the documentation or understanding the implementation . GitHub has exposed its REST API as an OpenAPI 3.0 standard document.
In Github 's REST API , you can clearly feel that everything in REST is considered a kind of asset : when you see any address of Github, or any open API, users can intuitively understand Its corresponding resources, perhaps this is the very important value of the REST style.
Made more stringent specifications for its own open API through several levels such as resources and media types, borrowing the specifications of OpenAPI 3.0 . Through careful reading of this document, it is not difficult to find that its advantages are very obvious:
The subordinate classification of resources is handled very well, which can make developers understand and understand intuitively;

There are object descriptions and corresponding solutions for the changes of various interfaces;

Make full use of request headers, for example, interfaces that use other data formats (not JSON) can be formulated through the Accept header field of HTTP;

The processing of some error codes is more detailed. For example, when accessing private resources without authorization, it is not 401 (Unauthorized) but 404 (not found). The advantage of this is to prevent attackers from easily finding internal resources (according to, Strictly speaking, this is not a very "universal RESTful" specification, this is a customized specification scheme, but it is very meaningful) ;
2.3. RESTful and GraphQL
Even though the RESTful style has been applied on a large scale and has gradually become the de facto standard for Web APIs, it is still imperfect and has huge challenges: Multi-terminal (multiple data interactions) : For example, when obtaining an article, it may be necessary to request multiple interfaces such as GET /posts/ and GET /posts//comments ;
Even though the RESTful style has been applied on a large scale and has gradually become the de facto standard for Web APIs, it is still imperfect and has huge challenges: Multi-terminal (multiple data interactions): For example, when obtaining an article, it may be necessary to request multiple interfaces such as GET /posts/ and GET /posts//comments ; Excessive acquisition of data: For example, when the client already has some data and only needs to acquire part of the content, after requesting the backend, it still obtains all the predetermined data, and there is a problem of excessive acquisition; API version control: When the API is upgraded, it may bring huge migration costs and uncontrollable maintenance risks; Weak type: After a lot of content is passed through the URL, the type may change, it is not easy to specify a specific data type, and there may be certain risks in the processing of the backend; The client's grasp of the data structure is uncontrollable: before requesting an interface, the client cannot specify or clarify the data structure it returns, which leads to over-reliance on the detail of the document; Is difficult to adapt to some specifications: for example, some specifications indicate that the RESTful style does not allow verbs in URLs, but in actual production, once special circumstances occur, it will be difficult to choose; To overcome the related shortcomings of the RESTful style and the OpenAPI specification, Facebook came up with the GraphQL specification. As we all know QL is a query (Query Language), so GraphQL more often refers to the query language of Web API. Compared with RESTful, GraphQL has obvious advantages: Get everything in one request: Usually, a GraphQL service only needs to expose one endpoint so that the client can transmit the necessary query statements to meet the overall data acquisition/query requirements; Client-driven : GraphQL provides a declarative syntax for clients to specify exactly the fields they need. This eliminates the possibility of redundant and insufficient data due to clients declaring their data needs to the GraphQL server based on the schema; API versioning: Since everything in GraphQL is schema-driven, adding new fields doesn't affect existing fields, and GraphQL also provides @deprecated annotations for deprecated fields, so extending GraphQL is not a problem. This removes the need for API versioning; Strong typing: Unlike RESTful, GraphQL can specify the data type, or enforce the specification of the data type;
2.4. How to standardize a RESTful API
Before preparing to define a RESTful-style open API specification, in addition to understanding how typical RESTful cases are specified, you must also have a deep understanding of your own business, and make some specializations according to your own business situation. Processing , such as Google Cloud's API, integrates certain specifications of RESTful style and gRPC according to network requirements ; Github stipulates that from the perspective of security, unauthorized viewing of certain Repo effects means that resources do not exist.
When customizing RESTful style open API specifications, you can fully refer to Google Cloud's open API design guidelines, through resource-related definitions, method-related definitions, standardization of fields, definition of errors, and abstraction of general capabilities, Several aspects such as security and permissions are defined in detail. At the same time, the following principles should be followed:
Specification customization purpose: to help developers design simple, consistent, and easy-to-use network APIs;

Specification customization principle:

Specification documents should be comprehensive, accurate and unambiguous (use requirement-level keywords ("must", "must not", "required", "should", "should not", "should", "not SHOULD", "RECOMMENDED", "MAY" and "OPTIONAL"), and interpreted as described in RFC 2119 );

The six constraints of the RESTful style should be used as the basic criteria, the universal specification or the OpenAPI 3.0 specification should be used as a reference or basis, the existing cognition should not be overturned on a large scale, and a specific specification model can be appropriately designed according to business requirements. ;

The specification document should be extensible, and over time, new styles and design patterns should be adopted and approved to add relevant content to the specification document. In this spirit, continue to improve the specification documentation and provide ample room for the art and craft of API design;

When customizing the specification document, it is necessary to fully consider the quality of developers and the differences in the business understanding of different people. On the premise of not affecting the overall expression of the specification, and on the premise of ensuring high quality and high consistency, it must provide a certain degree of fault tolerance. Schemes, even fault-tolerant schemes, should provide complete rules, such as naming rules, expression rules, structure rules, etc.;
Universality of the specification:
The primary purpose is to regulate the development and exposure of open APIs for the developers of the internal collaborative development team to ensure the unity and standardization of open APIs;

Secondly, it is necessary to expose it to the outside world, so that users of the open API can fully understand the design purpose, concept, and specifications of this set of open APIs;

Finally, an influential, universal, and typical RESTful style open API specification should be formed. For example, Google Cloud's open API design guide can be called the best practice in this field and has a certain reference. and learning is also extremely influential;

Related Articles

Explore More Special Offers

  1. Short Message Service(SMS) & Mail Service

    50,000 email package starts as low as USD 1.99, 120 short messages start at only USD 1.00