How to Deliver High-quality, High-consistency Restful Apis and Supporting Products at Scale

RESTful APIs Introduction:

RESTful APIs.In recent years, Internet technology has developed rapidly, and the API ecosystem has expanded rapidly. From RPC to REST to GraphQL, different styles and specifications, to a certain extent, are indeed helping developers better develop and use APIs in projects. , but there has not been very substantial progress in how to deliver high-quality, high-consistency APIs (especially RESTful APIs) and supporting products. The author of this article will combine his own development experience and large-scale cloud vendors. Open API design experience, through reading RESTful related papers, exploring typical RESTful style API specifications (Google Cloud and Github's RESTful style API specifications), as well as collaborative development and delivery of RESTful APIs with high consistency in multiple teams difficulties faced.

RESTful APIs Summary

RESTful APIs.In recent years, Internet technology has developed rapidly, and the API ecosystem has expanded rapidly. From RPC to REST to GraphQL, different styles and specifications, to a certain extent, are indeed helping developers better develop and use APIs in projects. , but there has not been very substantial progress in how to deliver high-quality, high-consistency APIs (especially RESTful APIs) and supporting products. The author of this article will combine his own development experience and large-scale cloud vendors. Open API design experience, through reading RESTful related papers, exploring typical RESTful style API specifications (Google Cloud and Github's RESTful style API specifications), as well as collaborative development and delivery of RESTful APIs with high consistency in multiple teams The difficulties faced, through the API delivery process, RESTful API style analysis, specification customization, tool construction, etc., to analyze how to deliver high-quality, high-consistency RESTful API and supporting products on a large scale, and the author will Combined with the work experience of Tencent Cloud API team, and taking Alibaba Cloud Serverless API as an example, the research and discussion on the design of high-quality and high-consistency API specifications and the construction of API publishing platform are carried out.

This article consists of three main chapters, as well as a preface and a summary. The three main chapters are:

Chapter 1: RESTful APIs.What is RESTful:

This chapter will learn and understand RESTful through research on RESTful papers, typical RESTful-style platforms (Google Cloud, Github), and further exploration of OpenAPI3.0, Swagger, and GraphQL. And discuss "how to standardize a set of RESTful APIs";

Chapter 2: RESTful APIs.Team Collaboration and High Quality /Consistency:

This chapter will focus on building the connection point between internal teams and external users through team collaboration and challenges faced when releasing APIs in a unified manner. to explore and discuss;

Chapter 3: Large-scale API Delivery and Supporting Product Solutions:

This chapter will go through the process of large-scale API delivery, RESTful-style specifications, open RESTful-style API specifications/design guidelines, and the construction of an internal unified open API publishing platform We will discuss how to deliver high-quality, high-consistency RESTful APIs on a large scale based on our experience in designing projects such as Explorer and internal tools in Tencent Cloud API team, as well as Alibaba Cloud Serverless-related open APIs as examples. Comprehensive exploration of supporting products;

1 RESTful APIs.Introduction

If the API is the interface definition for the interaction between modules or subsystems, then the quality of the API design will directly affect the development complexity when the system is connected, as well as the subsequent operation and maintenance complexity ; Similarly, a complete set of standardized The scientific and open API will also have a greater impact on the user's mind, the overall brand image of the product, and the user's sense of trust to a certain extent. In the process of large-scale open API design and external release, there are usually two major challenges:

For large-scale open API products jointly maintained by multiple teams, it is very challenging to achieve high quality and high consistency;

Different from internal APIs, when APIs of cloud products and open APIs of payment products are exposed to the outside world, more efforts should be devoted not only to performance , stability , and security , but also to standardization , science , and overall Sexually enhanced construction; because such APIs usually have the following characteristics:

System-level docking : Once exposed, it is difficult to cancel/change, because it is likely to involve the stability of external user systems;

Huge number of APIs : It usually involves multiple products, so the number of APIs is relatively large, and may even exceed thousands; such a large number of open APIs enables users to quickly discover , understand , use , and debug becomes a very challenging thing;

High consistency and interoperability convenience : usually maintained by multiple business lines; its style and product interoperability between multiple teams need to be unified and guaranteed (including but not limited to a unified SDK , a unified signature method , The same parameters are named uniformly , the style is uniform , etc.);

RESTful APIs.Stability and security are the lifeline : because it is usually directly related to the user's system, if a problem occurs, it may directly affect the user's system, so the security and stability of this part are like the lifeline, which is very important. There must be a clear protection;
In view of the above challenges and the characteristics of large-scale cloud products /payment APIs, the following points need to be paid attention to when delivering large-scale, high-quality, high-consistency RESTful APIs:

Rules for internal teams:

RESTful APIs.Perfect open API design guidelines: for example, API specifications based on RESTful style, etc., which will include naming specifications, data structure specifications, error message specifications, etc.;

Scientific open API release process: Once the API is external, it may involve system-level docking, so it is difficult to withdraw and change. When the API is released, there must be a complete process, including but not limited to API entry, pre-release, self-testing, testing, Review, release and other processes;

Tools/capabilities provided to internal teams:
Complete business abstraction capabilities: such as upper-layer abstraction capabilities such as frequency limitation, unified authentication, and unified disaster tolerance ;

Unified open API publishing tools: a complete internal tool chain for API entry, testing, and publishing, assisting internal teams to publish open APIs more simply, conveniently and scientifically;

Unified open API release review and inspection work: Provide complete open API pre-launch inspection, testing and other related solutions;

Products delivered to external users:
Documents in a unified style: It is necessary to provide users with unified and standardized open API interface documents, interface change records, upgrade plans, etc.;
Unified style supporting products: It is necessary to provide users with a unified SDK, API Explorer and other related supporting products;

2 RESTful APIs.What is RESTful

With the rapid development of Internet technology, although RESTful-style open APIs have become the "de facto standard" in the industry, in fact RESTful describes an architectural style network system, and there is no clear standard, as Wikipedia and Baidu Encyclopedia say Same thing: RESTful is more of a design style. This design style was defined and described by Dr. Roy Fielding, one of the main authors of the HTTP specification, in his doctoral dissertation "Architectural Styles and the Design of Network-based Software Architectures". Among the three current mainstream Web service interaction schemes, REST is simpler and clearer than SOAP (Simple Object Access protocol, Simple Object Access Protocol) and XML-RPC. Whether it is URL processing or Payload encoding, REST is all about It tends to be designed and implemented in a simpler and lighter way. There are also many typical and excellent RESTful API interfaces, such as Gihub, Rapidapi, Google, etc.

2.1 RESTful APIs.RESTful style and specification

Through the reading and analysis of Roy Fielding's doctoral thesis "Architectural Styles and the Design of Network-based Software Architectures", especially the part of the study on RESTful in Chapter 5, we can see that this paper is only a few styles of RESTful constraints, and no norm-setting was carried out. Therefore, whether in Baidu Encyclopedia or Wikipedia, the related entries of RESTful are very clearly explained: RESTful is a style, not a specification.

In Chapter 5 of Roy Fielding's doctoral dissertation (Representational State Transfer (REST)), Roy Fielding argues that REST provides a set of architectural constraints that, when applied as a whole, emphasize the scalability of component interactions and the generality of interfaces properties, independent deployment of components, and intermediate components used to reduce interaction delays, enhance security, and encapsulate legacy systems. At the same time, this set of architectural constraints is also summarized into six core parts:

•no status
•layered system
•Code on Demand (optional)
•unified interface

2.1.1 RESTful APIs.Common RESTful style specifications

Although the RESTful style is accepted by more people and applied to their own business, because RESTful is not a specification more, because it does not impose strong constraints on the specific expression of the API, the actual production It is difficult to implement RESTful APIs with "high quality and high consistency". At this time, the open API specification/design guide supporting RESTful is particularly important. At present, most developers see the general understanding of the RESTful style including the following parts:
URI specification
•Do not use uppercase, lowercase uniformly;
•Replace underscores ( _ ) with dashes ( - );
•The vocabulary in the URI should use nouns, try not to use verbs;
•Nouns denote collections of resources, use the plural;
•The hierarchy is expressed in the URL, which is used for object navigation according to the entity association relationship, generally according to the id navigation;
•The API and version are expressed in the URL, for example:
•GET: safe and idempotent , get representation;
•POST: Insecure and not idempotent , use the instance number managed by the server (automatically generated) to create resources, create sub-resources, and partially update resources. If they have not been modified, the resources will not be updated (optimistic lock);
•PUT: Unsafe but idempotent , create a resource with the instance number managed by the client, update the resource by replacement, and update the resource if it is not modified (optimistic lock);
•DELETE: Unsafe but idempotent , deletes resources;
•PATCH: Update resources on the server (client provides changed properties);
•status code
•Use status codes to indicate service status. Common conventions are (only examples are given):
•200 (OK): if the existing resource is changed;
•404 (not found): The resource does not exist;
•503 (Service Unavailable): The service is currently unable to process the request;
•server response
•Do not return plain text, it should be a JSON object, because this can return standard structured data;
•When an error occurs, do not return the 200 status code. There is an inappropriate way to return the 200 status code even if an error occurs, and put the error information in the data body;
•Provide a link, the user of the API may not know how the URL is designed. One solution is to provide relevant links in the response to facilitate the next steps. In this way, users can discover other URLs as long as they remember one URL;

2.1.2 RESTful APIs.OpenAPI3.0 and Swagger

When it comes to RESTful-style open API specifications, OpenAPI3.0 and Swagger cannot be mentioned. In the official Repo of OpenAPI3.0 , we can see this description:

The OpenAPI Specification is a community-driven open specification within the OpenAPI Initiative, a Linux Foundation Collaborative Project.
this specification is not intended to cover every possible style of HTTP APIs, but does include support for REST APIs.

Therefore, we can think that RESTful is a style, OpenAPI 3.0 is its latest specification, and Swagger is a related tool for this specification .
The OpenAPI Specification (OAS) defines a standard, language-agnostic RESTful API interface specification that allows both developers and operating systems to view and understand the functionality of a service without access to source code, documentation, or network traffic inspection (either It is convenient for humans to learn and read, and it is also convenient for machines to read). When OAS is properly defined, developers can understand and interact with remote services with minimal implementation logic.

In the specification, you can see that many things are clearly stated and defined, such as:

Regarding the version of OpenAPI : The Open API specification uses a version number that conforms to the Semantic Versioning 2.0.0 (semver) specification;
Format : A document that conforms to the Open API specification is a self-contained JSON object that can be written in JSON or YAML format;
Document structure : An OpenAPI document can be a single file or can be split into multiple files, and the connected part is determined by the user.

Data type : The original data type in OAS is based on the type supported by JSON Schema Specification Wright Draft 00 ;
Rich text formatting : description fields throughout the specification are marked as supporting CommonMark markdown formatting;
Relative reference of URL : Unless explicitly specified, all attribute values of URL type can be relative addresses, as defined in RFC3986, with Server Object as Base URI;
In addition, the structure and the objects in the structure (such as Server, Paths, etc.) are specified and defined in detail.

2.1.3 RESTful APIs.Summary

When we carefully read Roy Fielding's doctoral dissertation "Architectural Styles and the Design of Network-based Software Architectures", we will gradually find that RESTful is more often not a standard, nor a technology, service, or component, but a kind of Style, the idea behind RESTful is to use the existing features and capabilities of the Web, and better use some of the guidelines and constraints in existing Web standards.
As he said in the paper: "The purpose of my writing this article is to understand and evaluate the architecture design of network-based application software under the premise of conforming to the architecture principles, and to obtain a powerful, high-performance, An architecture suitable for communication. REST refers to a set of architectural constraints and principles. If an architecture conforms to the constraints and principles of REST, we call it a RESTful architecture.”
With the development of time, RESTful has gradually produced some specifications on top of the style. There are general specifications, universal specifications, and specifications customized by different organizations and different manufacturers according to their own business requirements, but no matter How we can be sure: To have a RESTful style, you must have a complete set of specifications. Especially under the premise of multi-team collaborative development and a large number of interfaces, this specification is obviously more important. If RESTful and specification are compared to human society, I think that RESTful is a moral quality, and OpenAPI is a legal requirement; the quality is uneven. When the time comes, it is up to the law to make the final statute .

2.2 RESTful APIs.Typical API platform/ecology and RESTful exploration

At present, there are many websites or platforms applying RESTful style. Typical examples include:
•Issue of a repo on Github :
•Alibaba Cloud Function Compute console :
•Knowing questions and answers :
In addition to these very classic websites adopting the RESTful style, many personal website projects are also adopting the RESTful style, such as:
•My personal blog, a list of articles under a tag :
•A small program of mine gets the comment content of a topic :{question id}
Of course, some open source frameworks themselves are RESTful by default, for example:
•The Django framework in the Python web framework, when the article is processed in the background :
Through these typical RESTful-style websites exemplified above, it is not difficult to find one thing: they seem to be the same, but they are different . E.g:
•In the question/answer section of a Github repo's issue and Zhihu, you can see that the former issue is plural, and the latter's question and answer are singular;
•In the Python web framework, the background update of the Django framework, you can see that it includes a verb change;
It is not difficult to find one thing at this time, whether it is a RESTful style API designed by individuals or some classic RESTful API cases, they are partially customized specifications based on RESTful constraints, and this customized specification It is often highly compatible with its own business, that is, it will be customized according to its own business characteristics.

2.2.1. RESTful APIs.Github case

RESTful APIs.When it comes to RESTful APIs, we have to say GIthub. For the Github we have seen so far, whether it is the path of the website or the open API, it can be considered as a very standard RESTful style, and it can even be considered as almost all of it. The interfaces follow the OpenAPI specification, which can also be confirmed in the Github documentation:
OpenAPI is a standard specification for describing REST APIs. The OpenAPI description allows humans and computers to discover the functionality of an API without first reading the documentation or understanding the implementation . GitHub has exposed its REST API as an OpenAPI 3.0 standard document.
In Github 's REST API , you can clearly feel that everything in REST is considered a kind of asset : when you see any address of Github, or any open API, users can intuitively understand Its corresponding resources, perhaps this is the very important value of the REST style.
made more stringent specifications for its own open API through several levels such as resources and media types, borrowing the specifications of OpenAPI 3.0 . Through careful reading of this document, it is not difficult to find that its advantages are very obvious:
•The subordinate classification of resources is handled very well, which can make developers understand and understand intuitively;
•There are object descriptions and corresponding solutions for the changes of various interfaces;
•Make full use of request headers, for example, interfaces that use other data formats (not json) can be formulated through the Accept header field of http;
•The processing of some error codes is more detailed. For example, when accessing private resources without authorization, it is not 401 (Unauthorized) but 404 (not found). The advantage of this is to prevent attackers from easily finding internal resources (according to Strictly speaking, this is not a very "universal RESTful" specification, this is a customized specification scheme, but it is very meaningful) ;

2.2.2. RESTful APIs.Google Cloud case

If Github's RESTful API specification is further customized and improved on the basis of OpenAPI 3.0, and is a method to quickly form its own specification by borrowing existing specifications, then Google Cloud's open API specification is a very creative Its own gRPC and RESTful have been organically combined, and a unique but very classic open API design guide has been born .
of resource-oriented design is very clearly defined . It also directly indicates that this design document is a design guide customized by Google Cloud according to some situations, that is, it meets the RESTful specification, but it is actually a specification defined by itself:
RPC APIs are usually designed in terms of interfaces and methods. Over time, there are more and more interfaces and methods, and the end result can be a large and confusing API interface, as the developer has to learn each method individually. Obviously, this is time-consuming and error-prone. The REST architectural style was introduced primarily for use with HTTP/1.1, but it also helps with this problem. HTTP REST APIs have been a huge success on the internet. In 2010, approximately 74% of public web APIs were HTTP REST (or REST-like) APIs, and most APIs used JSON as the transport format. Although HTTP/JSON APIs are very popular on the Internet, they carry less traffic than traditional RPC APIs. For example, about half of all internet traffic in the US during peak hours is video content, and apparently few people use the HTTP/JSON API to deliver such content for performance reasons. Within the data center, many companies use socket-based RPC APIs to carry most network traffic, which can involve orders of magnitude more data (in bytes) than public HTTP/JSON APIs.
The reason why Google Cloud's open API design guide is unique, but very classic , is because in this guide, not only can you see the detailed design of the typical RESTful style open API specification, but you can also clearly see why This design, what is the value of this design, it can be considered that this API practical guide is very valuable for learning and research.
In this design guide, you can see some of Google's focus in the open API design process:
•Its core principle is to define named resources that can be controlled in a small number of ways. These resources and methods are called API "nouns" and "verbs";
•When using the HTTP protocol, resource names are naturally mapped to URLs, and methods are naturally mapped to HTTP's POST, GET, PUT, PATCH, and DELETE;
•A collection contains a list of resources of the same type;
•A resource has some state and zero or more child resources. Each sub-resource can be a simple resource or a collection resource;
In the guide, the steps to take when designing a resource-oriented API are also very clearly described:

•Determine the type of resource provided by the API ;
•Identify relationships between resources ;
•Determine resource name schemes based on types and relationships ;
•Determine the resource architecture ;
•attach a minimal set of methods to the resource ;
In addition, this design guide also defines some rules, such as some naming rules. Taking the naming rule as an example, you can see what Google Cloud considered when defining this rule :
•Have a unified goal : simple, intuitive, consistent;
•To have certain specifications for the target :
•Avoid name overload and use different names for different concepts;
•In order to simplify naming, widely accepted abbreviations or abbreviations may be used. For example, API is better than Application Programming Interface ;
•Except for field names and enumeration values , all definitions in the .proto file must use names in the UpperCamelCase format defined by the Google Java style ;
•To be specific to special cases, consider the quality of developers, etc .: Since many developers are not native English speakers, one of the goals of these naming conventions is to ensure that the API is easily understood by most developers . For methods and resources, we encourage the use of simple, consistent and minimal vocabulary for naming;
With goals, specifications, cases, and humanitarianism , Google Cloud's design guidelines for open API documents can be regarded as an important reference for many developers when designing APIs, and can also be called RESTful open APIs A classic case of design documentation.

2.2.3. Summary

Although there are some universal RESTful-style open API specifications, and some organizations promote the construction of RESTful-style specifications (such as OpenAPI 3.0, etc.), in actual production, if you want to deliver high-quality, high-consistency on a large scale RESTful style API, it is still very necessary to develop a set of RESTful style specifications for the business itself; whether it is similar to the Github solution, part of the "upgrade" based on OpenAPI 3.0; or Google Cloud's "self-contained faction", according to its own Requirements, organic combination of gRPC and RESTful; RESTful-style open API specifications like this are always needed.
moral quality for developers or development teams , then the norms formulated on top of this style are the real laws . Only when everyone has moral quality and has the law as a support, can they Ensure high-quality, high-consistency RESTful API delivery.
and GraphQL
Even though the RESTful style has been applied on a large scale and has gradually become the de facto standard for Web APIs, it is still imperfect and has huge challenges:
•Multi-terminal (multiple data interactions) : For example, when obtaining an article, it may be necessary to request multiple interfaces such as GET /posts/ and GET /posts//comments ;
•Excessive acquisition of data : For example, when the client already has some data and only needs to acquire part of the content, after requesting the backend, it still obtains all the predetermined data, and there is a problem of excessive acquisition;
•API version control : When the API is upgraded, it may bring huge migration costs and uncontrollable maintenance risks;
•Weak type : After a lot of content is passed through the URL, the type may change, it is not easy to specify a specific data type, and there may be certain risks in the processing of the backend;
•The client's grasp of the data structure is uncontrollable : before requesting an interface, the client cannot specify or clarify the data structure it returns, which leads to over-reliance on the detail of the document;
•is difficult to adapt to some specifications : for example, some specifications indicate that RESTful style does not allow verbs in urls, but in actual production, once special circumstances occur, it will be difficult to choose;
To overcome the related shortcomings of the RESTful style and the OpenAPI specification, Facebook came up with the GraphQL specification. As we all know QL is a query (Query Language), so GraphQL more often refers to the query language of Web API. Compared with RESTful, GraphQL has obvious advantages:
•Get everything in one request : Usually, a GraphQL service only needs to expose one endpoint so that the client can transmit the necessary query statements to meet the overall data acquisition/query requirements;
•Client-driven : GraphQL provides a declarative syntax for clients to specify exactly the fields they need. This eliminates the possibility of redundant and insufficient data due to clients declaring their data needs to the GraphQL server based on the schema;
•API versioning : Since everything in GraphQL is schema driven, adding new fields doesn't affect existing fields, and GraphQL also provides @deprecated annotations for deprecated fields, so extending GraphQL is not a problem. This removes the need for API versioning;
•Strong typing : Unlike RESTful, GraphQL can specify the data type, or enforce the specification of the data type;
•Transport layer agnostic : This is a really great benefit of GraphQL. The API server can exchange information through protocols like HTTP, HTTPS, WebSockets, TCP, UDP, etc. This is because GraphQL cares little about how information is exchanged between client and server;
Of course, GraphQL is not perfect at present, and it also has some shortcomings:
•caching function is immature : browser and mobile phone caching are not supported, which is different from RESTful services that use the local HTTP caching mechanism;
•Verification and Error Reporting : RESTful services utilize HTTP status codes to handle different errors that may be encountered. For developers, this makes validation of APIs very simple and painless, but using GraphQL services always returns a 200 OK response;
•Exposure schemas and resource attacks : Unlike RESTful services, GraphQL services require clients to know the schema of the data to be queried; if you expose your API to a third party, you are basically exposing your internal data structures; you must be very careful because clients The client can initiate a connection query without a high cost, which may lead to a denial of service (DoS) attack on the server;
•Security - Authentication and Authorization : The GraphQL community is still confused about how to handle the security part of a GraphQL service;
•N+1 query problem : In RESTful services, it is easy to record the executed SQL query and further optimize it, but in the case of GraphQL, the parsing nature is dynamic, so it is difficult to get an exact query and further optimize it;
Although GraphQL is not perfect, it does not affect its gradual application by more people, platforms, and manufacturers. For example, Github currently fully supports GraphQL on the v4 version of the API. Of course, it is also because the RESTful style and GraphQL have a complementary relationship to a certain extent, so when many open APIs determine the form, they will discuss which scheme/standard/style to use. In fact, from a personal point of view:
•Proceed with a RESTful solution;
•In the later stage, if GraphQL is gradually more mature and the problems of security, exposure mode and resource attack are gradually solved, it can support two schemes/standards/styles at the same time like Github;

2.4. RESTful APIs.How to standardize a RESTful API

Before preparing to define a RESTful-style open API specification, in addition to understanding how typical RESTful cases are specified, you must also have a deep understanding of your own business, and make some specializations according to your own business situation. Processing , such as Google Cloud's API, integrates certain specifications of RESTful style and gRPC according to network requirements ; Github stipulates that from the perspective of security, unauthorized viewing of certain Repo effects means that resources do not exist.
When customizing RESTful style open API specifications, you can fully refer to Google Cloud's open API design guidelines, through resource-related definitions, method-related definitions, standardization of fields, definition of errors, and abstraction of general capabilities, Several aspects such as security and permissions are defined in detail. At the same time, the following principles should be followed:
•Specification customization purpose: to help developers design simple, consistent and easy -to-use network APIs;
•Specification customization principle:
•Specification documents should be comprehensive, accurate and unambiguous (use requirement-level keywords ("must", "must not", "required", "should", "should not", "should", "not SHOULD", "RECOMMENDED", "MAY" and "OPTIONAL"), and interpreted as described in RFC 2119 );
•The six constraints of the RESTful style should be used as the basic criteria, the universal specification or the OpenAPI 3.0 specification should be used as a reference or basis, and the existing cognition should not be overturned on a large scale, and a specific specification model can be appropriately designed according to business requirements. ;
•The specification document should be extensible, and over time, new styles and design patterns should be adopted and approved to add relevant content to the specification document. In this spirit, continue to improve the specification documentation and provide ample room for the art and craft of API design;

•When customizing the specification document, it is necessary to fully consider the quality of developers and the differences in business understanding of different people. On the premise of not affecting the overall expression of the specification, and on the premise of ensuring high quality and high consistency, it must provide a certain degree of fault tolerance. Schemes, even fault-tolerant schemes, should provide complete rules, such as naming rules, expression rules, structure rules, etc.;

•Universality of the specification:
•The primary purpose is to regulate the development and exposure of open APIs for the developers of the internal collaborative development team to ensure the unity and standardization of open APIs;
•Secondly, it is necessary to expose it to the outside world, so that users of the open API can fully understand the design purpose, concept and specifications of this set of open APIs;
•Finally, an influential, universal, and typical RESTful style open API specification should be formed. For example, Google Cloud's open API design guide can be called the best practice in this field and has a certain reference. and learning, are also extremely influential;
3 Teamwork and high quality/high consistency
When a project or a product decides to open an API to the outside world, it usually encounters a very challenging problem: how to ensure the quality and consistency of the open API? Such challenges are especially huge among cloud vendors. Whether it is Tencent Cloud or Alibaba Cloud, when exposing high-quality and highly consistent APIs , they are faced with very serious quality and consistency issues. The export of the open API exposed to the outside world is unified and converged to the same team, with strong regulations and strong restrictions, such as Tencent Cloud API team, Alibaba Cloud's POP team, etc.
So under the premise of team collaboration, how to ensure the high quality and high consistency of open APIs in the process of releasing open APIs to the outside world? Can this problem be solved with a complete set of specifications? Why should Tencent Cloud and Alibaba Cloud converge to a team as a unified exit under the premise of unified specifications? In addition to a clear style, a clear process, and a perfect specification, should there be supporting tools to assist in the release of high-quality, high-consistency APIs ?
3.1 Conflict and integration of team collaboration and API ecology
It is normal for a product/project to be implemented by multiple teams, but this collaboration often brings about a "non-integration" of the open API ecosystem. There are many reasons for this:
•Different members have different understandings of business ;
•The business functions that different teams are responsible for are different, and the concerns of different teams are not synchronized ;
•Different developers have different qualities (including work attitude, English level and personal character, etc.) ;
the real thing that the author is a cloud API product manager in a cloud as an example: Different cloud products have interfaces to obtain instances, but even if the same instance is obtained, the perception and definition of it are different in different teams; The behavior defined by team A is GetInstance , and the behavior defined by team B is DescribeInstance . It is possible to understand both from the perspective of different people. There is no right or wrong, but if you are a cloud API team, Under the premise of uniformly exposing the API, it is unreasonable to have two forms of expression, not to mention that in cloud vendors, different products have interfaces to obtain instances , so discarding the GetInstance and DescribeInstance just mentioned is Isn't it possible that there are more forms such as InstanceInformation and ObtainInstance ? If a developer uses the open API of a cloud product , the behavior of the same interface of different products is completely different, wouldn’t it make the developer feel that the cloud vendor is unprofessional, the content is messy, and casual; of trust. The same thing, also research and think in Github's API:
•When to use edit and when to use update ;
•When to use delete and when to use remove ;
From the above real cases, we can find that team collaboration is inevitable. How to maintain a high degree of consistency for the products after collaboration, especially for API products that are open to the outside world, before there is no absolute closure, this will be "collaboration" and "" The huge conflict of "unification", so how to solve this conflict has become a huge challenge for most cross-team collaboration products/projects that open APIs to the outside world.
Both Alibaba Cloud's POP team and Tencent Cloud API team have adopted both soft and hard solutions when resolving this conflict . Therefore, "soft" means subconscious training and clear norms; the so-called "Hard" means that there must be a unified publishing platform with mandatory closing to ensure the unity, compatibility, security and high quality of the final release. Taking a RESTful open API as an example, there are three levels of requirements:
•Subconscious : To define APIs in a RESTful style
•Normative level : There needs to be a very clear design guideline, which is not only internal, but also external; and this guideline cannot be ambiguous;
•Mandatory closing : The so-called mandatory closing is the last line of defense for us to expose APIs to the outside world. This line of defense requires manual review. To ensure the overall consistency, it is necessary to converge as much as possible to one part for unified review;
To sum up, if you want to resolve conflicts, weaken challenges, make team collaboration and high quality/high consistency no longer conflict, and integrate organically, you must explore the above three aspects: subconsciousness, strong norms , and forced closing.
3.2 The style and norms of team collaboration
The so-called style and specification of teamwork is something that needs to be paid attention to before and during the design of an open API.
3.2.1 Style is clear
Before the API is ready to be developed/upgraded, it is first necessary to clarify the open API style exposed to the outside world, such as RESTful, GraphQL, or other specifications/standards, such as RPC. Although the design guidelines for open APIs must remain active to a certain extent, and are constantly compatible with updated specifications (for example, Github's open API supports RESTful style in v3 and GraphQL in v4), it still needs to be designed at the beginning of the design. To clarify the style of the current version, this article will take the RESTful style as the main direction of exploration. In the future, if GraphQL develops and iterates more rapidly, users' demands will gradually become stronger, and they can lace with Github and consider the solution of RESTful style GraphQL.
3.2.2 Specification customization
The formulation of the specification is after the style is determined. Taking the RESTful style as an example, it is necessary to formulate several levels of specifications such as resource-related, method-related, field standardization, and errors. Resource-related definitions

Because RESTful is resource-oriented to a certain extent, it is necessary to clarify the details of resources when formulating specifications: the types and relationships of resources; the upper-level abstraction of resources, the unification of similar resources under multiple products/businesses; the definition specification of custom resources; The specific expressions are as follows:
•Resource content sorting : especially for resources that may be expressed in the same way by different teams. For example, taking cloud products as an example, the id of a machine may be defined as instanceId at the cloud host , and a data may be defined as resourceId at the database. , at this time, it is necessary to evaluate whether these two represent the same kind of content, both of which are the id of resources or instances. If it can be determined that they are of the same kind, their resource names should be specified in the upper layer, for example, they are called instance or resource , etc., to avoid Similar concepts appear in different interfaces, and different problems appear;
•Determination of resource affiliation : According to the logic, from left to right , the set should gradually include resources, so when defining resources, it is necessary to clarify the affiliation of resources, to avoid the occurrence that the x module viewed by A includes the y module , while B is The y module includes the x module , which is a conflict of inclusion relationships between products. For example, a product definition includes a project under a service , while another product includes a service under the project . At this time, concept fusion and adjustment must be carried out to achieve External unified expression;
•Resource naming convention : Refers to whether verbs can be used, when to use verbs, what linking symbols are used, and the longest name for each resource; for example, viewing all books in a computer category can be expressed as //api.example .com/v1/category/computer/books ;
•Resource expression specification : whether to use a resource name or resource id; for example, the example of using resource name in 1.1 can also be expressed by resource id: // , but what? When to use id and when to use name, this needs to be clearly defined;
•Determination of certain parameters : It is impossible for the business to completely determine all functions through the URI, such as query operations, it may be designed as //{keywords} , at this time also Some general parameters need to be specified; in addition, some request methods (such as POST), etc., may also need to be clear and standardized for some data structures;
•Clarify the custom content : provide as many examples or cases as possible, but there are often some very special cases in the specification, so at this time, it is necessary to make additional agreements for some customizable content of the business team . Such as the situation described in Google Cloud: Since many developers are not native English speakers, one of the goals of these naming conventions is to ensure that the API can be easily understood by most developers . For methods and resources, we encourage the use of simple, consistent and minimal vocabulary for naming; Method-Related Definitions

In addition to the clear definition of resource-related content, it is also necessary to abstract some common methods, and to agree on some custom methods, and also map related methods to HTTP methods:
•The core method of the specification: When formulating the specification, it is necessary to specify the common methods that all teams can use, such as List, Get, Create, Delete, Start, Restart, Stop, Update and other methods; at the same time, the mapping relationship between these methods and HTTP must be clear For example, the standard methods List and Get are mapped to HTTP, Get, Create is mapped to HTTP, POST, etc.;
•Specification of custom methods : Even after we have specified a large number of core methods and abstracted many general methods, in actual production, because the business handled by each team is complex, there will still be some custom methods. There are two workarounds:
•Similar to Google Cloud's solution: that is, directly in the URL: ;
•Put the specified method in the Header: for example, specify Action: customVerb in the Header ; Standardize fields

This part includes abstract business fields and free fields, and also clearly defines custom-defined specifications:
•Specification of standardized fields : At this time, some fields can be standardized by some abstract nouns. For example, the time of creating an entity can be fixed as create_time . This part is similar to the resource content combing part in the resource-related definition , but the former focuses more on resources, while When standardizing fields here, it is not only resources, but also time, status, etc.;
•Specification of custom field specification : For some fields that cannot be abstracted, the specification can be carried out in three stages:
•try to use the standard fields provided;
•When the standard fields cannot be satisfied, the provided independent fields can be used to combine, for example, describe and information fields are provided to combine into describe_information ;
•When the standard fields cannot be satisfied, you can customize the fields, but you must follow certain rules (the rules here can be as detailed as possible, such as only nouns, underscores for linkers, etc.); Defining errors

In addition to the above, the specification also needs to define the status of the error, as well as the structure of the error and the error code:
•System error : It is necessary to unify and standardize system-level errors. The so-called system - level errors refer to the occurrence of some problems due to uncontrollable factors, such as unresponsive back-end services. At this time, the platform side specifies the error code and error structure. return;
•Business error : some errors caused by the business itself, such as the parameter passed by the user is wrong, does not exist, etc.; at this time, it is necessary to abstract some error codes (the first-level error code and the second-level error code can be provided if necessary, for example, INVALID_ARGUMENT can be increased by two level error code ARGUMENT_TYPE_ERROR , ARGUMENT_NAME_ERROR , etc.;);
•Custom errors : Custom errors refer to scenarios that cannot be covered by system errors and business errors under very special circumstances. The business team can customize error codes, but such errors need to be formulated according to certain specifications;
•Error structure/composition : Although some error codes will be abstracted, for example, the error structure of the Google Cloud protocol is as follows:
"error" : {
"code" : 400 ,
"message" : "API key not valid. Please pass a valid API key." ,
"status" : "INVALID_ARGUMENT" ,
"details" : [
"reason" : "API_KEY_INVALID" ,
"domain" : "" ,
"metadata" : {
"service" : ""
At this point, customized error messages can be specified according to actual business requirements. Other definitions

•Definition of corresponding content : In some response headers and Body, it is also necessary to define a fixed structure or specification, such as the location of RequestId; for requests that cannot get results quickly after long-term practice, the return body is composed;
•Abstraction of general capabilities : for example, whether paging capabilities, sorting capabilities, and filtering capabilities need to be uniformly defined as upper-level abstract capabilities;
•Definition of permissions and security : For example, to check whether certain resources return 401 or 404 without authorization (for example, Github will remind 404 when anonymously accessing a private Repo, which is to a certain extent to prevent attacks, data guessing, and data leakage);
•Others : other numbers include frequency, including caching strategy, including Etag, etc.;

3.3 Construction of connection points between internal teams and external users

If it is said that the clarity of API style and the completion of design guidelines are things to be done before API development, and API development process is important to follow relevant specifications. Then the API external release platform is a very important key point after the API development and before the launch. This key point can also be considered as the core connection point between the internal team and external users. Through this connection point, it will be officially exposed to the outside world on a large scale. High-quality, high-consistency RESTful-style open APIs and supporting products. Therefore, the importance of this link point can be imagined. Usually, the contents of the API's external publishing platform include:
•API entry: This part includes the storage of basic API information and the abstract capabilities of the upper layer, such as unified authentication rules, overclocking restrictions for different business abstractions, disaster recovery strategies, etc.;
•API testing/experience: This part includes self-testing, pre-release testing, and status preview after the API goes online, so that the development team and the API platform can more clearly and intuitively determine the final delivery form of the API; of course, It also includes previews of supporting products, such as documentation, SDK, API Explorer, etc.;
•Review of API style and specifications: This part is what the API team needs to do on the platform, mainly to ensure consistent API style, compatibility, and correct use of specifications;
•API finalization and release: When everything is determined and completed, the platform should provide complete external release determination logic and the ability to release unified externally;
•Automatic release of API supporting resources: This part mainly refers to the standardized production and launch of related supporting capabilities after the API is released to the outside world:
•API Documentation
•SDK for multiple languages
•Corresponding sandbox environment , etc.
•Updates to API Explorer
•The update of the corresponding resource aggregation page (such as error center, etc., API center, etc.)
To sum up, since the platform carries the key work of opening the API and exposing it to the outside world, the value of the platform is self-evident:
•Inside :
•Determine the overall quality, uniformity of the API
•Provide API-level specifications, maintenance plans, release tools, and testing and verification platforms internally
•The last line of defense against the overall problem of convergence
•External :
•Deliver a unified mind
•mind that delivers scientific norms
•Delivery of supporting facilities (e.g. documentation, SDK, etc.)
4 Large-scale delivery of API and supporting product solutions
Under the premise of multi-team collaboration, how to deliver high-quality, high-consistency RESTful APIs and supporting products on a large scale is a very challenging task. As mentioned above, under certain preconditions, team collaboration is inevitable, but this will inevitably have a huge conflict with the open API that exposes "consistency" to the outside world, so how to deliver on a large scale while maintaining consistency, To reduce the negative impact of conflicts, it is necessary to have a clear open API style, a complete API release process, clear open API design guidelines, and corresponding tools/platforms:

Related Articles

Explore More Special Offers

  1. Short Message Service(SMS) & Mail Service

    50,000 email package starts as low as USD 1.99, 120 short messages start at only USD 1.00

phone Contact Us