Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog PouchContainer and RPC

PouchContainer and RPC

In this article, we will introduce the key concepts of Remote Procedure Call (RPC) and briefly discuss its implementation using Alibaba PouchContainer.

Remote Procedure Call (RPC) is a computer communication protocol that allows the program running on one computer to call subprograms on another computer, without coding for remote interaction. In the object-oriented programming paradigm, RPC calls are represented by remote invocation or remote method invocation (RMI). In simple terms, RPC is a method to invoke a remote known procedure through the network. RPC is a design and concept, but not a specific technology. RPC involves three key points:

  1. Communication Protocol: Remote invocation requires the support of bottom-layer network communication protocols that can implement network transmission, for example, HTTP/TCP and even UDP.
  2. Coding Protocol: The client and server use streams to communicate with each other, so the serialization and deserialization processes are involved. The client and server must use the same serialization protocol.
  3. Method Mapping: To determine the method used by the client to invoke the remote server, the server (or a third-party service discovery center) must maintain a method list for service provision.

These three key points constitute a simple RPC frame. The Go Programming Language (Golang) maintains a simple RPC frame, of which the source code are located at src/net/rpc. Its implementation is simple:

  1. Server: It provides the service registration methods. It uses the gob coding serialization frame and several server booting and listening methods by default.
  2. Client: It provides several server connection methods. It uses the gob coding serialization frame and two service invocation methods by default.
    1. Go: Asynchronous invocation
    2. Call: Synchronous invocation. Actually, the Go methods are invoked. A channel without buffer is introduced to read the procedure blocks, implementing the synchronization.
  3. jsonrpc: The Golang provides a gob serialization frame, which performs excellently; however, it is supported by only the Golang. The official RPC of Go can implement json serialization, but it supports only TCP connections but does not support HTTP due to the limitation in design.

The official RPC frame of Go is insufficient for large projects. Golang officially recommends the GRPC frame, which is maintained in community. GRPC provides all basic functions and fully utilizes the features of Golang. You can learn a lot just by reading the source code.

GRPC

As just mentioned, Golang officially recommends GRPC. What are the advantages of GRPC?

  1. GRPC supports multiple languages. Currently, GRPC supports C++, go, java, python, node, and Ruby.
  2. GRPC has a powerful IDL feature and uses ProtoBuf to define services. ProtoBuf is a high-performance, widely used data serialization protocol developed by Google.
  3. GRPC is designed based on HTTP/2.
  4. With its source code published by Google, GRPC has many active users in the developer community.

1
Source: https://grpc.io/

PouchContainer Architecture

The PouchContainer community has provided a clear architecture diagram. However, the diagram is extremely detailed, which makes it difficult to illustrate RPC on PouchContainer. The following is a simplified figure:

2

The arrows point to the service end. When the Pouchcontainer Daemon (Pouchd) starts, two types of services are also started: HTTP Server and RPC Server. The PouchContainer was designed to work with Docker, so both Pouch CLI and Docker CLI can send requests through the HTTP server started by Pouchd to operate the PouchContainer. In addition, PouchContainer cooperates with CNCF and is compatible with Kubernetes. It implements the PouchContainer CRI defined by the Kubernetes community. The RPC server related to CRI can be started using enable-cri. By default, Pouchd does not start any service. In addition, the Containerd provides container service to the bottom layer of PouchContainer, and they interact through RPC client.

GRPC Application in PouchContainer

As we know, PouchContainer provides the RPC server to be invoked by Kubelet on CRI Manager, and invokes the bottom-layer Containerd through RPC Client. The GRPC frame is used in this process.

RPC server in CRI Manager

The following is the simplified codes for server booting:

// Start CRI service with CRI version: v1alpha2
func runv1alpha2(daemonconfig *config.Config, containerMgr mgr.ContainerMgr, imageMgr mgr.ImageMgr) error {
   ······
   criMgr, err := criv1alpha2.NewCriManager(daemonconfig, containerMgr, imageMgr)

   ······
   server := grpc.NewServer(),
   runtime.RegisterRuntimeServiceServer(server, criMgr)
   runtime.RegisterImageServiceServer(server, criMgr)  
   ······

   service.Serve()  
   ······
}

The simplified codes are similar to the demo officially provided by GRPC, including the following steps:

  1. grpc.NewServer()
  2. Service registration
  3. service.Serve() booting service

GRPC has done many jobs. We only need to write the proto files to define services and use tools to generate the corresponding server and client codes. The server additionally carries out the previous three steps. Due to the limitation on Kubernetes and service requirements, CRI Manager needs to offer more services. Therefore, the PouchContainer team expanded the CRIs. The steps are as follows:

  1. Modify the official proto file of Kubernetes. Two CRI versions are available: v1alpha1 and v1alpha2.
  2. Recompile the stub file using tools. The command is as follows:
    protoc  --proto_path=. --proto_path=../../../../../vendor/ --proto_path=${GOPATH}/src/github.com/google/protobuf/src --gogo_out=plugins=grpc:. *.proto
  3. Modify the source code related to CRI Manager to offer more interconnection services.

RPC Client in ctrd

In the simplified PouchContainer architecture, you can find that Pouchd interacts with Containerd through ctrd. For better understanding, we can see the NewDaemon workflow:

3

To understand what ctrd.NewClient did, let's look at the following source code.

// NewClient connect to containerd.
func NewClient(homeDir string, opts ...ClientOpt) (APIClient, error) {
   // set default value for parameters
   copts := clientOpts{
      rpcAddr:                unixSocketPath,
      grpcClientPoolCapacity: defaultGrpcClientPoolCapacity,
      maxStreamsClient:       defaultMaxStreamsClient,
   }

   for _, opt := range opts {
      if err := opt(&copts); err != nil {
         return nil, err
      }
   }

   client := &Client{
      lock: &containerLock{
         ids: make(map[string]struct{}),
      },
      watch: &watch{
         containers: make(map[string]*containerPack),
      },
      daemonPid:      -1,
      homeDir:        homeDir,
      oomScoreAdjust: copts.oomScoreAdjust,
      debugLog:       copts.debugLog,
      rpcAddr:        copts.rpcAddr,
   }

   // start new containerd instance.
   if copts.startDaemon {
      if err := client.runContainerdDaemon(homeDir, copts); err != nil {
         return nil, err
      }
   }

   for i := 0; i < copts.grpcClientPoolCapacity; i++ {
      cli, err := newWrapperClient(copts.rpcAddr, copts.defaultns, copts.maxStreamsClient)
      if err != nil {
         return nil, fmt.Errorf("failed to create containerd client: %v", err)
      }
      client.pool = append(client.pool, cli)
   }

   logrus.Infof("success to create %d containerd clients, connect to: %s", copts.grpcClientPoolCapacity, copts.rpcAddr)

   scheduler, err := scheduler.NewLRUScheduler(client.pool)
   if err != nil {
      return nil, fmt.Errorf("failed to create clients pool scheduler")
   }
   client.scheduler = scheduler

   return client, nil
}

The code can be understood by dividing it into the following steps:

  1. Construct a client object and set the corresponding parameters.
  2. Start ContainerdDaemon as required.
  3. Initially create RPC clients.
  4. Assign a scheduler to clients. The key is step 3, in which grpcClientPoolCapacity clients are repeatedly created and added to the client object pool. There are three key points:
    1. Client object pool. There is no GRPC official connection pool.
    2. newWrapperClient. Created GRPC client object.
    3. maxStreamsClient. According to the verification of the PouchContainer team, the maximum number of grpc-go client streams is 100. The GRPC bottom layer is based on the HTTP2 protocol, so each request is encapsulated into the HTTP2 stream. Because the GRPC does not provide an official connection pool, PouchContainer defines a connection pool to improve the concurrency capacity of clients and offers a schedule in the Get methods. By default, PouchContainer provides a recently least used scheduling algorithm. You can also define other schedulers. Actually, to avoid repeated work, Containerd provides an official RPC client package to encapsulate the codes on RPC clients. The newWrapperClient also invokes the containerd.New() method in the client package. Therefore, the codes of ctrd are not as much as you think.

Conclusion

The component decoupling was considered in the design of PouchContainer architecture, so you will easily understand the PouchContainer overall design through the architecture figure. As a member in container ecosystem, PouchContainer is connected to the upper-layer Kubernetes through RPC, and connected to the lower-layer Containerd by reusing what is already available. Then PouchContainer can implement its own features such as rich container.

Architecture Container Trends Computing Open Source Pouch RPC
0 0 0
Share on

Read previous post:

Understanding the Kubelet Core Execution Frame

Read next post:

PouchContainer Engineering Quality Practice

Alibaba Cloud Native Community

480 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

480 posts | 48 followers

Related Products

More Posts by Alibaba Cloud Native Community

See All