Higress Practice: Use 30 Lines of Code to Write a Wasm Go Plugin

By Chengtan and Rufeng

Preface

This article introduces a Demo of Open-Source Higress and explains the principle and mechanism behind it.

In order to run the Demo in this article, you must install Higress in the Kubernetes cluster and make sure the following quickstart configuration takes effect.

https://github.com/alibaba/higress/releases/download/v0.5.2/quickstart.yaml

The function to be implemented in this Demo is a Mock response, which is to return HTTP responses according to the configured content. This article describes:

Code Writing: Code Logic Parsing
Plugin Effective: This section describes how the code is compiled, packaged, and deployed to take effect.
Testing Plugin Function: This section describes how the global granularity and route/domain-level granularity take effect.
Principle of Plugins Taking Effect: This section reviews the overall process and explains how the plugin takes effect.
Three Revolutionary Features: An introduction to the convenience brought about by the Wasm plugin mechanism for gateway plugin development

Code Writing

package main

import (
    . "github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
    "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
    "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
    "github.com/tidwall/gjson"
)

func main() {
    SetCtx(
        "my-plugin",
        ParseConfigBy(parseConfig),
        ProcessRequestHeadersBy(onHttpRequestHeaders),
    )
}

type MyConfig struct {
    content string
}

func parseConfig(json gjson.Result, config *MyConfig, log Log) error {
    config.content = json.Get("content").String()
    return nil
}

func onHttpRequestHeaders(ctx HttpContext, config MyConfig, log Log) types.Action {
    proxywasm.SendHttpResponse(200, nil, []byte(config.content), -1)
    return types.ActionContinue
}

You can see three functions in the code above:

main: The plugin defines the plugin context through the main function, including the plugin name, the function used to parse configurations, and the function used to process the request/response
parseConfig: This function is mounted to the plugin configuration parsing phase through the ParseConfigBy specified in SetCtx. The three parameters passed in are:

json: The configuration passed into the plugin is uniformly serialized into a json dictionary object, and parseConfig is provided for parsing.
config: parseConfig outputs the parsed plugin configuration to the MyConfig object.
log: It provides the log output interface.

onHttpRequestHeaders: proxywasm.SendHttpResponse called in the function. It is used to return HTTP responses. This function is mounted to the execution phase of parsing the request Header through the ProcessRequestHeadersBy specified in SetCtx. Other mounting methods include:

ProcessRequestBodyBy: Mount to the execution phase of the parsing request Body
ProcessResponseHeadersBy: Mount to the execution phase of constructing the response Header
ProcessResponseBodyBy: Mount to the execution phase of constructing the response Body

The three parameters passed in are:

ctx: It is used to obtain the request context (such as scheme, method, and path). ctx can be used to set a custom context, and access across execution phases is supported.
config: It provides the custom configuration parsed by parseConfig.
log: It provides the log output interface.

The plugin function implemented by this 30-line code is simple. Here are some examples of complex functions:

https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions

Here is the detailed usage document of the plugin SDK (EN TBD):

https://higress.io/en-us/docs/user/wasm-go.html

This plugin SDK is implemented based on the proxy-wasm-go-sdk of the Tetrate community. If you pay attention to the details at the lower level, you can see:

https://github.com/tetratelabs/proxy-wasm-go-sdk

https://github.com/alibaba/higress/blob/main/plugins/wasm-go/pkg/wrapper

The Wasm-Go SDK of Higress encapsulates the details of plugin context processing through the generic feature introduced by Go 1.18, thus reducing the amount of code required for plugin development. Developers can focus on the logic of configuration parsing and request/response processing.

Plugin Effective

After the code is written, there are three steps to implement the plugin logic:

Compile: Compile the Go code into a Wasm format file
Image Push: Package the Wasm file into a Docker image and push it to the image registry
Issue the Configurations: Create the WasmPlugin resource on Kubernetes

Compile

Compile the Go file main.go above into plugin.wasm

tinygo build -o plugin.wasm -scheduler=none -target=wasi main.go

Image Push

Write Dockerfile:

FROM scratch
COPY plugin.wasm ./

Build and push the Docker image (The official image registry of Higress is used in this example):

docker build -t higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0 .
docker push higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0

Issue the configurations

Write wasmplugin.yaml
The configuration description is listed below:

selector: The higress-gateway installed in the higress-system namespace by default is selected to implement the plugin.
pluginConfig: The plugin configuration, which is eventually converted to the MyConfig object in the preceding code
url: Enter the image address, which must start with "oci://"

In addition to these configurations, you can define advanced configurations (such as the execution phase and priority of the plugin). Please see the official document of Istio API for more information:

https://istio.io/latest/docs/reference/config/proxy_extensions/wasm-plugin/

# wasmplugin.yaml
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: mock-response
  namespace: higress-system
spec:
  selector:
    matchLabels:
      higress: higress-system-higress-gateway
  pluginConfig:
    content: "hello higress"
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0

Create this resource through kubectl:

kubectl apply -f wasmplugin.yaml

Test the Plugin Function

Based on the previously effective quickstart.yaml, the current Ingress access topology in the cluster is listed below:

If the plugin does not take effect:

Request/foo will return the HTTP response "foo"
Request/bar will return the HTTP response "bar"

Take Effect Globally

The wasmplugin.yaml is issued based on the preceding effective plugin phase. The effect after it takes effect is listed below:

Request/foo will return the HTTP response "hello higress"
Request/bar will return the HTTP response "hello higress"

Domain Name and Route-Level Effective Plugin

Modify the configuration of wasmplugin.yaml:

# wasmplugin.yaml
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: mock-response
  namespace: higress-system
spec:
  selector:
    matchLabels:
      higress: higress-system-higress-gateway
  pluginConfig:
    content: "hello higress"
    _rules_:
    - content: "hello foo"
      _match_route_:
      - "default/foo"
    - content: "hello bar"
      _match_route_:
      - "default/bar"
    - content: "hello world"
      _match_domain_:
      - "*.example.com"
      - "www.test.com"
  url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0

The rule list rules is added to the pluginConfig. You can specify the matching method in the rule and enter the corresponding effective configuration:

match_route: It matches Ingress to take effect. The match format is the namespace where the Ingress resides + "/" + Ingress name.
_match_domain_: It matches the domain name to take effect. You only need to enter a domain name. Wildcard characters are supported.

Make sure the modified configuration takes effect:

kubectl apply -f wasmplugin.yaml

You can see the effect below:

Request/foo will return the HTTP response "hello foo" (matching the first rule)
Request/bar will return the HTTP response "hello bar" (matching the second rule)
The request www.example.com will return the HTTP response "hello world" (matching the third rule)
The request www.abc.com will return the HTTP response "hello higress" (no matching rule, global configuration used)

Principle of Plugins Taking Effect

Here is a brief explanation of the effective mechanism of the plugin:

The user compiles the code into a Wasm file
The user builds the Wasm file into a Docker image
The user pushes the Docker image to the image registry
The user creates the WasmPlugin resource
Istio watches changes of the WasmPlugin resource
The xDS proxy process in Higress Gateway obtains the configuration from Istio and finds the image address of the plugin.
xDS proxy pulls images from the image registry.
xDS proxy extracts the Wasm file from the image.
The envoy process in the Higress Gateway obtains the configuration from the xDS proxy and discovers the local path of the Wasm file.
Envoy loads the Wasm file from the local file.

Here, Envoy uses the Extension Config Discovery Service (ECDS) mechanism to obtain the configuration and load the Wasm file, which realizes the update of the Wasm file and direct hot loading without causing any connection interruption, and the business traffic is completely lossless.

Three Revolutionary Features

The preceding Wasm plugin mechanism brings three revolutionary features to gateway custom plugin development.

Feature 1: Plugin Lifecycle and Gateway Decoupling

This feature is mainly due to the WasmPlugin mechanism design of Istio. You can compare it with the plugin mechanism of Kubernetes Nginx Ingress.

Installing a plugin

There are two options:

mount your plugin into /etc/nginx/lua/plugins/ in the ingress-nginx pod
Build your own ingress-nginx image (like the example) and install your plugin during image build

https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/lua/plugins/README.md

As we can see, in order to load the custom plugin, Nginx Ingress needs to mount the Lua file to the pod or install the plugin during the image build. As such, the lifecycle of the plugin is bound to the gateway. If the plugin logic is updated, a new version of the plugin is required to be released, and a new version of the gateway needs to be released or redeployed.

With the WasmPlugin mechanism, if you need to publish a new version of the plugin, you only need to build an image of the plugin itself and issue it to take effect. You can also manage the version of the plugin based on the tag of the image. This way, if the plugin changes, there is no need to redeploy the gateway, but it is non-destructive to traffic in combination with the ECDS mechanism of Envoy.

Feature 2: High-Performance Multi-Language Support

Based on the capabilities of Wasm, plugins can be written in multiple languages, which is more friendly for developers. Another way to implement the multi-language development of plugins is the external process/service plugin based on RPC and gateway process communication. This mode has additional I/O overhead, and additional processes/services bring additional O&M complexity. Currently, people are more concerned with the performance of the Wasm plugin. From our test data, the instruction execution performance is not as good as the native C++ language, but it is on par with Lua and far better than the external plugin.

For a piece of logic: Request headers are configured 20 consecutive times, obtained 20 consecutive times, and removed 20 consecutive times. We compared the processing performance of Wasm implemented in Lua and other languages. The following is a comparison of the impact on the latency of a single request:

Implementing Language	Request Latency Increase
Lua	0.20 ms
Wasm (C++)	0.19 ms
Wasm (Go)	0.20 ms
Wasm (Rust)	0.21 ms
Wasm (AssemblyScript)	0.21 ms

Feature 3: Secure Sandbox

Currently, Envoy supports various Wasm runtimes (such as V8, WAMR, and wasmtime). These runtimes all provide secure sandbox capabilities. As such, the Wasm plugin will not cause the Envoy host process to crash even if the logic (such as access to null pointers and uncaught exceptions, occurs). In addition, you can perform Fail Open processing through configuration when an exception occurs in the plugin logic and skip the execution logic of the plugin to minimize the impact on businesses.

Open-Source Community

Thanks to the pre-work done by the Istio/Envoy community, Higress was able to enable WasmPlugin for Ingress resources, enhancing the custom extension capabilities of the Ingress Controller.

Thanks to the proxy-wasm-go-sdk implemented by the Tetrate community, Higress encapsulated wasm-go SDK on this basis, lowering the threshold for developing plugins.

Higress completed some Bugfix work on the Wasm capability of Istio/Envoy, which was merged into the upstream community.

You are welcome to contribute to the plugin of Higress and other community ecology. Please refer to the following documents for contributions to Higress:

https://higress.io/en-us/docs/developers/guide_dev.html

Community

Higress Practice: Use 30 Lines of Code to Write a Wasm Go Plugin

Preface

Code Writing

Plugin Effective

Compile

Image Push

Issue the configurations

Test the Plugin Function

Take Effect Globally

Domain Name and Route-Level Effective Plugin

Principle of Plugins Taking Effect

Three Revolutionary Features

Feature 1: Plugin Lifecycle and Gateway Decoupling

Feature 2: High-Performance Multi-Language Support

Feature 3: Secure Sandbox

Open-Source Community

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

5401280606175155 April 5, 2023 at 9:17 am

Alibaba Cloud Native Community

Related Products

Managed Service for Prometheus