By Chengtan and Rufeng
This article introduces a Demo of Open-Source Higress and explains the principle and mechanism behind it.
In order to run the Demo in this article, you must install Higress in the Kubernetes cluster and make sure the following quickstart configuration takes effect.
https://github.com/alibaba/higress/releases/download/v0.5.2/quickstart.yaml
The function to be implemented in this Demo is a Mock response, which is to return HTTP responses according to the configured content. This article describes:
package main
import (
. "github.com/alibaba/higress/plugins/wasm-go/pkg/wrapper"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
"github.com/tidwall/gjson"
)
func main() {
SetCtx(
"my-plugin",
ParseConfigBy(parseConfig),
ProcessRequestHeadersBy(onHttpRequestHeaders),
)
}
type MyConfig struct {
content string
}
func parseConfig(json gjson.Result, config *MyConfig, log Log) error {
config.content = json.Get("content").String()
return nil
}
func onHttpRequestHeaders(ctx HttpContext, config MyConfig, log Log) types.Action {
proxywasm.SendHttpResponse(200, nil, []byte(config.content), -1)
return types.ActionContinue
}
You can see three functions in the code above:
The three parameters passed in are:
The plugin function implemented by this 30-line code is simple. Here are some examples of complex functions:
https://github.com/alibaba/higress/tree/main/plugins/wasm-go/extensions
Here is the detailed usage document of the plugin SDK (EN TBD):
https://higress.io/en-us/docs/user/wasm-go.html
This plugin SDK is implemented based on the proxy-wasm-go-sdk of the Tetrate community. If you pay attention to the details at the lower level, you can see:
https://github.com/tetratelabs/proxy-wasm-go-sdk
https://github.com/alibaba/higress/blob/main/plugins/wasm-go/pkg/wrapper
The Wasm-Go SDK of Higress encapsulates the details of plugin context processing through the generic feature introduced by Go 1.18, thus reducing the amount of code required for plugin development. Developers can focus on the logic of configuration parsing and request/response processing.
After the code is written, there are three steps to implement the plugin logic:
Compile the Go file main.go above into plugin.wasm
tinygo build -o plugin.wasm -scheduler=none -target=wasi main.go
Write Dockerfile:
FROM scratch
COPY plugin.wasm ./
Build and push the Docker image (The official image registry of Higress is used in this example):
docker build -t higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0 .
docker push higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0
Write wasmplugin.yaml
The configuration description is listed below:
In addition to these configurations, you can define advanced configurations (such as the execution phase and priority of the plugin). Please see the official document of Istio API for more information:
https://istio.io/latest/docs/reference/config/proxy_extensions/wasm-plugin/
# wasmplugin.yaml
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
name: mock-response
namespace: higress-system
spec:
selector:
matchLabels:
higress: higress-system-higress-gateway
pluginConfig:
content: "hello higress"
url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0
Create this resource through kubectl:
kubectl apply -f wasmplugin.yaml
Based on the previously effective quickstart.yaml, the current Ingress access topology in the cluster is listed below:
If the plugin does not take effect:
The wasmplugin.yaml is issued based on the preceding effective plugin phase. The effect after it takes effect is listed below:
Modify the configuration of wasmplugin.yaml:
# wasmplugin.yaml
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
name: mock-response
namespace: higress-system
spec:
selector:
matchLabels:
higress: higress-system-higress-gateway
pluginConfig:
content: "hello higress"
_rules_:
- content: "hello foo"
_match_route_:
- "default/foo"
- content: "hello bar"
_match_route_:
- "default/bar"
- content: "hello world"
_match_domain_:
- "*.example.com"
- "www.test.com"
url: oci://higress-registry.cn-hangzhou.cr.aliyuncs.com/plugins/demo:1.0.0
The rule list rules is added to the pluginConfig. You can specify the matching method in the rule and enter the corresponding effective configuration:
Make sure the modified configuration takes effect:
kubectl apply -f wasmplugin.yaml
You can see the effect below:
Here is a brief explanation of the effective mechanism of the plugin:
Here, Envoy uses the Extension Config Discovery Service (ECDS) mechanism to obtain the configuration and load the Wasm file, which realizes the update of the Wasm file and direct hot loading without causing any connection interruption, and the business traffic is completely lossless.
The preceding Wasm plugin mechanism brings three revolutionary features to gateway custom plugin development.
This feature is mainly due to the WasmPlugin mechanism design of Istio. You can compare it with the plugin mechanism of Kubernetes Nginx Ingress.
Installing a plugin
There are two options:
https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/lua/plugins/README.md
As we can see, in order to load the custom plugin, Nginx Ingress needs to mount the Lua file to the pod or install the plugin during the image build. As such, the lifecycle of the plugin is bound to the gateway. If the plugin logic is updated, a new version of the plugin is required to be released, and a new version of the gateway needs to be released or redeployed.
With the WasmPlugin mechanism, if you need to publish a new version of the plugin, you only need to build an image of the plugin itself and issue it to take effect. You can also manage the version of the plugin based on the tag of the image. This way, if the plugin changes, there is no need to redeploy the gateway, but it is non-destructive to traffic in combination with the ECDS mechanism of Envoy.
Based on the capabilities of Wasm, plugins can be written in multiple languages, which is more friendly for developers. Another way to implement the multi-language development of plugins is the external process/service plugin based on RPC and gateway process communication. This mode has additional I/O overhead, and additional processes/services bring additional O&M complexity. Currently, people are more concerned with the performance of the Wasm plugin. From our test data, the instruction execution performance is not as good as the native C++ language, but it is on par with Lua and far better than the external plugin.
For a piece of logic: Request headers are configured 20 consecutive times, obtained 20 consecutive times, and removed 20 consecutive times. We compared the processing performance of Wasm implemented in Lua and other languages. The following is a comparison of the impact on the latency of a single request:
Implementing Language | Request Latency Increase |
Lua | 0.20 ms |
Wasm (C++) | 0.19 ms |
Wasm (Go) | 0.20 ms |
Wasm (Rust) | 0.21 ms |
Wasm (AssemblyScript) | 0.21 ms |
Currently, Envoy supports various Wasm runtimes (such as V8, WAMR, and wasmtime). These runtimes all provide secure sandbox capabilities. As such, the Wasm plugin will not cause the Envoy host process to crash even if the logic (such as access to null pointers and uncaught exceptions, occurs). In addition, you can perform Fail Open processing through configuration when an exception occurs in the plugin logic and skip the execution logic of the plugin to minimize the impact on businesses.
Thanks to the pre-work done by the Istio/Envoy community, Higress was able to enable WasmPlugin for Ingress resources, enhancing the custom extension capabilities of the Ingress Controller.
Thanks to the proxy-wasm-go-sdk implemented by the Tetrate community, Higress encapsulated wasm-go SDK on this basis, lowering the threshold for developing plugins.
Higress completed some Bugfix work on the Wasm capability of Istio/Envoy, which was merged into the upstream community.
You are welcome to contribute to the plugin of Higress and other community ecology. Please refer to the following documents for contributions to Higress:
Interpreting KubeVela 1.7: Taking Over Your Existing Workloads
Exploration and Practice of Proxyless Mesh for Spring Cloud Applications
507 posts | 48 followers
FollowAlibaba Cloud Native Community - September 12, 2023
Alibaba Cloud Native Community - November 18, 2024
Alibaba Cloud Native Community - July 20, 2023
Alibaba Cloud Native Community - February 2, 2024
Alibaba Cloud Native Community - April 11, 2024
Alibaba Cloud Native Community - November 15, 2023
507 posts | 48 followers
FollowMulti-source metrics are aggregated to monitor the status of your business and services in real time.
Learn MoreMore Posts by Alibaba Cloud Native Community
5401280606175155 April 5, 2023 at 9:17 am
nice article. CCSP Course