Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Higress Open Source Plugin Server Simplifies the Challenges of Private Deployment for Wasm Plugins

Higress Open Source Plugin Server Simplifies the Challenges of Private Deployment for Wasm Plugins

The article unveils Higress’s new Wasm Plugin Server, an HTTP-based tool that streamlines private deployment and management of Higress Wasm plugins.

By Dao Feng

This article will introduce a recent advancement by Higress in the Wasm plugin ecosystem—Higress Wasm Plugin Server. This new component addresses the pain points of users pulling plugins during private deployment of the Higress gateway, optimizing the download and management efficiency of plugins.

Repository address: https://github.com/higress-group/plugin-server

Wasm Plugins: Higress's Extension Capabilities and Challenges

Since its open source release, Higress has regarded Wasm technology as a core means of gateway extension. The engineering reliability, sandbox security, hot update capability brought by Wasm, as well as features such as domain/route-level effectiveness, Redis access capabilities, and AI feature support built by the Higress team enrich the gateway's extensibility and provide performance improvements and cost reductions for enterprise users. Through custom Wasm plugins, users can implement logic for authentication, encryption and decryption, session management, etc., at the gateway layer according to their business needs, reducing additional resource consumption and lowering the processing burden on backend services.

Despite the advantages of Wasm plugin technology itself, we still face some practical challenges in real enterprise-level deployments and large-scale application scenarios, mainly reflected in the following aspects:

1. Challenges of Private Deployment due to OCI Mechanism

Currently, the download and management of Higress Wasm plugins mainly depend on OCI (Open Container Initiative) repositories.

About OCI, oras, and Docker

  • OCI (Open Container Initiative): OCI aims to define open standards for container images and runtimes to ensure interoperability between different container technologies. In the cloud-native ecosystem, OCI image repositories (like Docker Hub, Harbor, registry.k8s.io, etc.) are the standard way to distribute container images. Higress originally released Wasm plugins as OCI Artifacts, packaging them into OCI-compliant products stored in OCI repositories.
  • Docker: Docker is currently the most popular container platform, which uses OCI images as its core distribution format. Typically, we use the docker pull and docker push commands to pull and push container images.
  • oras (OCI Registry As Storage): oras is a command-line tool that allows users to store and manage any content in OCI repositories, not just container images. For non-standard container images like Higress Wasm plugins (Wasm modules), oras provides a convenient way to interact with OCI repositories (for example, pulling and pushing Wasm plugins).

Challenges of OCI Mechanism

Although the OCI mechanism is a standard and efficient way in cloud-native environments, for some enterprises, especially in private deployment scenarios with strict requirements on network security, the introduction of OCI repositories has become a significant barrier, raising the following issues:

  • Technical Barriers and Inconvenience in Tool Usage: Many users may be accustomed to direct installation of applications or deployment through simple package managers, and may not be familiar with the use of container images, OCI standards, and OCI tools like oras that are specifically used for non-image content. This increases the difficulty in getting started with Wasm plugins and the operational complexity.
  • Network Restrictions and Private Deployment: Many enterprises have strict internal network policies that impose significant restrictions on accessing external public networks (like Docker Hub, GitHub Container Registry, etc.). In private deployment environments, the inability to access external public repositories can lead to the failure to configure and update plugins.
  • Additional Infrastructure Setup: To pull plugins in a private environment, users might need to deploy and maintain an internal OCI repository separately. This undoubtedly adds to the complexity of deployment and the operational costs. For users who only need to use Wasm plugins simply, setting up and managing a complete OCI ecosystem for just a few plugins seems overly cumbersome.
  • Difficulties in Plugin Migration: When migrating Wasm plugins between different private environments (such as development, testing, production), due to the independence of OCI repositories, users often need to manually pull, push, and manage versions of plugins, while having to adapt to different repository authentication and network configurations, increasing operational complexity and the possibility of errors.

2. Redundant Downloads and Performance Overhead: Concerns of Always Policy

When the Higress gateway pulls Wasm plugins, it supports the configuration of plugin pulling policies, defaulting to IfNotPresent, which means that if the local plugin exists, it will not pull it again. This is reasonable in most cases. However, when users want the Wasm plugin to update promptly (for example, when iterating frequently in development and testing environments) or want to ensure that they are always using the latest version of the plugin, they tend to set the policy to Always, resulting in the following issues:

  • Network Latency and Bandwidth Consumption: The Always policy means that every time a Wasm plugin is referenced or the gateway Pod is restarted, it will attempt to re-download the plugin from the OCI repository. This introduces unnecessary network latency and consumes bandwidth resources.
  • Redundant Operations: Even if the content of the plugin has not changed, the Always policy will still trigger a download. Although the OCI protocol itself supports content hash verification, the client still needs to communicate with the repository to confirm.

3. User Experience and Operational Complexity:

The above issues collectively lead to unnecessary complexity when users configure and use Wasm plugins. We hope to provide a simpler, more intuitive way to distribute plugins, enabling users to focus more on implementing business logic.

It is precisely based on these pain points that we have developed the Higress Wasm Plugin Server.

Higress Plugin Server: Simple and Efficient Distribution Based on HTTP

Our core idea is to provide a simple, highly available HTTP-based file server to distribute Wasm plugins. The reason for choosing HTTP is that it is ubiquitous, easy to deploy, easy to integrate, and in many enterprise internal environments, HTTP file services are standard configuration and simple to set up.

This new component, higress-plugin-server, takes on the following core functions and goals:

  • Functional Goals: To provide an HTTP file server that supports the gateway to download Wasm plugins via paths like http://higress-plugin-server.higress-system.svc/plugins/<plugin-name>/<version>/plugin.wasm.
  • High Availability: As a core component, it must be able to run stably to ensure the SLA of the plugin distribution service.
  • Version Management: The image internally integrates all Wasm plugins and is strictly bound to plugin versions. The current design idea for plugin versions is to decouple plugin versions from Higress gateway versions, using an overlay approach. If there is future plugin version management, the Plugin Server can directly support it; if the situation remains unchanged, it can also distinguish plugin versions through the md5 field in the metadata provided by the Plugin Server.
  • Compatibility: Seamlessly coexist with existing OCI plugin loading methods, allowing users to flexibly configure plugin pulling policies based on their needs.

Its launch brings the following advantages to users:

  • Powerful Tool for Private Deployment: Get rid of dependency on complex OCI repositories. Users only need to directly use our pre-built higress-plugin-server image to quickly and securely distribute Wasm plugins in any network environment.
  • Simplified and Out-of-the-Box: For most users, no excessive additional configuration is needed, and they may not even need to be aware of the presence of the OCI repository. Simply adding a parameter to the default Helm Chart installation command of Higress allows for the automatic deployment and configuration of the plugin server, achieving an out-of-the-box experience.
  • Performance and High Availability Assurance: HTTP file services are efficient and easy to cache; after being deployed in the internal network, they can reduce the latency and failure rate of plugin downloads. Deploying multiple replicas through K8s can achieve high availability.
  • Seamless Coexistence with Existing Mechanisms: Users can still configure the OCI repository address through the Higress Console or environment variables, and the two do not interfere with each other.

Architecture Design and Implementation: The Mechanism from OCI to HTTP

To achieve the above goals, we designed and built the higress-plugin-server component from scratch.

3.1 Independent Component: higress-plugin-server Repository

We created a GitHub repository https://github.com/higress-group/plugin-server , with the following repository structure:

plugin-server/
├── Dockerfile                  # Core for building the Nginx image
├── nginx.conf                  # Nginx configuration file for serving static files
├── pull_plugins.py             # Plugin download script for pulling plugins from OCI
├── plugins.properties           # Plugin list and version configuration
├── deploy/                     # Kubernetes deployment files (Service & Deployment)
│   └── service.yaml
│   └── deployment.yaml
└── .github/workflows/          # GitHub Action pipeline definition for automated builds and pushes
    └── build-plugin-server-image-and-push.yml

3.2 Build Process: Internal Pulling of OCI Plugins and Transforming into HTTP Services

The core of plugin-server is its Dockerfile:

# Build Stage: Handling Plugins and Metadata
FROM python:3.11-alpine AS builder

# Install system dependencies, including ORAS client
RUN apk add --no-cache \
    wget \
    ca-certificates \
    && update-ca-certificates
RUN set -eux; \
    ORAS_VERSION="1.2.3"; \
    ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/'); \
    wget -O /tmp/oras.tar.gz "https://github.com/oras-project/oras/releases/download/v${ORAS_VERSION}/oras_$(echo ${ORAS_VERSION})_linux_${ARCH}.tar.gz" \
    && tar -zxvf /tmp/oras.tar.gz -C /usr/local/bin \
    && rm -rf /tmp/oras.tar.gz oras \
    && oras version

# Copy and execute Python script to pull plugins
WORKDIR /workspace
COPY pull_plugins.py plugins.properties ./
RUN python3 pull_plugins.py

# Run Stage: Final Image
FROM docker.io/nginx:alpine

# Copy generated files from build stage
COPY --from=builder /workspace/plugins /usr/share/nginx/html/plugins

# Copy Nginx configuration
COPY nginx.conf /etc/nginx/nginx.conf

# Expose port and start Nginx
EXPOSE 8080
CMD ["nginx", "-g", "daemon off;"]

Key Point Analysis:

  • Plugin Pulling: In the builder phase, we use the Python script pull_plugins.py and the ORAS client tool to pull all necessary Wasm plugins from Higress's official Wasm plugin repository.
  • File Organization: The structure is organized as {name}/{version}//plugin.wasm, for example, plugins/ai-proxy/1.0.0/plugin.wasm. This structure is clear and easy to access via HTTP paths.
  • Metadata: Besides the plugin.wasm file itself, we also generate a metadata.txt file that contains information such as plugin name, size, MD5 value, creation/modification time, etc. This provides a foundation for achieving smarter plugin updates in the future (for instance, the gateway under the Always strategy can use MD5 comparisons to determine whether to re-download).
  • Nginx: The final image is based on the lightweight nginx:alpine. nginx.conf configures Nginx as a static file server, enabling autoindex on (facilitating debugging and viewing directory contents) and default_type application/octet-stream (ensuring that Wasm files can be downloaded correctly).
  • Streamlined and Efficient: Multi-stage builds ensure that the final Nginx image only contains the minimal file set required for operation, avoiding unnecessary dependencies and keeping the image size slim.

3.3 Plugin and Image Version Management

We have set a clear version tag of 1.0.0 for the higress-plugin-server image, rather than following the version evolution of higress-gateway and higress-console, primarily considering that it will not require many version releases.

Through GitHub Actions, we have implemented automated build and push processes. When there are updates to the main branch of the higress-plugin-server repository, GitHub Action will automatically trigger a build and push it to Higress's official image repository. In the future, we plan to listen for plugin-related PRs in the main Higress repository to trigger image builds for dynamic updates of plugins.

3.4 Kubernetes Integration

To enable users to conveniently deploy the higress-plugin-server, we have directly integrated its Kubernetes deployment configuration (Deployment and Service) into the Helm Chart of the main Higress repository (higress/helm/core).

  • New Chart Templates: We have added plugin-server-deployment.yaml and plugin-server-service.yaml in the higress/helm/core/templates directory.
  • Configurability: Through values.yaml, users can flexibly configure the number of replicas, image versions, resource limits, and other parameters of the higress-plugin-server.
  • One-Click Enablement: The core change is the introduction of a global Helm parameter, global.enablePluginServer. When users set this parameter to true, the Helm Chart will automatically render and deploy the higress-plugin-server component.
# higress/helm/core/values.yaml
...
global:
  enablePluginServer: false  # Default is false; users can set it to true to enable
...
pluginServer:
  name: "higress-plugin-server"
  replicas: 2  # Default is 2 replicas, recommended for high availability
  image: plugin-server
  hub: higress-registry.cn-hangzhou.cr.aliyuncs.com/higress
  tag: "1.0.0"  # Explicit image version
# ... other resource limits, port configurations, etc.

3.5 Console Interaction: The Final Step for User Experience

To achieve a seamless user experience, we have also modified the higress-console repository. When users choose to enable the plugin-server, the Higress Console will automatically configure the default download URL for Wasm plugins to http://higress-plugin-server.higress-system.svc/plugins/{version}/plugin.wasm.

# higress-console/helm/templates/deployment.yaml
...
spec:
template:
    containers:
      env:
        {{-ifand.Values.global.enablePluginServer(not(hasKey.Values.podEnvs"HIGRESS_ADMIN_WASM_PLUGIN_CUSTOM_IMAGE_URL_PATTERN"))}}
        -name:HIGRESS_ADMIN_WASM_PLUGIN_CUSTOM_IMAGE_URL_PATTERN
          value:"{{ .Values.pluginServer.urlPattern }}"
        {{-end}}
# higress-console/helm/values.yaml
...
pluginServer:
urlPattern:"http://higress-plugin-server.higress-system.svc/plugins/${name}/${version}/plugin.wasm"

Intelligent Judgment: It is worth noting that we have added logic for judgment: if the user has already manually configured the HIGRESS_ADMIN_WASM_PLUGIN_CUSTOM_IMAGE_URL_PATTERN environment variable, the user's configuration will take precedence to avoid automatically overriding the user's intent. This ensures flexibility and compatibility.

Higress Plugin Server User Guide

Now, let's introduce how to experience and use the Higress Plugin Server. We strive to make it as simplified as possible for users to get started.

4.1 One-Click Deployment

Starting from the next version of Higress (expected v2.1.5), you only need to add an additional Helm parameter when installing Higress to simultaneously deploy the core components of Higress and the Wasm Plugin Server:

helm repo add higress.io https://higress.cn/helm-charts
helm install higress -n higress-system higress.io/higress --create-namespace --set global.enablePluginServer=true --render-subchart-notes

After executing the above command, you can check the running Pod named higress-plugin-server via kubectl get pods -n higress-system.

4.2 Higress Console Automatic Adaptation: Default Download from Plugin Server

Once the higress-plugin-server is deployed and running, the Higress Console will automatically detect it and set it as the default download source for Wasm plugins.

No additional configuration is required, and you can see that the default download URL of the plugin has automatically updated in the plugin management page of the Higress Console:

http://higress-plugin-server.higress-system.svc/plugins/{version}/plugin.wasm

4.3 Flexible Configuration: Meeting Personalized Needs

  • Validate Plugin Download: You can enter any Pod inside the Higress Gateway and validate whether the plugin server is functioning correctly by using command-line tools, for example:
kubectl exec -it <higress-gateway-pod> -n higress-system -- curl http://higress-plugin-server.higress-system.svc/plugins/key-auth/1.0.0/metadata.txt

If the metadata information of the plugin is returned, it indicates that the plugin server is working normally.

  • Custom Images and Plugins: If you have customized Wasm plugins, you can build the higress-plugin-server image yourself. Clone the higress-plugin-server repository, modify the plugins.properties file to specify your own plugin list and OCI source, and then execute docker build to build and push it to your image repository. Finally, when installing via Helm, modify the pluginServer.image and pluginServer.tag parameters to specify your custom image.
  • imagePullPolicy Policy: By default, the pulling policy for Higress Wasm plugins is IfNotPresent. In the current version of higress-plugin-server, even if the policy is set to Always, the gateway will request the plugin.wasm file every time. In the future, we will utilize the MD5 and other information in metadata.txt to optimize the pulling logic under the Always policy, achieving intelligent caching and on-demand updates.

4.4 Using Plugin Server with Existing Versions

Since Higress v2.1.5 had not been released at the time of this article's publication, or due to concerns about upgrading existing versions to major versions, if you wish to use the Higress Plugin Server independently, you can refer to https://github.com/alibaba/higress/blob/main/helm/core/templates/plugin-server-deployment.yaml to deploy the Plugin Server independently

and configure the environment variables of the higress-console

HIGRESS_ADMIN_WASM_PLUGIN_CUSTOM_IMAGE_URL_PATTERN=http://higress-plugin-server.higress-system.svc/plugins/${name}/${version}/plugin.wasm

You can then use the Higress Plugin Server.

Looking Ahead

Frankly speaking, the complexity of this feature is not high, but the launch of the Higress Wasm Plugin Server is an important advancement by the Higress team in continuously optimizing user experience and lowering the barriers for enterprise deployment. It allows Wasm plugins to run smoothly in various complex private environments and lays a solid foundation for more advanced plugin management features in the future.

In the future, we will continue to optimize the higress-plugin-server:

  • Smarter Always Policy: Using the hash values in metadata.txt to achieve client-side validation of plugin content and avoid unnecessary repeated downloads.
  • Fine-Grained Version Management: Currently, although Wasm plugins provide a version number mechanism, there is a lack of practical updates for version numbers, and the 1.0.0 version has been used continuously. In the future, we will explore more flexible plugin version management mechanisms and support rollbacks, gray releases, etc.
  • Improved Observability: Enhance the monitoring and logging capabilities of the plugin server to help users quickly locate problems.

We believe that the Higress Wasm plugin ecosystem will continue to evolve with the improvement of these infrastructures. We welcome everyone to actively try the higress-plugin-server and provide us with valuable feedback and suggestions.

If you want to learn more about Alibaba Cloud API Gateway (Higress), please click: https://higress.ai/en/

Computing Developers Open Source Kubernetes cloud native WebAssembly WASM Higress OCI Mechanism
0 1 0
Share on

Read previous post:

Higress MCP Service Management Helps Build a Private MCP Market

Read next post:

20 Years of Financial Investment Data Accumulation, Officially Opening MCP Capabilities

Alibaba Cloud Native Community

631 posts | 55 followers

You may also like

Comments

Alibaba Cloud Native Community

631 posts | 55 followers

Related Products

  • Container Service for Kubernetes

    Alibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.

    Learn More

  • Function Compute

    Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.

    Learn More

  • Managed Service for Prometheus

    Multi-source metrics are aggregated to monitor the status of your business and services in real time.

    Learn More

  • ACK One

    Provides a control plane to allow users to manage Kubernetes clusters that run based on different infrastructure resources

    Learn More

More Posts by Alibaba Cloud Native Community

See All