Practical Guide: Deploying an MCP Gateway On‑Premises with Higress and Nacos

By Lu YinRen

Background

In the era of the AI explosion, there are already many AI assistants that help users answer questions through intelligent Q&A combined with RAG. Reliance solely on intelligent Q&A to assist customers in self-service answering is far from adequate. We need AI assistants to directly call upon existing and rich management controls or other interfaces, evolving towards a more powerful intelligent agent. We have chosen the currently hottest and gradually standard MCP as the communication transmission protocol between the model and the interface. There are already many introduction articles about MCP, which will not be elaborated on here.

In the scenario of enterprises providing external services, the MCP Server must address the following issues:

(1) How to maintain sessions using the SSE communication method in multi-instance high-availability service scenarios;

(2) How to dynamically update the MCP tool prompt for fast updating & debugging & verification;

(3) How to authenticate user tool calls in a tenant-isolated cloud service scenario.

Higress can effectively solve the above issue 1 while also having a complete operation monitoring system and a visually easy-to-use console interface. To solve issue 2, we introduced Nacos to handle the registration of backend services and manage the metadata and other information of MCP tools. In the entire MCP service, Higress acts as the MCP Proxy, while Nacos acts as the MCP Registry. The tenant isolation issue in problem 3 will be explained in detail in the authentication section below.

Higress and Nacos are both cloud-native applications. In terms of deployment, we naturally choose to use K8s clusters for cloud-native deployment. At the same time, many enterprises have their dedicated production network environments, generally disconnected from the external network. Therefore, this article will focus on how to utilize the community versions of Higress and Nacos (Apache-2.0 open-source license) for privatized deployment. Due to internal environmental restrictions, we cannot directly operate the K8s cluster through Helm for deployment, so this article will focus on how to perform role-based deployment on the K8s cluster using the docker images of Higress and Nacos.

Through this self-built gateway service, configuration can achieve zero-code expansion of tools. New application registration, expansion of tools under applications, and tool prompt updates can all be quickly accomplished through the service-integrated visual console, with updates and releases swiftly completed, the integration method is extremely simple! Update verification is extremely fast! At the same time, by leveraging Nacos' namespace capabilities, service and tool set isolation can be achieved, providing different users with different MCP tool sets.

Privatized Deployment

Higress

Higress supports three deployment methods: Helm, docker compose, and deployment based on an all-in-one docker image. The official recommendation is to use Helm for deploying production environments, with the necessary modules deployed across different pods. However, due to the aforementioned environmental reasons, we choose to use the third method based on the all-in-one docker image Dockerfile for deployment, deploying the components dependent on Higress as processes within the same pod to achieve high availability through multiple replicas, while also achieving a non-invasive deployment of K8s cluster ingress.

When we first tried to directly reference the docker image for deployment, a wasm plugin error occurred. Upon checking the error message, the problem arose while trying to download the wasm plugin via the OCI address. Moreover, Higress' implementation of MCP functionality relies on the wasm plugin, which is an unavoidable issue.

FROM higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:latest

Independent Deployment of WASM Plugin

The plugin-server project of Higress is designed to address the “pain points of pulling plugins when privatizing the Higress gateway, optimizing the download and management efficiency of plugins,” allowing Higress to download independently deployed plugin libraries via HTTP rather than accessing the external public repository through OCI, thus avoiding issues with network problems leading to the failure to pull plugins. The solution process is mainly divided into the following three steps:

(1) Privatized deployment of the plugin-server

FROM higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/plugin-server:1.0.0

(2) Apply for K8s Service (Cluster IP) for the plugin-server cluster

apiVersion: v1
kind: Service
metadata:
  name: higress-plugin-server
  namespace: higress-system
  labels:
    app: higress-plugin-server
    higress: higress-plugin-server
spec:
  type: ClusterIP
  ports:
    - port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app: higress-plugin-server
    higress: higress-plugin-server

The domain name resolution record format for the DNS built into the K8s cluster for this is <service-name>.<namespace>.svc.cluster.local.

In non-K8s scenarios, a VIP or SLB can be applied for the plugin-server cluster for service discovery and load balancing.

(3) Modify the built-in plugin download address of Higress

According to the examples in GitHub, declare the plugin download address in the Dockerfile based on the Higress image. Here, there is a point to note that the example given in the readme is in the format of environment variables.

In the Dockerfile, the declaration needs to be escaped; the format \${name}/\${version} can be correctly parsed.

...
# Template
ENV HIGRESS_ADMIN_WASM_PLUGIN_CUSTOM_IMAGE_URL_PATTERN=http://[applied k8s service address]/plugins/\${name}/\${version}/plugin.wasm
# MCP WASM plugin download URL
ENV MCP_SERVER_WASM_IMAGE_URL=http://[applied k8s service address]/plugins/mcp-server/1.0.0/plugin.wasm
...

After configuring the independent plugin HTTP download address and redeploying, it can be observed on the server that ports 8080 and 8443 can be properly monitored, indicating that Higress possesses the core data plane components for proxy and gateway functionality and can serve normally.

After resolving the wasm plugin download issue, the Higress service based on the docker image can be successfully initiated and run. However, under this deployment mode, each pod is independent, peer, includes all components, and is a fully functional Higress service, thus high availability needs to be achieved through multiple replicas.

In this deployment mode, it is impractical to maintain services & change configurations through Higress' integrated console. Only configuration changes for a single instance can be performed, and instances cannot synchronize configurations. Therefore, the downside in this mode is that configurations must be maintained within the project code, and whenever changes are necessary, the release process must be followed to push the configurations to each instance. However, in our scenario, the need for configuration changes is minimal.

Sticky Sessions

In the MCP SSE communication method, it is inherently necessary to resolve the issue of sticky sessions, which Higress solves for us based on Redis. After deploying the Redis instance in advance, enable Higress' MCP functionality, update the Redis configuration, and redeploy to utilize MCP functionality.

...
data:
  higress: |-
    mcpServer:
      enable: true
      sse_path_suffix: /sse
      redis:
        address: xxx.redis.zhangbei.rds.aliyuncs.com:6379
        username: ""
        password: "xxx"
        db: 0
...

This configuration file can be maintained within its own project based on the Higress image, and during deployment, the configuration files should be copied to the specified directory (in this deployment mode, all configuration files should be handled this way).

...
# custom config
COPY config/configmaps/higress-config.yaml /data/configmaps/higress-config.yaml
COPY config/mcpbridges/default.yaml /data/mcpbridges/default.yaml
COPY config/secrets/higress-console.yaml /data/secrets/higress-console.yaml
RUN chmod +x /data/configmaps/higress-config.yaml && \
    chmod +x /data/secrets/higress-console.yaml && \
    chmod +x /data/mcpbridges/*
...

When the entire MCP gateway is built and in use, the command PSUBSCRIBE mcp-server-sse:* can be used on Redis to see the following call information.

Custom Build Images

The official images generally require a small size, meeting the minimal runtime requirements, so many functions are actually not integrated into the Higress image. If your enterprise has its own agreed universal image or wants to integrate some new functionalities based on the original, such as using Alibaba Cloud's SLS and cloud monitoring, custom builds based on the all-in-one image's Dockerfile content are required. Here is a note: the glibc required by the envoy module in Higress is version 2.18 or above.

In fact, you only need to transplant the content of Higress's Dockerfile over and then declare the independent deployment WASM plugin download address to achieve customized builds and packaged deployments based on specified images for Higress.

Once the Higress service is constructed, it can proceed to the process of external public access: (1) one is to bind port 8001, where you can view the domain names of related configurations through the Higress console, limited to internal network access only. Note: In this mode, direct configuration changes through the console are not possible; (2) the other is to bind port 8080 to provide the domain name of the MCP gateway service.

The complete Dockerfile is as follows:

FROM [Internal base image]

# The following comes from the Higress all‑in‑one Dockerfile
ARG HUB=higress-registry.cn-hangzhou.cr.aliyuncs.com/higress
...

# Template
ENV HIGRESS_ADMIN_WASM_PLUGIN_CUSTOM_IMAGE_URL_PATTERN=http://[requested k8s service address]/plugins/\${name}/\${version}/plugin.wasm
# MCP WASM plugin download URL
ENV MCP_SERVER_WASM_IMAGE_URL=http://[requested k8s service address]/plugins/mcp-server/1.0.0/plugin.wasm
...

# Note: The Dockerfile will download the yq binary for the corresponding CPU architecture from GitHub.
# In an internal enterprise network, you can download it in advance.
COPY ./yq_linux_[arch] /usr/local/bin/yq
...

# Custom config
COPY config/configmaps/higress-config.yaml /data/configmaps/higress-config.yaml
COPY config/mcpbridges/default.yaml /data/mcpbridges/default.yaml
COPY config/secrets/higress-console.yaml /data/secrets/higress-console.yaml
RUN chmod +x /data/configmaps/higress-config.yaml && \
    chmod +x /data/secrets/higress-console.yaml && \
    chmod +x /data/mcpbridges/*
...

Nacos

The deployment of Nacos is relatively simple. Besides directly operating the K8s cluster through kubectl or nacos-operator tools for deployment, it can also be deployed based on the nacos-server image Dockerfile. Due to the internal environmental issues mentioned above, we choose to deploy the service on the K8s cluster based on the nacos-server image.

FROM nacos-registry.cn-hangzhou.cr.aliyuncs.com/nacos/nacos-server:latest

Cluster Mode Deployment

The consistency protocol used in Nacos cluster mode is based on Raft implementation, so a minimum of 3 instances must be deployed.

In the dockerfile referencing the nacos-server image, declare the cluster deployment mode. We check Nacos' startup script and find that if the peer-finder (plugin) directory does not exist and the $NACOS_SERVERS variable is defined, the values in the $NACOS_SERVERS variable will be written into the $CLUSTER_CONF file. The default path for the $CLUSTER_CONF file is /home/nacos/conf/cluster.conf, which defines the static member address list of the Nacos cluster, read when the cluster first starts, to inform each node where its “neighbors” are so they can discover each other, establish connections, and initialize the Raft consensus protocol.

...
PLUGINS_DIR="/home/nacos/plugins/peer-finder"
function print_servers() {
   if [[ ! -d "${PLUGINS_DIR}" ]]; then
    echo "" >"$CLUSTER_CONF"
    for server in ${NACOS_SERVERS}; do
      echo "$server" >>"$CLUSTER_CONF"
    done
  else
    bash $PLUGINS_DIR/plugin.sh
    sleep 30
  fi
}
...

Therefore, we can maintain the current [instance IP:port] list in the dockerfile for the Nacos cluster to read and initialize when starting.

...
ENV MODE=cluster

ENV NACOS_AUTH_TOKEN=xxx
ENV NACOS_AUTH_IDENTITY_KEY=xxx
ENV NACOS_AUTH_IDENTITY_VALUE=xxx

ENV NACOS_SERVERS="10.0.0.1:8848 10.0.0.2:8848 10.0.0.3:8848"

# nacos username and password
ENV NACOS_USERNAME=xxx
ENV NACOS_PASSWORD=xxx
...

Dynamic Discovery Between Instances

The disadvantage of the above static IP list method is obvious. It is a static configuration, and when there is a scaling change in the cluster, instances cannot automatically update the member IP list, which requires manual modification and publishing, a very tedious process that may seriously affect the stability of online services. Moreover, in a cloud-native containerized context, IPs are not fixed and may change anytime due to failover, maintaining a static IP list goes against cloud-native principles. This method is totally not recommended in a production environment.

Returning to the docker-startup.sh script, the peer-finder plugin can be used to achieve inter-instance discovery, replacing the need to manually maintain the cluster.conf file. The peer-finder plugin operates based on the domain name of the K8s cluster Headless Service and will execute commands similar to nslookup to find all healthy Pods' IPs under the Service, analogous to service discovery capabilities source script, thus eliminating the need to maintain instance IP lists manually.

However, the peer-finder's operation relies on the StatefulSet instance deployment mode, which requires each instance to have a fixed instance name. Due to our internal environmental limitations, we are currently only deploying stateless instances, so we cannot use peer-finder for this purpose. Yet, we can refer to the implementation ideas of the peer-finder script to write our own startup script.

(1) First, apply for a Headless Service for the Nacos cluster.

apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  namespace: mcp-nacos
  labels:
    app: mcp-nacos
    nacos: mcp-nacos
spec:
  clusterIP: None
  ports:
  - name: peer-finder-port
    port: 8848
    protocol: TCP
    targetPort: 8848
  selector:
    app: mcp-nacos
  sessionAffinity: None
  type: ClusterIP

(2) Here, modify the nacos-docker startup script to provide a simple implementation (for reference only)

...
Original docker-startup.sh content
...

# New content
# Comment out the JAVA startup command
# exec $JAVA ${JAVA_OPT}
export JAVA_OPT # Export JAVA startup parameters so they can be read below

HEADLESS_SERVICE_FQDN="xxx.svc.cluster.local"
CLUSTER_CONF_FILE="/home/nacos/conf/cluster.conf"
UPDATE_SCRIPT="/home/nacos/bin/update-cluster.sh" # Atomic update script
NACOS_START_CMD="$JAVA $JAVA_OPT"

# 1. Dynamically create the update-cluster.sh script
cat > ${UPDATE_SCRIPT} << 'EOF'
#!/bin/bash
set -e
NACOS_PORT=${NACOS_APPLICATION_PORT:-8848}
CLUSTER_CONF_FILE="/home/nacos/conf/cluster.conf"
TMP_CONF_FILE="/home/nacos/conf/cluster.conf.tmp"
> "${TMP_CONF_FILE}"

# Read the raw nslookup output from stdin
awk '
/^Name:/ { flag=1; next }
flag && /^Address:/ { print $2; flag=0 }
' | while IFS= read -r ip; do
    if [ -n "$ip" ]; then
      echo "${ip}:${NACOS_PORT}" >> "${TMP_CONF_FILE}"
    fi
done

# Sort to keep file content stable and avoid unnecessary updates
sort -o "${TMP_CONF_FILE}" "${TMP_CONF_FILE}"

# Only update when new and old configs differ
# Check whether the old file exists
if [ ! -f "${CLUSTER_CONF_FILE}" ] || ! cmp -s "${TMP_CONF_FILE}" "${CLUSTER_CONF_FILE}"; then
    echo "[$(date)][update-script] Peer list changed. Updating config."
    mv "${TMP_CONF_FILE}" "${CLUSTER_CONF_FILE}"
    echo "[$(date)][update-script] cluster.conf updated:"
    cat "${CLUSTER_CONF_FILE}"
else
    rm "${TMP_CONF_FILE}"
fi
EOF
chmod +x ${UPDATE_SCRIPT}

# 2. Initialization loop before startup
MAX_INIT_RETRIES=30
RETRY_COUNT=0
MIN_PEERS=3 # Expected minimum replica count of the cluster
echo "[INFO] Initializing cluster config. Waiting for at least ${MIN_PEERS} peers to be available..."
while true; do
  # Pipe the nslookup output directly into the update script
  nslookup "${HEADLESS_SERVICE_FQDN}" | ${UPDATE_SCRIPT}

  # Check the line count of the generated config file
  LINE_COUNT=$(wc -l < "${CLUSTER_CONF_FILE}")

  if [ "${LINE_COUNT}" -ge "${MIN_PEERS}" ]; then
    echo "[INFO] Initial cluster.conf is ready with ${LINE_COUNT} peers."
    break
  fi

  RETRY_COUNT=$((RETRY_COUNT+1))
  if [ "${RETRY_COUNT}" -gt "${MAX_INIT_RETRIES}" ]; then
    echo "[WARN] Could not find ${MIN_PEERS} peers after ${MAX_INIT_RETRIES} retries. Starting with ${LINE_COUNT} peers found."
    break
  fi

  echo "[INFO] Found ${LINE_COUNT} peers. Waiting for more... Retrying in 5 seconds."
  sleep 5
done

# 3. Start our own monitoring loop in the background
(
  while true; do
    sleep 15 # Check every 15 seconds
    echo "[$(date)][monitor] Checking for peer updates..."
    nslookup "${HEADLESS_SERVICE_FQDN}" | ${UPDATE_SCRIPT}
  done
) &

# 4. Start the main Nacos process
echo "[INFO] Starting Nacos server..."
exec sh -c "${NACOS_START_CMD}"

This way, the member IP list in the cluster.conf file will be automatically updated.

It is still recommended to use StatefulSet deployment mode for online production, combined with the capabilities of peer-finder to achieve mutual discovery between instances, rather than using stateless instances and writing scripts to achieve this. We will also upgrade to the StatefulSet mode for deployment in the future.

Configure External MySQL

In the cluster deployment mode, the built-in Derby database of Nacos, which does not support data sharing, cannot be used. An external MySQL database needs to be configured. After deploying the MySQL instance in advance, initialize the tables according to the mysql-schema.sql database configuration file within Nacos, and then write the MySQL configuration information into the Dockerfile.

...
# mysql config
ENV SPRING_DATASOURCE_PLATFORM=mysql
ENV MYSQL_DATABASE_NUM=1
ENV MYSQL_SERVICE_HOST=xxx.mysql.zhangbei.rds.aliyuncs.com
ENV MYSQL_SERVICE_PORT=3306
ENV MYSQL_SERVICE_DB_NAME=nacos
ENV MYSQL_SERVICE_USER=xxx
ENV MYSQL_SERVICE_PASSWORD=xxx
...

When exposing services for Nacos, only the 8080 port of the Nacos console needs to be exposed and restricted to internal network access. This is because Nacos is used internally for maintaining and managing MCP tool metadata information without user-side awareness; and both Higress and Nacos are deployed in the internal K8s cluster, with internal communication being routed through K8s' Service, so there is no need to expose Nacos' 8848 port to the public network.

Apply for K8s Service for Higress Usage

Note that Higress pulls/subscribes to configurations in Nacos via gRPC calls, and the Service needs to expose both ports 8848 and 9848 for Higress' use.

apiVersion: v1
kind: Service
metadata:
  name: pre-oss-mcp-nacos-endpoint
  namespace: aso-oss-mcp-nacos
  labels:
    app: mcp-nacos
    nacos: mcp-nacos
spec:
  type: ClusterIP
  ports:
  - name: subscribe-port
    port: 8848
    protocol: TCP
    targetPort: 8848
  - name: grpc-port
    port: 9848
    protocol: TCP
    targetPort: 9848
  selector:
    app: nacos

Similarly, if you want to use an internal corporate image or want to integrate some new functionalities based on the original, such as using Alibaba Cloud's SLS and cloud monitoring functionalities, you can also perform customized build deployment based on Nacos' Dockerfile.

Authentication

Higress itself provides rich authentication capabilities. If your enterprise has already built its gateway based on Higress and utilized the authentication capabilities provided by Higress, this scenario allows for the direct reuse of the original solution.

In another scenario, there may be multiple service providers within the enterprise, each using different authentication methods. As shown in the image below, a certain service provider will perform RAM authentication on the user Cookie carried in the request through an interceptor, while another service provider will use Tengine Lua scripts for custom authentication, and subsequent registered services may have other authentication methods.

On one hand, we do not want to use Higress' authentication capabilities to override all authentication scenarios, as the development and maintenance costs would be too high. We prioritize directly reusing existing authentication capabilities from service providers; on the other hand, if authentication is needed at the gateway layer, it requires storing AK or authentication information on Higress, which is not a suitable practice from a security standpoint.

The recommended approach here is to pass the authentication information directly to the service provider when invoking MCP tools, allowing the service provider to complete the authentication.

MCP Verification

https://www.nacos.io/blog/nacos-gvr7dx_awbbpb_lup4w7e1cv6wktac/#_top

According to the operational example in the documentation, we can conduct a simple full-link test verification. The process mainly consists of the following three steps:

(1) Register the service in Nacos and configure the metadata information for the MCP tool:

Create service information in the public namespace

Register your service as a permanent instance on the machine (here, for the swift validation of the black screen login machine operation; in the online production environment, white screen operation is still needed).

curl -X POST 'http://127.0.0.1:8848/nacos/v1/ns/instance?namespaceId=[namespace]&serviceName=[service_name]&groupName=[group_name]&ip=[服务域名]&port=[服务端口]&ephemeral=false'

After registration, you will be able to see the registered service configuration and health status on the Nacos console.

Next, configure the MCP tool on the Nacos console, adding a simple tool that can select a no-parameter GET interface and publish it.

{
  "requestTemplate": {
    "url": "/xxx/list.json",
    "method": "GET",
    "headers": [],
    "argsToUrlParam": true
  },
  "responseTemplate": {
    "body": "{{.}}"
  }
}

(2) Configure the source of the MCP Nacos service in Higress:

To quickly test, we have disabled Nacos authentication; however, it is recommended to enable Nacos authentication in the online environment.