Connect a Dify application to ARMS Application Monitoring - Application Real-Time Monitoring Service

Application Real-Time Monitoring Service (ARMS) lets you monitor Dify applications using a Python agent. The Python agent, developed by Alibaba Cloud, is a data collection agent for Python that provides automatic instrumentation based on the OpenTelemetry standard.

Step 1: Deploy a Dify application in a container cluster

Build a Dify application by following the instructions in Build a custom AI Q&A assistant using Dify.
In the Container Service for Kubernetes (ACK) cluster that hosts your Dify application, install the agent installation assistant (ack-onepilot), grant access permissions to ARMS resources, and enable Application Monitoring. For more information, see Install the Python agent for Container Service for Kubernetes (ACK) and Container Compute Service (ACS) using the ack-onepilot component.

Step 2: Create the dify-bootstrap configuration item

Create a Dify startup script. This script downloads and installs the agent, and then starts the Dify application.

Obtain the original dify script and make the following modifications:

Add the agent download and installation commands to the beginning of the script.

python3 -m ensurepip --upgrade
pip3 config set global.index-url https://mirrors.aliyun.com/pypi/simple/ && pip3 config set install.trusted-host mirrors.aliyun.com
pip3 install aliyun-bootstrap && aliyun-bootstrap -a install

In the startup section at the end of the script, add the aliyun-instrument startup command.

Expand to view the complete sample script

#!/bin/bash

set -e
python3 -m ensurepip --upgrade
pip3 config set global.index-url https://mirrors.aliyun.com/pypi/simple/ && pip3 config set install.trusted-host mirrors.aliyun.com
pip3 install aliyun-bootstrap && aliyun-bootstrap -a install

mkdir -p logs

if [[ "${MODE}" == "worker" ]]; then

  # Get the number of available CPU cores
  if [ "${CELERY_AUTO_SCALE,,}" = "true" ]; then
    # Set MAX_WORKERS to the number of available cores if not specified
    AVAILABLE_CORES=$(nproc)
    MAX_WORKERS=${CELERY_MAX_WORKERS:-$AVAILABLE_CORES}
    MIN_WORKERS=${CELERY_MIN_WORKERS:-1}
    CONCURRENCY_OPTION="--autoscale=${MAX_WORKERS},${MIN_WORKERS}"
  else
    CONCURRENCY_OPTION="-c ${CELERY_WORKER_AMOUNT:-1}"
  fi

  exec celery -A app.celery worker -P ${CELERY_WORKER_CLASS:-gevent} $CONCURRENCY_OPTION --loglevel ${LOG_LEVEL} \
    -Q ${CELERY_QUEUES:-dataset,generation,mail,ops_trace,app_deletion}

else
  if [[ "${DEBUG}" == "true" ]]; then
    exec flask run --host=${DIFY_BIND_ADDRESS:-0.0.0.0} --port=${DIFY_PORT:-5001} --debug
  else
    exec aliyun-instrument gunicorn \
      --bind "${DIFY_BIND_ADDRESS:-0.0.0.0}:${DIFY_PORT:-5001}" \
      --workers ${SERVER_WORKER_AMOUNT:-1} \
      --worker-class ${SERVER_WORKER_CLASS:-gevent} \
      --timeout ${GUNICORN_TIMEOUT:-200} \
      --access-logfile logs/gunicorn_access.log \
      --error-logfile logs/gunicorn_error.log \
      --access-logformat '%(h)s|%(l)s|%(u)s|%(t)s|%(r)s|%(s)s|%(b)s|%(f)s|%(a)s|DURATION:%(L)s' \
      --log-level info \
      app:app
  fi
fi

Add a configuration item in the ACK console.
1. Log on to the Container Service for Kubernetes console. On the Clusters page, click the name of the target cluster.
2. In the navigation pane on the left, choose Configurations > ConfigMaps. In the dify-system namespace, click Create.
  - ConfigMap Name: dify-bootstrap (example)
  - Name: entrypoint.sh
  - Value: The Dify script that you created in the previous step.
3. Click OK to save the configuration item.

Step 3: Edit ack-dify-api

On the Workloads > Deployments page of the target cluster, navigate to the dify-system namespace. Find ack-dify-api and click Edit in the Actions column.

In the Environment Variable section, click Add and add the following variables.

Type	Variable Name	Variable	Description
Custom	GEVENT_ENABLE	true	Required.
Custom	OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT	true	Collects input/output content. Default value: true. Collection is enabled by default. If disabled, when a user sends a query, the agent collects only the size of fields such as input and output for models, tools, and knowledge bases, but not the content of these fields.
Custom	PROFILER_GENAI_SPLITAPP_ENABLE	false	Splits large model applications. Default value: false. Applications are not split by default. If enabled, the reported data is split into Large Language Model (LLM) sub-applications. Each large model application, such as a Dify Workflow, Agent, or Chat App, corresponds to a separate ARMS application.
Custom	OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT	`<integer_value>`	Limits the length of collected content. Replace `<integer_value>` with an integer that specifies the maximum character length. Default value: Not set. There is no limit by default. If enabled, the length of reported Span attribute values is limited. Attribute values that exceed the specified character length are truncated.

In the Lifecycle section, set Start to ["/bin/bash","/app/api/docker/entrypoint.sh"].
In the Volume section, click Add Local Storage.
- PV Type: Configuration Item
- Name: bootstrap
- Mount Source: dify-bootstrap
- Container Path: /app/api/docker
In the upper-right corner, click Update to save the configuration.
After the agent is installed, you can view the monitoring data on the LLM Application Monitoring > Application List page of the ARMS console. For more information, see View monitoring data.
Important
Ensure that the Dify application is generating data.

Step 4: Monitor dify-plugin-daemon

Dify-Plugin-Daemon is the plugin manager for Dify, which is responsible for executing Dify's large language models (LLMs), plugins, and Agent policies. You can use the Go Agent to monitor Dify-Plugin-Daemon. To enable Go monitoring, you must modify the Dockerfile, recompile the image, and configure environment variables. The Go agent for Dify-Plugin-Daemon can also mount a Python agent for the plugin runtime if required. To connect the agent, perform the following steps:

Modify the Dockerfile and recompile the corresponding image.

For example, make the following modifications to local.dockerfile:

Do not monitor the Python plugin

If you do not need to monitor the Python plugin, modify the Dockerfile as follows:

FROM golang:1.23-alpine AS builder

ARG VERSION=unknown

# Copy the project.
COPY . /app

# Set the working directory.
WORKDIR /app

# Use goproxy if you have network issues.
# ENV GOPROXY=https://goproxy.cn,direct

# Download ARMS instgo.
RUN wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo

RUN chmod 777 instgo

# Build with instgo.
RUN ./instgo go build \
    -ldflags "\
    -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.VersionX=${VERSION}' \
    -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.BuildTimeX=$(date -u +%Y-%m-%dT%H:%M:%S%z)'" \
    -o /app/main cmd/server/main.go

# Copy entrypoint.sh.
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh

FROM ubuntu:24.04

WORKDIR /app

# Check build arguments.
ARG PLATFORM=local

# Install Python 3.12 if PLATFORM is set to local.
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl python3.12 python3.12-venv python3.12-dev python3-pip ffmpeg build-essential \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* \
    && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1;

# Preload tiktoken.
ENV TIKTOKEN_CACHE_DIR=/app/.tiktoken

# Install dify_plugin to speed up environment setup, test uv, and preload tiktoken.
RUN mv /usr/lib/python3.12/EXTERNALLY-MANAGED /usr/lib/python3.12/EXTERNALLY-MANAGED.bk \
    && python3 -m pip install uv \
    && uv pip install --system dify_plugin \
    && python3 -c "from uv._find_uv import find_uv_bin;print(find_uv_bin());" \
    && python3 -c "import tiktoken; encodings = ['o200k_base', 'cl100k_base', 'p50k_base', 'r50k_base', 'p50k_edit', 'gpt2']; [tiktoken.get_encoding(encoding).special_tokens_set for encoding in encodings]"

ENV UV_PATH=/usr/local/bin/uv
ENV PLATFORM=$PLATFORM
ENV GIN_MODE=release

COPY --from=builder /app/main /app/entrypoint.sh /app/

# Run the server. Use sh as the entrypoint to prevent the process from becoming the root process.
# Use bash to recycle resources.
CMD ["/bin/bash", "-c", "/app/entrypoint.sh"]

Monitor the Python plugin

To monitor the Python plugin, modify the Dockerfile as follows:

FROM golang:1.23-alpine AS builder

ARG VERSION=unknown

# Copy the project.
COPY . /app

# Set the working directory.
WORKDIR /app

# Use goproxy if you have network issues.
# ENV GOPROXY=https://goproxy.cn,direct

# Download ARMS instgo.
RUN wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo

RUN chmod 777 instgo

# Build with instgo.
RUN INSTGO_EXTRA_RULES="dify_python" ./instgo go build \
    -ldflags "\
    -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.VersionX=${VERSION}' \
    -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.BuildTimeX=$(date -u +%Y-%m-%dT%H:%M:%S%z)'" \
    -o /app/main cmd/server/main.go

# Copy entrypoint.sh.
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh

FROM ubuntu:24.04

WORKDIR /app

# Check build arguments.
ARG PLATFORM=local

# Install Python 3.12 if PLATFORM is set to local.
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl python3.12 python3.12-venv python3.12-dev python3-pip ffmpeg build-essential \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* \
    && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1;

# Preload tiktoken.
ENV TIKTOKEN_CACHE_DIR=/app/.tiktoken

# Install dify_plugin to speed up environment setup, test uv, and preload tiktoken.
RUN mv /usr/lib/python3.12/EXTERNALLY-MANAGED /usr/lib/python3.12/EXTERNALLY-MANAGED.bk \
    && python3 -m pip install uv \
    && uv pip install --system dify_plugin \
    && python3 -c "from uv._find_uv import find_uv_bin;print(find_uv_bin());" \
    && python3 -c "import tiktoken; encodings = ['o200k_base', 'cl100k_base', 'p50k_base', 'r50k_base', 'p50k_edit', 'gpt2']; [tiktoken.get_encoding(encoding).special_tokens_set for encoding in encodings]"

ENV UV_PATH=/usr/local/bin/uv
ENV PLATFORM=$PLATFORM
ENV GIN_MODE=release

COPY --from=builder /app/main /app/entrypoint.sh /app/

# Run the server. Use sh as the entrypoint to prevent the process from becoming the root process.
# Use bash to recycle resources.
CMD ["/bin/bash", "-c", "/app/entrypoint.sh"]

Deploy the dify-plugin-daemon application and report monitoring data.
Containerized deployment
In the YAML file for the dify-plugin-daemon application, add the following labels under the spec.template.metadata path.
```
labels:
  aliyun.com/app-language: golang
  armsPilotAutoEnable: 'on'
  armsPilotCreateAppName: "dify-daemon-plugin"
```
Non-containerized deployment
When you deploy the component, add the following environment variables.
```
ARMS_LICENSE_KEY: xxx // The ARMS LicenseKey for your account
ARMS_REGION_ID: cn-heyuan // The ID of the region
ARMS_ENABLE: true
ARMS_APP_NAME: dify-plugin-daemon
```
Obtain the LicenseKey by calling the DescribeTraceLicenseKey OpenAPI operation.
View dify-plugin-daemon monitoring data.
On the Application List page, navigate to the dify-plugin-daemon application. The monitoring details are displayed as follows:
Trace details:

(Optional) Step 5: Monitor dify-sandbox

Dify 1.0.0 and later versions introduce dify-sandbox to manage plugin lifecycles and background task calls. To monitor this component, perform the following steps:

Modify the ./build/build_[amd64|arm64].sh file.

Add the instgo download command to the file.
The following is a sample download command. For download commands for other regions and architectures, see Download Instgo.
```
wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
chmod 777 instgo
```
Add the instgo command before the go build command.

The following is an example of the modified script for amd64:

rm -f internal/core/runner/python/python.so
rm -f internal/core/runner/nodejs/nodejs.so
rm -f /tmp/sandbox-python/python.so
rm -f /tmp/sandbox-nodejs/nodejs.so
wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
chmod 777 instgo
echo "Building Python lib"
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 ./instgo go build -o internal/core/runner/python/python.so -buildmode=c-shared -ldflags="-s -w" cmd/lib/python/main.go &&
echo "Building Nodejs lib" &&
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 ./instgo go build -o internal/core/runner/nodejs/nodejs.so -buildmode=c-shared -ldflags="-s -w" cmd/lib/nodejs/main.go &&
echo "Building main" &&
GOOS=linux GOARCH=amd64 ./instgo go build -o main -ldflags="-s -w" cmd/server/main.go
echo "Building env"
GOOS=linux GOARCH=amd64 ./instgo go build -o env -ldflags="-s -w" cmd/dependencies/init.go

Compile the image.

For example, for amd64:

docker build -t sandbox:0.2.9  -f docker/amd64/dockerfile .

Deploy the dify-sandbox application and report monitoring data.
Containerized deployment
In the YAML file for the dify-sandbox application, add the following labels under the spec.template.metadata path.
```
labels:
  aliyun.com/app-language: golang
  armsPilotAutoEnable: 'on'
  armsPilotCreateAppName: "dify-sandbox"
```
Non-containerized deployment
When you deploy the component, add the following environment variables.
```
ARMS_LICENSE_KEY: xxx // The ARMS LicenseKey for your account
ARMS_REGION_ID: cn-heyuan // The ID of the region
ARMS_ENABLE: true
ARMS_APP_NAME: dify-sandbox
```
Obtain the LicenseKey by calling the DescribeTraceLicenseKey OpenAPI operation.
View dify-sandbox monitoring data.
On the Application List page, navigate to the dify-sandbox application. The monitoring details are displayed as follows:
Trace details:

Reference

For more information about how to specify a Python agent version, see How to install an agent of a specific version in a specific region.