Application Real-Time Monitoring Service:Integrate Dify applications with ARMS application monitoring

View monitoring data for workflow applications

Example application details:

The application details page provides tabs including Overview, Performance Analysis, Token Analysis, LLM Operations, Provided Services, Dependent Services, Trace Analysis, Exception Analysis, and Session Analysis. The Overview page displays the following monitoring metrics:

• Statistics cards: Model Call Count, Token Usage, Trace Count, Span Count, Session Count, User Count with day-over-day changes

• Operation Type Distribution pie chart including TASK, TOOL, CHAIN, LLM, RETRIEVER types

• Avg LLM call per request and Request Count Trend line charts

• Model Call Ranking (Top 5) and Request Count User Ranking (Top 5)

• Session Count Trend line chart

Example trace details: The trace collects detailed information about nodes such as LLM calls, retrieval, and tool calls.

The trace details page shows a Trace waterfall chart on the left, with each span annotated with type (TASK, TOOL, LLM, RETRIEVER), duration, and token count. The top section summarizes Trace ID, total duration, application count, interface count, and Total tokens. After clicking a span, the right-side details panel shows the span's metadata (Application Name, Interface Name, Hostname, spanId, start/end time, etc.). LLM-type spans also display input Prompt and Output Messages content.

1. Modify the Dockerfile and rebuild the image. The following example shows modifications to local.dockerfile:

   FROM golang:1.23-alpine AS builder
   
   ARG VERSION=unknown
   
   # copy project
   COPY . /app
   
   # set working directory
   WORKDIR /app
   
   # using goproxy if you have network issues
   # ENV GOPROXY=https://goproxy.cn,direct
   
   # download arms instgo
   RUN wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
   
   RUN chmod 777 instgo
   
   # instgo build
   RUN INSTGO_EXTRA_RULES="dify_python" ./instgo go build \
       -ldflags "\
       -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.VersionX=${VERSION}' \
       -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.BuildTimeX=$(date -u +%Y-%m-%dT%H:%M:%S%z)'" \
       -o /app/main cmd/server/main.go
   
   # copy entrypoint.sh
   COPY entrypoint.sh /app/entrypoint.sh
   RUN chmod +x /app/entrypoint.sh
   
   FROM ubuntu:24.04
   
   WORKDIR /app
   
   # check build args
   ARG PLATFORM=local
   
   # Install python3.12 if PLATFORM is local
   RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl python3.12 python3.12-venv python3.12-dev python3-pip ffmpeg build-essential \
       && apt-get clean \
       && rm -rf /var/lib/apt/lists/* \
       && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1;
   
   # preload tiktoken
   ENV TIKTOKEN_CACHE_DIR=/app/.tiktoken
   
   # Install dify_plugin to speedup the environment setup, test uv and preload tiktoken
   RUN mv /usr/lib/python3.12/EXTERNALLY-MANAGED /usr/lib/python3.12/EXTERNALLY-MANAGED.bk \
       && python3 -m pip install uv \
       && uv pip install --system dify_plugin \
       && python3 -c "from uv._find_uv import find_uv_bin;print(find_uv_bin());" \
       && python3 -c "import tiktoken; encodings = ['o200k_base', 'cl100k_base', 'p50k_base', 'r50k_base', 'p50k_edit', 'gpt2']; [tiktoken.get_encoding(encoding).special_tokens_set for encoding in encodings]"
   
   ENV UV_PATH=/usr/local/bin/uv
   ENV PLATFORM=$PLATFORM
   ENV GIN_MODE=release
   
   COPY --from=builder /app/main /app/entrypoint.sh /app/
   
   # run the server, using sh as the entrypoint to avoid process being the root process
   # and using bash to recycle resources
   CMD ["/bin/bash", "-c", "/app/entrypoint.sh"]

   Note

   The INSTGO_EXTRA_RULES option enables automatic monitoring of the plugin runtime. If you do not want the plugin agent to be launched when the plugin-daemon starts, remove INSTGO_EXTRA_RULES="dify_python" from the build file.

2. Configure environment variables:

   • ECS environment

     Parameter	Example value	Description
     ARMS_LICENSE_KEY	xxx@xxx	The License Key.
     ARMS_REGION_ID	cn-heyuan	The region.
     ARMS_ENABLE	true	Enables or disables the agent.
     ARMS_APP_NAME	dify-plugin-daemon	The application name.

   • Containerized ack-onepilot environment: In the YAML configuration file for the dify-plugin-daemon application, add the following labels under spec.template.metadata.

     labels:
       aliyun.com/app-language: golang
       armsPilotAutoEnable: 'on'
       armsPilotCreateAppName: "dify-daemon-plugin"

3. Deploy and view monitoring data for dify-plugin-daemon. On the Application List page, go to the dify-plugin-daemon application.

   After successful integration, you can view the following monitoring data on the Application Overview page of the ARMS console:

   • Summary metric cards: Request Count, Error Count, Average Latency, Instance Count with day-over-day/week-over-week changes

   • Time series charts: Request Count/1m, Error Count/1m, Latency/1m (including P99, P90, P75, Average), CPU Usage Peak

   • Top 5 rankings: Request Count ranking, Error Count ranking, Average Latency ranking, CPU Usage Peak Instance ranking

   The top navigation tabs include Application Overview, Application Topology, Provided Services, Dependent Services, Trace Analysis, Instance Monitoring, Application Diagnostics, Scenario Analysis, Event Analysis, Application Configuration.

   Trace details:

   Select insight-agent-plugin-daemon in the serviceName filter to view the complete distributed trace of plugin LLM request calls. The Trace waterfall chart shows multiple spans including:

   • GET entry request

   • connect (redis/sqlalchemy connections)

   • SELECT dify (database queries)

   • /v1/sandbox/run (POST 200, sandbox execution, with httpx/go multi-layer calls)

   • UPDATE dify (database updates)

   Span durations range from 131µs to 34.95ms. The right-side Span details panel shows hostname, start/end time, status, and Attributes including net.protocol.name=HTTP, component.name, endpoint=/plugin/tenant_id/dispatch/llm/inv....

4. View plugin monitoring data. After the agent is attached, the plugin-daemon automatically starts the agent for the plugin runtime. Each plugin runtime corresponds to an observable application with the name {plugin_daemon_name}_plugin_{plugin_name}_{plugin_version}. For example, if the application name for the plugin-daemon is local-dify-plugin-daemon and version 0.0.53 of the tongyi plugin is installed, an application named local-dify-plugin-daemon_plugin_tongyi_0.0.53 is automatically created.

1. Modify the Dockerfile and rebuild the corresponding image. Modify the ./build/build_[amd64|arm64].sh file.

   1. Add the command to download instgo. The following is a sample command. For download commands for other regions and architectures, see Download instgo.

      wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
      chmod 777 instgo

   2. Add the instgo command before the go build command. The following is an example for amd64:

      rm -f internal/core/runner/python/python.so
      rm -f internal/core/runner/nodejs/nodejs.so
      rm -f /tmp/sandbox-python/python.so
      rm -f /tmp/sandbox-nodejs/nodejs.so
      wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
      chmod 777 instgo
      echo "Building Python lib"
      CGO_ENABLED=1 GOOS=linux GOARCH=amd64 ./instgo go build -o internal/core/runner/python/python.so -buildmode=c-shared -ldflags="-s -w" cmd/lib/python/main.go &&
      echo "Building Nodejs lib" &&
      CGO_ENABLED=1 GOOS=linux GOARCH=amd64 ./instgo go build -o internal/core/runner/nodejs/nodejs.so -buildmode=c-shared -ldflags="-s -w" cmd/lib/nodejs/main.go &&
      echo "Building main" &&
      GOOS=linux GOARCH=amd64 ./instgo go build -o main -ldflags="-s -w" cmd/server/main.go
      echo "Building env"
      GOOS=linux GOARCH=amd64 ./instgo go build -o env -ldflags="-s -w" cmd/dependencies/init.go
      

2. Configure environment variables.

   • Integrate through Kubernetes containers: In the YAML configuration file for the dify-plugin-daemon application, add the following labels under spec.template.metadata.

     labels:
       aliyun.com/app-language: golang
       armsPilotAutoEnable: 'on'
       armsPilotCreateAppName: "dify-daemon-plugin"
     

   • Manual integration:

     Parameter	Example value	Description
     ARMS_LICENSE_KEY	xxx	The License Key.
     ARMS_REGION_ID	cn-heyuan	The region.
     ARMS_ENABLE	true	Enables or disables the agent.
     ARMS_APP_NAME	dify-plugin-daemon	The application name.

3. Deploy and view sandbox monitoring data. On the Application List page, go to the dify-sandbox application. The monitoring details are as follows:

   Trace details:

   This Trace has a total duration of 8.7s, involving 5 applications and 1039 interfaces. The page indicates Exception Detected. Component calls include redis 575 times, sqlalchemy 337 times, httpx 14 times, etc. The core path in the Span waterfall chart shows SELECT dify queries followed by multi-layer /v1/sandbox/run calls (POST httpx 35.96ms → SERVER go 28.39ms → multi-level INTERNAL go calls). Span details show application name insight-agent-sandbox, interface /v1/sandbox/run, hostname sandbox-7489d68cd5-xcqv8, with normal status code.