AgentLoop:Instrument a Dify application

View workflow monitoring data

The application details page contains the following tabs: Overview, Performance Analysis, Token Analysis, LLM Operations, Provided Services, Dependent Services, Trace Analysis, Exception Analysis, and Session Analysis. The Overview tab displays the following monitoring data:

• Key metric cards: Model Calls, Token Usage (with day-over-day comparison), Traces, Spans, Sessions, and Users

• Operation type distribution pie chart (TASK, TOOL, CHAIN, LLM, RETRIEVER)

• Line charts for average LLM calls per request, request count trend, and session count trend

• Model Call Ranking (Top 5) bar chart and Request Count User Ranking (Top 5) table

Trace details example: The trace details include node information, such as LLM calls, retrieval, and tool invocation.

The trace details page displays a waterfall chart on the left that shows step nodes (types include CHAIN, TOOL, TASK, LLM, and RETRIEVER), each with a Gantt chart time bar. The panel on the right shows detailed information about the selected node, including the application name, endpoint, hostname, spanId, parentSpanId, and status code. The Additional Information section at the bottom allows you to view the input and output message content of the node, such as JSON input for alert analysis and root cause analysis results.

1. Modify the Dockerfile and rebuild the corresponding image. The following code shows an example of a modified local.dockerfile:

   FROM golang:1.23-alpine AS builder
   
   ARG VERSION=unknown
   
   # copy project
   COPY . /app
   
   # set working directory
   WORKDIR /app
   
   # using goproxy if you have network issues
   # ENV GOPROXY=https://goproxy.cn,direct
   
   # download arms instgo
   RUN wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
   
   RUN chmod 777 instgo
   
   # instgo build
   RUN INSTGO_EXTRA_RULES="dify_python" ./instgo go build \
       -ldflags "\
       -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.VersionX=${VERSION}' \
       -X 'github.com/langgenius/dify-plugin-daemon/internal/manifest.BuildTimeX=$(date -u +%Y-%m-%dT%H:%M:%S%z)'" \
       -o /app/main cmd/server/main.go
   
   # copy entrypoint.sh
   COPY entrypoint.sh /app/entrypoint.sh
   RUN chmod +x /app/entrypoint.sh
   
   FROM ubuntu:24.04
   
   WORKDIR /app
   
   # check build args
   ARG PLATFORM=local
   
   # Install python3.12 if PLATFORM is local
   RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl python3.12 python3.12-venv python3.12-dev python3-pip ffmpeg build-essential \
       && apt-get clean \
       && rm -rf /var/lib/apt/lists/* \
       && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.12 1;
   
   # preload tiktoken
   ENV TIKTOKEN_CACHE_DIR=/app/.tiktoken
   
   # Install dify_plugin to speedup the environment setup, test uv and preload tiktoken
   RUN mv /usr/lib/python3.12/EXTERNALLY-MANAGED /usr/lib/python3.12/EXTERNALLY-MANAGED.bk \
       && python3 -m pip install uv \
       && uv pip install --system dify_plugin \
       && python3 -c "from uv._find_uv import find_uv_bin;print(find_uv_bin());" \
       && python3 -c "import tiktoken; encodings = ['o200k_base', 'cl100k_base', 'p50k_base', 'r50k_base', 'p50k_edit', 'gpt2']; [tiktoken.get_encoding(encoding).special_tokens_set for encoding in encodings]"
   
   ENV UV_PATH=/usr/local/bin/uv
   ENV PLATFORM=$PLATFORM
   ENV GIN_MODE=release
   
   COPY --from=builder /app/main /app/entrypoint.sh /app/
   
   # run the server, using sh as the entrypoint to avoid process being the root process
   # and using bash to recycle resources
   CMD ["/bin/bash", "-c", "/app/entrypoint.sh"]

   Note

   The INSTGO_EXTRA_RULES option enables automatic monitoring of the plugin runtime. To prevent the plugin agent from launching with the plugin-daemon, remove INSTGO_EXTRA_RULES="dify_python" from the build file.

2. Configure the environment variables:

   • For ECS environments:

     Parameter	Value	Description
     ARMS_LICENSE_KEY	xxx@xxx	Your License Key.
     ARMS_REGION_ID	cn-heyuan	Your region ID.
     ARMS_ENABLE	true	Enables or disables the agent.
     ARMS_APP_NAME	dify-plugin-daemon	The application name.

   • For containerized ack-onepilot environments: In the YAML configuration for the dify-plugin-daemon application, add the following labels under spec.template.metadata.

     labels:
       aliyun.com/app-language: golang
       armsPilotAutoEnable: 'on'
       armsPilotCreateAppName: "dify-daemon-plugin"

3. Deploy and view monitoring data for dify-plugin-daemon. On the application list page, navigate to the dify-plugin-daemon application. The application overview page displays the following monitoring information:

   • Metric cards: Requests, Errors, Average Latency, and Instances, with day-over-day and week-over-week changes

   • Trend charts: Requests/1m bar chart, Errors/1m line chart, Latency/1m line chart (with P99, P90, and P75 percentiles), and Peak CPU Utilization ring chart

   • Top 5 ranking panels: Request Count, Error Count, Average Latency, and Peak CPU Utilization by Instance

   The trace waterfall chart shows the call chain details of insight-agent-plugin-daemon. The key spans include:

   • GET (python/redis, 1.95 ms)

   • connect (python/sqlalchemy)

   • SELECT dify (sqlalchemy, 34.95 ms)

   • /v1/sandbox/run multi-layer calls (POST httpx → SERVER go net_http_server → INTERNAL, approximately 28–30 ms)

   • UPDATE dify (sqlalchemy, 2.38 ms)

   The details panel on the right shows the hostname, start time, end time, spanId, parentSpanId, status code, and an Attributes list (such as net.protocol.name, component.name, and endpoint) for the selected span.

4. View monitoring data for plugins. After you install the agent, the plugin-daemon automatically starts the agent for the plugin runtime. Each plugin runtime corresponds to a separate observable application, named in the format {plugin_daemon_name}_plugin_{plugin_name}_{plugin_version}. For example, if your plugin-daemon application is named local-dify-plugin-daemon and you have installed version 0.0.53 of the tongyi plugin, an application named local-dify-plugin-daemon_plugin_tongyi_0.0.53 is automatically created. The plugin monitoring overview page displays the following information:

   • Top tabs: Application Overview, Application Topology, Provided Services, Dependent Services, Trace Analysis, Instance Monitoring, Scenario Analysis, and Application Configuration

   • Key metric cards: requests, errors, average latency, and instances (with day-over-day and week-over-week changes)

   • Time-series charts: trends for request count, error count, latency, and peak CPU utilization

   • Top 5 rankings: request count, error count, and average latency, showing plugin endpoint-level calls (such as invoke_llm, get_ai_model_schemas, and invoke_rerank)

1. Modify the build/build_[amd64|arm64].sh file and rebuild the corresponding image.

   1. Add the command to download instgo. The following code shows an example. For download commands for other regions and architectures, see Download instgo.

      wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
      chmod 777 instgo

   2. Add the instgo command before go build. The following code shows an example for the amd64 architecture:

      rm -f internal/core/runner/python/python.so
      rm -f internal/core/runner/nodejs/nodejs.so
      rm -f /tmp/sandbox-python/python.so
      rm -f /tmp/sandbox-nodejs/nodejs.so
      wget "http://arms-apm-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/instgo/instgo-linux-amd64" -O instgo
      chmod 777 instgo
      echo "Building Python lib"
      CGO_ENABLED=1 GOOS=linux GOARCH=amd64 ./instgo go build -o internal/core/runner/python/python.so -buildmode=c-shared -ldflags="-s -w" cmd/lib/python/main.go &&
      echo "Building Nodejs lib" &&
      CGO_ENABLED=1 GOOS=linux GOARCH=amd64 ./instgo go build -o internal/core/runner/nodejs/nodejs.so -buildmode=c-shared -ldflags="-s -w" cmd/lib/nodejs/main.go &&
      echo "Building main" &&
      GOOS=linux GOARCH=amd64 ./instgo go build -o main -ldflags="-s -w" cmd/server/main.go
      echo "Building env"
      GOOS=linux GOARCH=amd64 ./instgo go build -o env -ldflags="-s -w" cmd/dependencies/init.go
      

2. Configure the environment variables.

   • For K8s container deployments: In the YAML configuration for the dify-plugin-daemon application, add the following labels under spec.template.metadata.

     labels:
       aliyun.com/app-language: golang
       armsPilotAutoEnable: 'on'
       armsPilotCreateAppName: "dify-daemon-plugin"
     

   • For manual integration:

     Parameter	Value	Description
     ARMS_LICENSE_KEY	xxx	Your License Key.
     ARMS_REGION_ID	cn-heyuan	Your region ID.
     ARMS_ENABLE	true	Enables or disables the agent.
     ARMS_APP_NAME	dify-plugin-daemon	The application name.

3. Deploy and view monitoring data for the sandbox. On the application list page, navigate to the dify-sandbox application. The Application Overview tab displays the following summary metrics:

   • Requests: 3,051.00 (1.60% day-over-day)

   • Errors: 0.00 (0% day-over-day)

   • Average Latency: 47.7 ms (1.36% day-over-day)

   • Instances: 9 (0% week-over-week)

   The time-series charts show Requests/1m, Errors/1m, Latency/1m (with P99, P90, and P75 curves), and CPU utilization. The Top 5 Provided Services ranking shows the primary request path as /v1/sandbox/run with 3,051 requests, 0 errors, and an average latency of 47.7 ms.

   The ARMS trace details page shows a trace with a total duration of 8.7 seconds, spanning 5 applications and 1,039 endpoints, with detected anomalies. Component tags include redis (575), sqlalchemy (337), net_http_server (16), and httpx (14). The span waterfall chart shows the multi-layer call chain from SELECT dify to /v1/sandbox/run POST 200, illustrating the complete path of Dify calling the sandbox. The span details panel shows the application name insight-agent-sandbox, endpoint /v1/sandbox/run, hostname sandbox-7489d68cd5-xcqv8, with a normal status code. The Attributes section shows fields such as net.protocol.name, component.name, and other detailed information.