Send Dify observability data to Alibaba Cloud ClickHouse Enterprise Edition - ApsaraDB for ClickHouse

This article shows you how to send observability data from your Dify application to Alibaba Cloud ClickHouse Enterprise Edition and analyze it with the one-stop observability service. This process helps you monitor application performance and resolve issues more efficiently.

Prerequisites

Your Alibaba Cloud ClickHouse Enterprise Edition instance must have the one-stop observability service enabled.
This article uses Tongyi Qianwen as an example. You need to obtain an API key to authenticate calls to the large language model (LLM).

Procedure

This example deploys the business service and the OpenTelemetry Collector on the same ECS instance. For production environments, we recommend deploying the business service and the OpenTelemetry Collector separately.

Step 1: Deploy the OpenTelemetry Collector

Configure the ECS instance.
1. Purchase an ECS instance. We recommend assigning a public IP address during purchase for public network access.
2. Add the IP address of the ECS instance to the allowlist of Alibaba Cloud ClickHouse.
  - Private IP address: If the ECS instance and the Alibaba Cloud ClickHouse Enterprise Edition instance are in the same Virtual Private Cloud (VPC), add the private IP address of the ECS instance to the allowlist. In this case, use the VPC endpoint for the Alibaba Cloud ClickHouse instance in the following steps.
  - Public IP address: If the ECS instance cannot connect to the Alibaba Cloud ClickHouse Enterprise Edition instance over the private network, add the public IP address of the ECS instance to the allowlist. In this case, use the public endpoint for the Alibaba Cloud ClickHouse instance in the following steps.
  You can connect to the ECS instance and run the following command to test the connection to the Alibaba Cloud ClickHouse instance. A response of Ok. indicates a successful connection.
```
curl http://cc-xxxxxxx-clickhouse.clickhouseserver.pre.rds.aliyuncs.com:8123
```

Connect to the ECS instance and run the following commands to install otelcol-contrib.

mkdir llm-demo
cd llm-demo
wget -O otelcol-contrib.rpm https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.136.0/otelcol-contrib_0.136.0_linux_amd64.rpm
yum install otelcol-contrib.rpm

Configure the OpenTelemetry Collector to receive observability data over gRPC and HTTP, and send it to the Alibaba Cloud ClickHouse instance over HTTP.

Copy the following command and modify the configuration as described:

In the receivers section, we recommend replacing the endpoint of the OTLP receiver with the endpoint of your OpenTelemetry Collector service.
In the exporters section:
- Configure the connection details for your Alibaba Cloud ClickHouse instance, including endpoint, username, and password.
- Configure the database and table names for storing observability data in your Alibaba Cloud ClickHouse instance. This includes database, traces_table_name, logs_table_name, and metrics_tables.

cat > /etc/otelcol-contrib/config.yaml << EOF
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
    timeout: 5s
    send_batch_size: 5000
exporters:
  clickhouse:
    endpoint: http://cc-xxxxxx-clickhouse.clickhouseserver.pre.rds.aliyuncs.com:8123?dial_timeout=10s&compress=lz4&async_insert=1
    username: test
    password: password
    # ttl: 72h
    traces_table_name: otel_traces
    logs_table_name: otel_logs
    metrics_tables:
      gauge: 
        name: otel_metrics_gauge
      sum: 
        name: otel_metrics_sum
      summary: 
        name: otel_metrics_summary
      histogram: 
        name: otel_metrics_histogram
      exponential_histogram: 
        name: otel_metrics_exp_histogram
    create_schema: false
    timeout: 5s
    database: clickobserve_service
    sending_queue:
      queue_size: 1000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
EOF

Start the OpenTelemetry Collector.
```
service otelcol-contrib restart
```

Step 2: Deploy and start Dify

Run the following command to deploy Dify.

git clone https://github.com/langgenius/dify.git

Run the following commands to prepare the Dify configuration file for editing.
```
cd dify/docker
cp .env.example .env
vim .env
```
In the .env file, set the following parameters:
- Set ENABLE_OTEL to true.
- Set OTEL_EXPORTER_OTLP_PROTOCOL to HTTP.
- Set OTLP_BASE_ENDPOINT to the OpenTelemetry Collector endpoint accessible from the Dify container, such as http://<YOUR_ECS_IP>:4318.
Start the Dify service.

For instructions on installing Docker, see Install and use Docker and Docker Compose.
```
docker compose up -d
```
Go to http://<IP_ADDRESS>/install to register a user and sign in. Replace <IP_ADDRESS> with the IP address of the server running Dify.

Step 3: Create a chat assistant

Hover over your profile picture in the upper-right corner and click Settings.
In the left-side navigation pane, click Model Providers, then select and install a model provider. This example uses Tongyi Qianwen.
After the installation is complete, find the model in the To be configured section. Click Setup, enter your API key, and save.
In the Dify interface, click Studio > Chatbot at the top. Click Create Blank App, name the application, and then click Create.

Define the prompt. The following is an example prompt:

You are an AI assistant for Alibaba Cloud ClickHouse. Your purpose is to answer customer questions about the ClickHouse database product and its features. Your answers must be accurate and based on the provided knowledge base. If a question is outside your knowledge, honestly state that you do not know the answer to maintain integrity. Communicate with customers in a friendly and approachable tone. You can use emojis to enhance the interaction.

In the upper-right corner of the page, click Publish > Run to start a conversation.

The following is an example conversation:

Question

Can you tell me something about ClickHouse?

Example response

Of course! Here is some key information about ClickHouse to help beginners get started quickly:
1. What is ClickHouse?
An open-source columnar database: Designed for large-scale data processing and excels in Online Analytical Processing (OLAP) scenarios.
High performance: Achieves ultra-fast queries through columnar storage, a vectorized execution engine, and native parallel computing.
SQL compatible: Supports standard SQL syntax and extends it with advanced features like window functions and aggregate functions.
Distributed architecture: Supports horizontal scaling and can easily handle petabytes of data.
2. Core features
Columnar storage: Data is stored by column (instead of by row), which reduces I/O overhead and improves compression ratios.
Vectorized execution: Processes data in batches by dividing it into chunks, accelerating computation by leveraging CPU instruction sets.
Multi-engine support:
MergeTree: The default engine, which supports sorting, partitioning, primary keys, and replicas.
Memory: An in-memory engine suitable for temporary data.
......

Step 4: Query and analyze observability data

On the details page of your Alibaba Cloud ClickHouse instance, click All-in-one Observe Suite in the left-side navigation pane to open the service portal.
On the main page, you can query and analyze the observability data from your Dify application.