ApsaraDB for ClickHouse:Export Java observability data to Cloud Database ClickHouse Enterprise Edition

Step 1: Deploy the OpenTelemetry Collector

1. Configure the ECS instance.

   1. Purchase an ECS instance. We recommend assigning a public IP address during purchase for public network access.

   2. Add the IP address of the ECS instance to the allowlist of Alibaba Cloud ClickHouse.

      • Private IP address: If the ECS instance and the Alibaba Cloud ClickHouse Enterprise Edition instance are in the same Virtual Private Cloud (VPC), add the private IP address of the ECS instance to the allowlist. In this case, use the VPC endpoint for the Alibaba Cloud ClickHouse instance in the following steps.

      • Public IP address: If the ECS instance cannot connect to the Alibaba Cloud ClickHouse Enterprise Edition instance over the private network, add the public IP address of the ECS instance to the allowlist. In this case, use the public endpoint for the Alibaba Cloud ClickHouse instance in the following steps.

      You can connect to the ECS instance and run the following command to test the connection to the Alibaba Cloud ClickHouse instance. A response of Ok. indicates a successful connection.

      curl http://cc-xxxxxxx-clickhouse.clickhouseserver.pre.rds.aliyuncs.com:8123

2. Connect to the ECS instance and run the following commands to install otelcol-contrib.

   mkdir llm-demo
   cd llm-demo
   wget -O otelcol-contrib.rpm https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.136.0/otelcol-contrib_0.136.0_linux_amd64.rpm
   yum install otelcol-contrib.rpm

3. Configure the OpenTelemetry Collector to receive observability data over gRPC and HTTP, and send it to the Alibaba Cloud ClickHouse instance over HTTP.

   Copy the following command and modify the configuration as described:

   • In the receivers section, we recommend replacing the endpoint of the OTLP receiver with the endpoint of your OpenTelemetry Collector service.

   • In the exporters section:

     • Configure the connection details for your Alibaba Cloud ClickHouse instance, including endpoint, username, and password.

     • Configure the database and table names for storing observability data in your Alibaba Cloud ClickHouse instance. This includes database, traces_table_name, logs_table_name, and metrics_tables.

   cat > /etc/otelcol-contrib/config.yaml << EOF
   receivers:
     otlp:
       protocols:
         grpc:
           endpoint: 0.0.0.0:4317
         http:
           endpoint: 0.0.0.0:4318
   processors:
     batch:
       timeout: 5s
       send_batch_size: 5000
   exporters:
     clickhouse:
       endpoint: http://cc-xxxxxx-clickhouse.clickhouseserver.pre.rds.aliyuncs.com:8123?dial_timeout=10s&compress=lz4&async_insert=1
       username: test
       password: password
       # ttl: 72h
       traces_table_name: otel_traces
       logs_table_name: otel_logs
       metrics_tables:
         gauge: 
           name: otel_metrics_gauge
         sum: 
           name: otel_metrics_sum
         summary: 
           name: otel_metrics_summary
         histogram: 
           name: otel_metrics_histogram
         exponential_histogram: 
           name: otel_metrics_exp_histogram
       create_schema: false
       timeout: 5s
       database: clickobserve_service
       sending_queue:
         queue_size: 1000
       retry_on_failure:
         enabled: true
         initial_interval: 5s
         max_interval: 30s
         max_elapsed_time: 300s
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [clickhouse]
       metrics:
         receivers: [otlp]
         processors: [batch]
         exporters: [clickhouse]
       logs:
         receivers: [otlp]
         processors: [batch]
         exporters: [clickhouse]
   EOF

4. Start the OpenTelemetry Collector.

   service otelcol-contrib restart

Step 2: Create a Java application

This section uses a simple Spring Boot dice-rolling service as an example.

1. Configure the Java environment.

   1. Deploy a Java environment. We recommend installing Java 17 or later.

   2. Download and install Gradle. We recommend installing Gradle 8 or later.

2. Initialize a Gradle project.

   In this example, the directory is named java-simple. After initialization, the directory structure is as follows:

   java-simple
   ├── app
   │   ├── build.gradle.kts
   │   └── src
   │       ├── main
   │       │   ├── java
   │       │   │   └── org
   │       │   │       └── example
   │       │   │           └── App.java
   │       │   └── resources
   │       └── test
   │           ├── java
   │           │   └── org
   │           │       └── example
   │           │           └── AppTest.java
   │           └── resources
   ├── gradle
   │   ├── libs.versions.toml
   │   └── wrapper
   │       ├── gradle-wrapper.jar
   │       └── gradle-wrapper.properties
   ├── gradle.properties
   ├── gradlew
   ├── gradlew.bat
   └── settings.gradle.kts

3. In the java-simple directory, modify or create the following files:

   build.gradle.kts

   plugins {
     id("java")
     id("org.springframework.boot") version "3.0.6"
     id("io.spring.dependency-management") version "1.1.0"
   }
   sourceSets {
     main {
       java.setSrcDirs(setOf("."))
     }
   }
   repositories {
     mavenCentral()
   }
   dependencies {
     implementation("org.springframework.boot:spring-boot-starter-web")
   }
   

   App.java

   package org.example;
   import org.springframework.boot.Banner;
   import org.springframework.boot.SpringApplication;
   import org.springframework.boot.autoconfigure.SpringBootApplication;
   @SpringBootApplication
   public class App {
     public static void main(String[] args) {
       SpringApplication app = new SpringApplication(App.class);
       app.setBannerMode(Banner.Mode.OFF);
       app.run(args);
     }
   }

   RollController.java

   package org.example;
   import java.util.Optional;
   import java.util.concurrent.ThreadLocalRandom;
   import org.slf4j.Logger;
   import org.slf4j.LoggerFactory;
   import org.springframework.web.bind.annotation.GetMapping;
   import org.springframework.web.bind.annotation.RequestParam;
   import org.springframework.web.bind.annotation.RestController;
   @RestController
   public class RollController {
     private static final Logger logger = LoggerFactory.getLogger(RollController.class);
     @GetMapping("/rolldice")
     public String index(@RequestParam("player") Optional<String> player) {
       int result = this.getRandomNumber(1, 6);
       if (player.isPresent()) {
         logger.info("{} is rolling the dice: {}", player.get(), result);
       } else {
         logger.info("Anonymous player is rolling the dice: {}", result);
       }
       return Integer.toString(result);
     }
     public int getRandomNumber(int min, int max) {
       return ThreadLocalRandom.current().nextInt(min, max + 1);
     }
   }

Step 3: Configure the OpenTelemetry Java Agent

1. Download the OpenTelemetry Java Agent.

   curl -L -O https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

2. Configure the environment variables.

   export JAVA_TOOL_OPTIONS="-javaagent:/../opentelemetry-javaagent.jar" \
     OTEL_TRACES_EXPORTER=otlp \
     OTEL_METRICS_EXPORTER=otlp \
     OTEL_LOGS_EXPORTER=otlp \
     OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
     OTEL_EXPORTER_OTLP_ENDPOINT=http://<IP_ADDRESS>:4318 \
     OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<IP_ADDRESS>:4318/v1/traces \
     OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<IP_ADDRESS>:4318/v1/metrics \
     OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://<IP_ADDRESS>:4318/v1/logs \
     OTEL_METRIC_EXPORT_INTERVAL=15000

   • JAVA_TOOL_OPTIONS: Set this to the actual path of the opentelemetry-javaagent.jar file.

   • Endpoint configurations: In http://<IP_ADDRESS>:4318, replace <IP_ADDRESS> with the IP address of the Otel Collector service endpoint accessible from your Java application.

   • For information about other parameters, see properties-exporters.

Step 4: Start and access the Java application

1. In the project's application directory, run the following commands to start the application.

   In this example, the project's application directory is ../java-simple/app.

   gradle assemble
   java -jar ./build/libs/app.jar

2. In your browser, access http://<IP_ADDRESS>:8080/rolldice. Replace <IP_ADDRESS> with the IP address of the server running the Java application.

   After a successful request, the page returns a random number, as shown in the following example:

   5

Step 5: Query and analyze the observability data

1. In the left navigation pane of the Cloud Database ClickHouse instance details page, click All-in-one Observe Suite and follow the on-screen instructions to open the service portal.

2. On the main page, query and analyze the observability data from your Java application.

   On the Search page in HyperDX, set the data source to otel_traces. Enter the SQL query SELECT Timestamp, ServiceName as service, StatusCode as level, round(Duration / 1e6) as duration, SpanName, and set the sort order to Timestamp DESC. In the left Filters panel, filter by SpanName (for example, GET /rolldice or PUBLISH) and ServiceName (for example, app or langgenius/dify). The results table, which includes columns such as Timestamp, service, level, duration, and SpanName, displays latency details for spans across services.