Consume logs with the Java SDK

Prerequisites

Simple Log Service is activated.
Simple Log Service SDK for Python is initialized.

Background information

You can call the PullLogs operation to retrieve log data from a specified cursor position. Simple Log Service supports applications written in Java, Python, and Go to consume data as individual consumers or as part of a consumer group.

Simple Log Service supports SPL in multiple scenarios, including real-time consumption, scan-based queries, and Logtail collection. For more information, see SPL syntax.

Ensure you have installed the Java SDK for Simple Log Service. For more information, see Install the Java SDK.

SDK consumption

This example shows how to read log data by calling the PullLogs operation.

Parameters

Parameter	Type	Required	Description
project	string	Yes	Project name: A project is the resource management unit of Simple Log Service. It serves as the primary boundary for multi-user isolation and access control. For more information, see Manage projects.
Logstore	string	Yes	Logstore name: A Logstore is the unit for collecting, storing, and querying log data in Simple Log Service. For more information, see Manage Logstores.
shardId	int	Yes	The ID of the shard in the Logstore.

Add Maven dependencies

Add the following dependencies to the pom.xml file in your project's root directory:

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version>2.5.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.openservices</groupId>
  <artifactId>aliyun-log</artifactId>
  <version>0.6.99</version>
</dependency>

Create the `PullLogsDemo.java` file

Sample code:

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.Consts;
import com.aliyun.openservices.log.common.LogGroupData;
import com.aliyun.openservices.log.common.Shard;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.PullLogsRequest;
import com.aliyun.openservices.log.response.ListShardResponse;
import com.aliyun.openservices.log.response.PullLogsResponse;

import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class PullLogsDemo {
    // The endpoint of Simple Log Service. In this example, the endpoint for the China (Hangzhou) region is used. For other regions, replace it with the actual endpoint.
    private static final String endpoint = "cn-hangzhou.log.aliyuncs.com";
    // Reads the AccessKey ID and AccessKey Secret from environment variables.
    private static final String accessKeyId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
    private static final String accessKeySecret = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
    // The project name.
    private static final String project = "your_project";
    // The Logstore name.
    private static final String logStore = "your_logstore";

    public static void main(String[] args) throws Exception {
        // Create a Simple Log Service client.
        Client client = new Client(endpoint, accessKeyId, accessKeySecret);
        // List the shards in the Logstore.
        ListShardResponse resp = client.ListShard(project, logStore);
        System.out.printf("%s has %d shards\n", logStore, resp.GetShards().size());
        Map<Integer, String> cursorMap = new HashMap<Integer, String>();
        for (Shard shard : resp.GetShards()) {
            int shardId = shard.getShardId();
            // Get the cursor to start consumption from the beginning of the shard. To start from the end, use Consts.CursorMode.END.
            cursorMap.put(shardId, client.GetCursor(project, logStore, shardId, Consts.CursorMode.BEGIN).GetCursor());
        }
        try {
            while (true) {
                // Pull logs from each shard.
                for (Shard shard : resp.GetShards()) {
                    int shardId = shard.getShardId();
                    PullLogsRequest request = new PullLogsRequest(project, logStore, shardId, 1000, cursorMap.get(shardId));
                    PullLogsResponse response = client.pullLogs(request);
                    // Logs are returned in LogGroups. You can process them based on your business logic.
                    List<LogGroupData> logGroups = response.getLogGroups();
                    System.out.printf("Get %d logGroup from logStore:%s:\tShard:%d\n", logGroups.size(), logStore, shardId);
                    // After processing the pulled logs, save the cursor for the next pull.
                    cursorMap.put(shardId, response.getNextCursor());
                }
            }
        } catch (LogException e) {
            System.out.println("error code :" + e.GetErrorCode());
            System.out.println("error message :" + e.GetErrorMessage());
            throw e;
        }
    }
}

SPL consumption

This example shows how to use SPL to filter log data when calling the PullLogs operation.