SLS SDK supports multiple programming languages. This topic describes how to use SLS SDK for Java to consume log data by calling the PullLogs operation, with or without SPL filtering.
Background
Call the PullLogs operation to query log data by cursor. For more information, see PullLogs. Java, Python, and Go applications can consume SLS data as consumers or consumer groups.
SLS supports SPL (Simple Log Service Processing Language) for real-time consumption, scan-based queries, and Logtail collection. For more information, see SPL syntax.
Procedure
Install the SLS SDK for Java before proceeding. For more information, see Install the Java SDK.
Use the SDK
The following example calls the PullLogs operation to consume log data. For more information, see PullLogs.
Parameters
|
Parameter
|
Type
|
Required
|
Description
|
|
project
|
string
|
Yes
|
The name of the project. For more information, see Manage projects.
|
|
logStore
|
string
|
Yes
|
The name of the Logstore. Logstores collect, store, and query logs. For more information, see Manage a logstore.
|
|
shardId
|
int
|
Yes
|
The ID of the shard in the Logstore. For more information, see Shard.
|
Add Maven dependencies
Open the pom.xml file in the root directory of your Java project and add the following code:
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.5.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.openservices</groupId>
<artifactId>aliyun-log</artifactId>
<version>0.6.99</version>
</dependency>
Create a file named PullLogsDemo.java
Sample code:
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.Consts;
import com.aliyun.openservices.log.common.LogGroupData;
import com.aliyun.openservices.log.common.Shard;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.PullLogsRequest;
import com.aliyun.openservices.log.response.ListShardResponse;
import com.aliyun.openservices.log.response.PullLogsResponse;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class PullLogsDemo {
// The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint.
private static final String endpoint = "cn-hangzhou.log.aliyuncs.com";
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
private static final String accessKeyId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
private static final String accessKeySecret = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// The name of the project.
private static final String project = "your_project";
// The name of the Logstore.
private static final String logStore = "your_logstore";
public static void main(String[] args) throws Exception {
// Create a client for Simple Log Service.
Client client = new Client(endpoint, accessKeyId, accessKeySecret);
// Query the shards of the Logstore.
ListShardResponse resp = client.ListShard(project, logStore);
System.out.printf("%s has %d shards\n", logStore, resp.GetShards().size());
Map<Integer, String> cursorMap = new HashMap<Integer, String>();
for (Shard shard : resp.GetShards()) {
int shardId = shard.getShardId();
// Use the BEGIN cursor or obtain a specific cursor to consume log data. If you want to use the END cursor to consume log data, use Consts.CursorMode.END.
cursorMap.put(shardId, client.GetCursor(project, logStore, shardId, Consts.CursorMode.BEGIN).GetCursor());
}
try {
while (true) {
// Obtain logs from each shard.
for (Shard shard : resp.GetShards()) {
int shardId = shard.getShardId();
PullLogsRequest request = new PullLogsRequest(project, logStore, shardId, 1000, cursorMap.get(shardId));
PullLogsResponse response = client.pullLogs(request);
// Obtain logs from log groups by logic. Logs are usually stored in log groups.
List<LogGroupData> logGroups = response.getLogGroups();
System.out.printf("Get %d logGroup from logStore:%s:\tShard:%d\n", logGroups.size(), logStore, shardId);
// Move the cursor after the obtained logs are processed.
cursorMap.put(shardId, response.getNextCursor());
}
}
} catch (LogException e) {
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}
Use the SDK and SPL
The following example calls the PullLogs operation for SPL-based log consumption. For more information, see PullLogs.
Parameters
|
Parameter
|
Type
|
Required
|
Description
|
|
project
|
string
|
Yes
|
The name of the project. For more information, see Manage projects.
|
|
logStore
|
string
|
Yes
|
The name of the Logstore. Logstores collect, store, and query logs. For more information, see Manage a logstore.
|
|
shardId
|
int
|
Yes
|
The ID of the shard in the Logstore. For more information, see Shard.
|
Add Maven dependencies
Open the pom.xml file in the root directory of your Java project and add the following code:
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.5.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.openservices</groupId>
<artifactId>aliyun-log</artifactId>
<version>0.6.99</version>
</dependency>
Create a file named PullLogsWithSPLDemo.java
Sample code:
import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.*;
import com.aliyun.openservices.log.common.Consts;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.PullLogsRequest;
import com.aliyun.openservices.log.response.ListShardResponse;
import com.aliyun.openservices.log.response.PullLogsResponse;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class PullLogsWithSPLDemo {
// The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint.
private static final String endpoint = "cn-hangzhou.log.aliyuncs.com";
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
private static final String accessKeyId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
private static final String accessKeySecret = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// The name of the project.
private static final String project = "your_project";
// The name of the Logstore.
private static final String logStore = "your_logstore";
public static void main(String[] args) throws Exception {
// Create a client for Simple Log Service.
Client client = new Client(endpoint, accessKeyId, accessKeySecret);
// Query the shards of the Logstore.
ListShardResponse resp = client.ListShard(project, logStore);
System.out.printf("%s has %d shards\n", logStore, resp.GetShards().size());
Map<Integer, String> cursorMap = new HashMap<Integer, String>();
for (Shard shard : resp.GetShards()) {
int shardId = shard.getShardId();
// Use the BEGIN cursor or obtain a specific cursor to consume log data. If you want to use the END cursor to consume log data, use Consts.CursorMode.END.
cursorMap.put(shardId, client.GetCursor(project, logStore, shardId, Consts.CursorMode.BEGIN).GetCursor());
}
try {
while (true) {
// Obtain logs from each shard.
for (Shard shard : resp.GetShards()) {
int shardId = shard.getShardId();
PullLogsRequest request = new PullLogsRequest(project, logStore, shardId, 1000, cursorMap.get(shardId));
request.setQuery("* | where cast(body_bytes_sent as bigint) > 14000");
request.setPullMode("scan_on_stream");
PullLogsResponse response = client.pullLogs(request);
// Obtain logs from log groups by logic. Logs are usually stored in log groups.
List<LogGroupData> logGroups = response.getLogGroups();
System.out.printf("Get %d logGroup from logStore:%s:\tShard:%d\n", logGroups.size(), logStore, shardId);
// Move the cursor after the obtained logs are processed.
cursorMap.put(shardId, response.getNextCursor());
}
}
} catch (LogException e) {
System.out.println("error code :" + e.GetErrorCode());
System.out.println("error message :" + e.GetErrorMessage());
throw e;
}
}
}