This topic describes how to use Aliyun Log Java Producer to write logs to Simple Log Service.
Prerequisites
A Resource Access Management (RAM) user is created, and the required permissions are granted to the RAM user. For more information, see Create a RAM user and grant permissions to the RAM user.
The ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables are configured. For more information, see Configure environment variables.
Important The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. We recommend that you use the AccessKey pair of a RAM user to call API operations or perform routine O&M.
We recommend that you do not save the AccessKey ID or AccessKey secret in your project code. Otherwise, the AccessKey pair may be leaked, and the security of all resources within your account may be compromised.
Simple Log Service SDK for Java is installed. For more information, see Install Simple Log Service SDK for Java.
Background information
Aliyun Log Java Producer is a high-performance class library provided for Java applications that run in big data and high concurrency scenarios. Compared with Simple Log Service API and Simple Log Service SDK for Java, Aliyun Log Java Producer provides multiple benefits that are related to log writing, such as high performance, logic isolation between computing and I/O, and resource management.
Features
Thread safety: All methods exposed by Aliyun Log Java Producer are thread-safe.
Asynchronous transmission: In most cases, Aliyun Log Java Producer immediately returns a response after you call the send method exposed by Aliyun Log Java Producer. Aliyun Log Java Producer caches and merges the data that you want to send and then sends the merged data in batches to improve throughput.
Automatic retries: Aliyun Log Java Producer retries requests based on the maximum number of retries and backoff time that you specify.
Behavior tracing: You can check whether data is sent and obtain information about retry attempts based on a Callback object or a Future object. This helps you trace issues and make decisions.
Contextual query: The logs that are generated by a producer are in the same context. You can view the logs that are generated before and after a specified log in Simple Log Service.
Graceful shutdown: All data that is cached by Aliyun Log Java Producer can be processed and you can receive notifications after you call the close method to exit your program.
Benefits
High performance
If you want to write a large amount of data to Simple Log Service by using limited resources, you must implement complex control logic to meet throughput requirements. You must configure multiple threads, cache policies, and send data in batches. You must also consider scenarios in which retries are required if data fails to be written. Aliyun Log Java Producer meets the preceding requirements. It provides high performance and simplifies the development process.
Asynchronous mode without data blocking
If the available memory is sufficient, Aliyun Log Java Producer caches the data that needs to be sent to Logstores. If you call the send method, Aliyun Log Java Producer can immediately return a response without blocking. This achieves logic isolation between computing and I/O. Then, you can obtain the result of data sending based on a specified Callback object or a returned Future object.
Resource management
You can configure parameters to manage the memory size that is used by Aliyun Log Java Producer to cache the data to send. You can also specify the number of threads that are used to send data. This prevents Aliyun Log Java Producer from consuming resources without limits and allows you to balance resource consumption and write throughput based on your business requirements.
Install Aliyun Log Java Producer
To use Aliyun Log Java Producer in a Maven project, you need to only add related dependencies to pom.xml. Maven automatically downloads related JAR packages. In this example, add the following code to <dependencies>:
<dependency>
<groupId>com.aliyun.openservices</groupId>
<artifactId>aliyun-log-producer</artifactId>
<version>0.3.10</version>
</dependency>
The jar-with-dependency version can resolve the version conflicts of dependencies. Add the following code to <dependencies>:
<dependency>
<groupId>com.aliyun.openservices</groupId>
<artifactId>aliyun-log</artifactId>
<version>0.6.35</version>
<classifier>jar-with-dependencies</classifier>
</dependency>
Examples
After you install Aliyun Log Java Producer, you can use the library to write Java code.
Note The underlying mechanism of Aliyun Log Java Producer calls the PutLogs operation to upload logs. The size of raw logs that can be uploaded each time is limited. For more information, see Data read and write and PutLogs.
Simple Log Service provides the following basic resources: projects, Logstores, shards, Logtail configurations, machine groups, LogItems, field names, and field values. Simple Log Service limits the numbers of projects, Logstores, shards, Logtail configurations, and machine groups, the size of a LogItem, and the lengths of the field name and field value for each LogItem. For more information, see Basic resources.
The first time you run code, you must enable the indexing feature for your Logstore in the Simple Log Service console. Then, wait for 1 minute and query logs.
If you query logs in the Simple Log Service console and the value length of a field in the returned logs exceeds the upper limit, the field value is truncated, and the excess part is not used for analysis. For more information, see Create indexes.
Callback
In this example, a file named SampleProducerWithCallback.java is created, and the generated logs are uploaded to Simple Log Service.
Sample code
import com.aliyun.openservices.aliyun.log.producer.Callback;
import com.aliyun.openservices.aliyun.log.producer.LogProducer;
import com.aliyun.openservices.aliyun.log.producer.Producer;
import com.aliyun.openservices.aliyun.log.producer.ProducerConfig;
import com.aliyun.openservices.aliyun.log.producer.ProjectConfig;
import com.aliyun.openservices.aliyun.log.producer.Result;
import com.aliyun.openservices.aliyun.log.producer.errors.ProducerException;
import com.aliyun.openservices.log.common.LogItem;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicLong;
public class SampleProducerWithCallback {
private static final Logger LOGGER = LoggerFactory.getLogger(SampleProducerWithCallback.class);
private static final ExecutorService threadPool = Executors.newFixedThreadPool(10);
public static void main(String[] args) throws InterruptedException {
final String project = "example-project";
final String logstore = "example-logstore";
String endpoint = "example-endpoint";
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
String accessKeyId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKeySecret = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
ProducerConfig producerConfig = new ProducerConfig();
final Producer producer = new LogProducer(producerConfig);
producer.putProjectConfig(new ProjectConfig(project, endpoint, accessKeyId, accessKeySecret));
int nTask = 100;
// The number of logs that have finished (either successfully send, or failed).
final AtomicLong completed = new AtomicLong(0);
final CountDownLatch latch = new CountDownLatch(nTask);
for (int i = 0; i < nTask; ++i) {
threadPool.submit(
new Runnable() {
@Override
public void run() {
//The maximum size of a LogItem (key) is 128 bytes. The maximum size of a LogItem (value) is 1 MB.
LogItem logItem = new LogItem();
logItem.PushBack("key1", "foo");
logItem.PushBack("key2", "bar");
try {
producer.send(
project,
logstore,
"your-topic",
"your-source",
logItem,
new SampleCallback(project, logstore, logItem, completed));
} catch (InterruptedException e) {
LOGGER.warn("The current thread has been interrupted during send logs.");
} catch (Exception e) {
LOGGER.error("Failed to send log, logItem={}, e=", logItem, e);
} finally {
latch.countDown();
}
}
});
}
// The following logic is implemented only when the process exits.
latch.await();
threadPool.shutdown();
try {
producer.close();
} catch (InterruptedException e) {
LOGGER.warn("The current thread has been interrupted from close.");
} catch (ProducerException e) {
LOGGER.info("Failed to close producer, e=", e);
}
LOGGER.info("All log complete, completed={}", completed.get());
}
private static final class SampleCallback implements Callback {
private static final Logger LOGGER = LoggerFactory.getLogger(SampleCallback.class);
private final String project;
private final String logStore;
private final LogItem logItem;
private final AtomicLong completed;
SampleCallback(String project, String logStore, LogItem logItem, AtomicLong completed) {
this.project = project;
this.logStore = logStore;
this.logItem = logItem;
this.completed = completed;
}
@Override
public void onCompletion(Result result) {
try {
if (result.isSuccessful()) {
LOGGER.info("Send log successfully.");
} else {
LOGGER.error(
"Failed to send log, project={}, logStore={}, logItem={}, result={}",
project,
logStore,
logItem.ToJsonString(),
result);
}
} finally {
completed.getAndIncrement();
}
}
}
}
Expected result
{"__source__":"your-source","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658715477","__time__":"1658715473","__topic__":"your-topic","key1":"foo","key2":"bar"}
{"__source__":"your-source","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658715477","__time__":"1658715473","__topic__":"your-topic","key1":"foo","key2":"bar"}
{"__source__":"your-source","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658715477","__time__":"1658715473","__topic__":"your-topic","key1":"foo","key2":"bar"}
{"__source__":"your-source","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658715477","__time__":"1658715473","__topic__":"your-topic","key1":"foo","key2":"bar"}
......
For more information, see Aliyun Log Java Producer.
Future
In this example, a file named SampleProducerWithFuture.java is created, and the generated logs are uploaded to Simple Log Service.
Sample code
import com.aliyun.openservices.aliyun.log.producer.LogProducer;
import com.aliyun.openservices.aliyun.log.producer.Producer;
import com.aliyun.openservices.aliyun.log.producer.ProducerConfig;
import com.aliyun.openservices.aliyun.log.producer.ProjectConfig;
import com.aliyun.openservices.aliyun.log.producer.Result;
import com.aliyun.openservices.aliyun.log.producer.errors.ProducerException;
import com.aliyun.openservices.aliyun.log.producer.errors.ResultFailedException;
import com.aliyun.openservices.log.common.LogItem;
import com.google.common.util.concurrent.FutureCallback;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
public class SampleProducerWithFuture {
private static final Logger LOGGER = LoggerFactory.getLogger(SampleProducerWithFuture.class);
private static final ExecutorService threadPool = Executors
.newFixedThreadPool(Math.max(Runtime.getRuntime().availableProcessors(), 1));
public static void main(String[] args) throws InterruptedException {
final String project = "example-project";
final String logstore = "example-logstore";
String endpoint = "example-endpoint";
// In this example, the AccessKey ID and AccessKey secret are obtained from environment variables.
String accessKeyId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
String accessKeySecret = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
ProducerConfig producerConfig = new ProducerConfig();
final Producer producer = new LogProducer(producerConfig);
producer.putProjectConfig(new ProjectConfig(project, endpoint, accessKeyId, accessKeySecret));
int n = 100;
// The number of logs that have finished (either successfully send, or failed).
final AtomicLong completed = new AtomicLong(0);
for (int i = 0; i < n; ++i) {
List<LogItem> logItems = new ArrayList<LogItem>();
for (int j = 0; j < 10; ++j) {
//The maximum size of a LogItem (key) is 128 bytes. The maximum size of a LogItem (value) is 1 MB.
LogItem logItem = new LogItem();
logItem.PushBack("key1", "foo" + j);
logItem.PushBack("key2", "bar" + j);
logItems.add(logItem);
}
try {
ListenableFuture<Result> f = producer.send(project, logstore, logItems);
Futures.addCallback(
f, new SampleFutureCallback(project, logstore, logItems, completed), threadPool);
} catch (InterruptedException e) {
LOGGER.warn("The current thread has been interrupted during send logs.");
} catch (Exception e) {
LOGGER.error("Failed to send logs, e=", e);
}
}
try {
producer.close();
} catch (InterruptedException e) {
LOGGER.warn("The current thread has been interrupted from close.");
} catch (ProducerException e) {
LOGGER.info("Failed to close producer, e=", e);
}
threadPool.shutdown();
while (!threadPool.isTerminated()) {
threadPool.awaitTermination(100, TimeUnit.MILLISECONDS);
}
LOGGER.info("All log complete, completed={}", completed.get());
}
private static final class SampleFutureCallback implements FutureCallback<Result> {
private static final Logger LOGGER = LoggerFactory.getLogger(SampleFutureCallback.class);
private final String project;
private final String logStore;
private final List<LogItem> logItems;
private final AtomicLong completed;
SampleFutureCallback(
String project, String logStore, List<LogItem> logItems, AtomicLong completed) {
this.project = project;
this.logStore = logStore;
this.logItems = logItems;
this.completed = completed;
}
@Override
public void onSuccess(@Nullable Result result) {
LOGGER.info("Send logs successfully.");
completed.getAndIncrement();
}
@Override
public void onFailure(Throwable t) {
if (t instanceof ResultFailedException) {
Result result = ((ResultFailedException) t).getResult();
LOGGER.error(
"Failed to send logs, project={}, logStore={}, result={}", project, logStore, result);
} else {
LOGGER.error("Failed to send log, e=", t);
}
completed.getAndIncrement();
}
}
}
Expected result
{"__source__":"172.16.4.254","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658716454","__time__":"1658716450","__topic__":"","key1":"foo0","key2":"bar0"}
{"__source__":"172.16.4.254","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658716454","__time__":"1658716450","__topic__":"","key1":"foo1","key2":"bar1"}
{"__source__":"172.16.4.254","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658716454","__time__":"1658716450","__topic__":"","key1":"foo2","key2":"bar2"}
{"__source__":"172.16.4.254","__tag__:__client_ip__":"47.100.XX.XX","__tag__:__receive_time__":"1658716454","__time__":"1658716450","__topic__":"","key1":"foo3","key2":"bar3"}
......
For more information, see Aliyun Log Java Producer.
Simple Log Service provides sample applications that are based on Aliyun Log Java Producer to help you get started. For more information, see Sample applications that are based on Aliyun Log Java Producer.
FAQ
Is the number of data writes limited?
Yes, the numbers and sizes of data reads and writes to Simple Log Service are limited. For more information, see Data read and write.
Simple Log Service provides the following basic resources: projects, Logstores, shards, Logtail configurations, machine groups, LogItems, field names, and field values. Simple Log Service limits the numbers of projects, Logstores, shards, Logtail configurations, and machine groups, the size of a LogItem, and the lengths of the field name and field value for each LogItem. For more information, see Basic resources.
What do I do if no data is written to Simple Log Service?
If no data is written to Simple Log Service, perform the following steps to troubleshoot the issue:
Check whether the versions of the aliyun-log-producer, aliyun-log, and protobuf-java JAR packages that are used in your project are consistent with the versions that are listed in the "Install Aliyun Log Java Producer" section of this topic. If the versions are inconsistent, upgrade your versions.
The send method of Aliyun Log Java Producer is used to asynchronously send data. You cannot obtain the returned data at the earliest opportunity. In this case, identify the cause of the data sending failure based on a Callback object or a returned Future object.
If the onCompletion method is not used in the Callback operation, check whether the producer.close() method is called before your program exits. Data is asynchronously sent by backend threads. To ensure that the cached data in the memory is not lost, we recommend that you call the producer.close() method before your program exits.
Aliyun Log Java Producer returns key behavior during runtime by using the Simple Logging Facade for Java (SLF4J) log framework. You can configure a log framework in your program and enable DEBUG-level logging. Check whether ERROR logs are returned.
If the issue persists, submit a ticket.
References
- If the response that is returned by Log Service contains error information after you call an API operation, the call fails. You can handle errors based on the error codes that are returned when API calls fail. For more information, see Error codes.
- Alibaba Cloud OpenAPI Explorer provides debugging capabilities, SDKs, examples, and related documents. You can use OpenAPI Explorer to debug Log Service API operations without the need to manually encapsulate or sign requests. For more information, visit OpenAPI Portal.
- Log Service provides the command-line interface (CLI) to meet the requirements for automated configurations in Log Service. For more information, see Log Service CLI.
- For more information about sample code, see Alibaba Cloud Log Service SDK for Java on GitHub.