Use Aliyun Log Go Producer to write logs to Simple Log Service -

If you want to compress logs and upload compressed logs to Simple Log Service to reduce the usage of network transmission resources when you use big data computing engines such as Flink, Spark, and Storm, Simple Log Service API and Simple Log Service SDK are not recommended because the API and SDK cannot meet the requirements for data writing in big data scenarios. In this case, you can use Aliyun Log Go Producer to upload logs to Simple Log Service in an efficient manner.

Prerequisites

A Resource Access Management (RAM) user is created, and the required permissions are granted to the RAM user. For more information, see Create a RAM user and grant permissions to the RAM user.
The ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables are configured. For more information, see Configure environment variables in Linux, macOS, and Windows.
Important
- The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. We recommend that you use the AccessKey pair of a RAM user to call API operations or perform routine O&M.
- We recommend that you do not save the AccessKey ID or AccessKey secret in your project code. Otherwise, the AccessKey pair may be leaked, and the security of all resources within your account may be compromised.
Simple Log Service SDK for Go is installed. For more information, see Install Simple Log Service SDK for Go.
The proto dependency is installed. You can run the go get -u github.com/gogo/protobuf/proto command to install the proto dependency.

Usage notes

In this example, the public Simple Log Service endpoint for the China (Hangzhou) region is used, which is cn-hangzhou.log.aliyuncs.com.

If you want to access Simple Log Service by using other Alibaba Cloud services that reside in the same region as your project, you can use the internal Simple Log Service endpoint, which is cn-hangzhou-intranet.log.aliyuncs.com.

For more information about the mappings between the endpoints and regions that are supported by Simple Log Service, see Endpoints.

Introduction to Aliyun Log Go Producer

Aliyun Log Go Producer is a high-performance class library provided for Go applications that run in big data and high concurrency scenarios. Compared with Simple Log Service API and Simple Log Service SDK, Aliyun Log Go Producer provides multiple benefits that are related to log writing, such as high performance, logic isolation between computing and I/O, and resource management. Aliyun Log Go Producer uses the sequential writing feature provided by Simple Log Service to ensure the upload sequence of logs.

The following figure shows the workflow of Aliyun Log Go Producer:

Limits

The underlying mechanism of Aliyun Log Go Producer calls the PutLogs operation to upload logs. The size of raw logs that can be uploaded each time is limited. For more information, see Data read and write.
The basic resources of Simple Log Service, such as projects, logstores, shards, and machine groups, also have limitations. For more information, see Basic resources.
The first time you run code, you must enable the indexing feature for your logstore in the Simple Log Service console. Then, wait for about one minute before querying logs.
If you query logs in the Simple Log Service console and the value length of a field in the returned logs exceeds the upper limit, the field value will be truncated, and the excess part will not be used for analysis. For more information, see Create indexes.

1. Configure the ProducerConfig data structure

producerConfig := producer.GetDefaultProducerConfig()
producerConfig.Endpoint = "cn-hangzhou.log.aliyuncs.com"
provider := sls.NewStaticCredentialsProvider(os.Getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"), os.Getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"), "")
producerConfig.CredentialsProvider = provider

You can use the ProducerConfig data structure to configure a sending policy. You can configure parameters for a sending policy based on the scenario. The following table describes the parameters.

Parameter	Type	Description
TotalSizeLnBytes	Int64	The maximum size of logs that can be cached by a producer instance. Default value: 100 MB.
MaxIoWorkerCount	Int64	The maximum number of goroutines that can be concurrently run by a producer instance. Default: 50. You can configure this parameter based on the performance of your server.
MaxBlockSec	Int	The maximum blocking time when the send method is called but the available space of a producer instance is insufficient. Default value: 60 seconds. If the maximum blocking time that you specify elapses and the available space of the producer instance is still insufficient, the send method throws the TimeoutException error. If you set this parameter to 0 and the available space of the producer instance is insufficient, the send method immediately throws the TimeoutException error. If you want to block the send method until the available space of the producer instance is sufficient, you must set this parameter to a negative value.
MaxBatchSize	Int64	The maximum size of logs that can be cached in a ProducerBatch collection. If the size of logs in a ProducerBatch collection is greater than or equal to the value of this parameter, the logs are delivered to Simple Log Service. Default value: 512 KB. Maximum value: 5 MB.
MaxBatchCount	Int	The maximum number of logs that can be cached in a ProducerBatch collection. If the number of logs in a ProducerBatch collection is greater than or equal to the value of this parameter, the logs are delivered to Simple Log Service. Default value: 4096. Maximum value: 40960.
LingerMs	Int64	The duration from the time when a ProducerBatch collection is created to the time when the logs in the ProducerBatch collection are delivered to Simple Log Service. Default value: 2 seconds. Minimum value: 100 milliseconds.
Retries	Int	The number of times that the logs in a ProducerBatch collection can be re-delivered to Simple Log Service after the first-time sending failure. Default value: 10. If the value of this parameter is less than or equal to 0, logs in a ProducerBatch collection enter the failed queue after the first-time sending failure.
MaxReservedAttempts	Int	The maximum number of attempts that can be returned. The number of attempts to deliver the logs of a ProducerBatch collection is recorded. By default, only the latest 11 attempts are returned. A larger value allows you to trace more details. However, more memory resources are consumed.
BaseRetryBackoffMs	Int64	The backoff time for the first retry. Default value: 100 milliseconds. A producer instance uses the exponential backoff algorithm. The backoff time for the Nth retry is calculated based on the following formula: Value of the BaseRetryBackoffMs parameter × 2^(N - 1).
MaxRetryBackoffMs	Int64	The maximum backoff time for a retry. Default value: 50 seconds.
AdjustShardHash	Bool	Specifies whether to modify the shardHash parameter if you configure the shardHash parameter when you call the send method. Default value: true.
Buckets	Int	The number of buckets into which data is grouped. This parameter takes effect only when you set the AdjustShardHash parameter to true. If this parameter takes effect, your producer instance automatically groups data into buckets based on the value of the shardHash parameter. If two data entries have different values for the shardHash parameter, the data entries cannot be grouped into the same bucket for data upload. This reduces the throughput of the producer instance. After the producer instance groups data, the system is more likely to upload the grouped data to Simple Log Service in batches. Valid values: [1, 256]. The value of this parameter must be an integer power of 2. Default value: 64.
CredentialsProvider	Interface	The interface that you can call to dynamically obtain an AccessKey ID, AccessKey secret, and Security Token Service (STS) token. You can configure this parameter based on your business requirements. This interface is expected to cache the returned AccessKey pair and ensure thread security.
NoRetryStatusCodeList	[ ]int	The error codes returned for requests that do not need retries. If a log fails to be uploaded to Simple Log Service and a specified error code is returned for the request, the request is not retried. Default values: 400 and 404.
UpdateStsToken	Func	The function that allows Aliyun Log Go Producer to automatically update STS tokens and add the tokens to your producer instance.
StsTokenShutDown	channel	The communication channel that allows you to disable automatic update of STS tokens. If you disable the communication channel, STS tokens are not automatically updated. If your producer instance is shut down and the value of this parameter is not `nil`, the producer instance calls the close method to disable the communication channel. This way, the automatic update is disabled.
Region	String	The region where Simple Log Service is deployed. This parameter is required if you set the AuthVersion parameter to AuthV4. Example: cn-hangzhou.
AuthVersion	String	The version of signing. Valid values: AuthV1 and AuthV4. For more information about the examples of AuthV4 signing, see producer_test.go.
UseMetricStoreURL	bool	The URL of the Metricstore that you want to use. You can use the URL to deliver logs to Simple Log Service. This improves the query performance in scenarios in which a large amount of time series exist.

2. Start a producer instance

When you call the producerInstance.Start() function, a goroutine is started to monitor whether logs are written to your producer instance and whether log groups that match the sending conditions exist. If yes, the goroutine delivers the matched log groups to the Simple Log Service Logstore that you specify.

producerInstance, err := producer.NewProducer(producerConfig)
if err != nil {
  panic(err)
}
ch := make(chan os.Signal)
signal.Notify(ch, os.Kill, os.Interrupt)
producerInstance.Start()

3. Call a send method to deliver logs to Simple Log Service

In this example, the HashSendLogListWithCallBack method is called to deliver logs. The following table describes the related parameters.

Parameter	Required	Description
project	Yes	The project.
logStore	Yes	The Logstore.
shardHash	No	The hash value for the logs you want to deliver. You can specify a hash value based on your business requirements. After you specify a hash value, logs are written to a specific shard in the specified Logstore based on the hash value. Note If you do not configure this parameter, logs are randomly written to a shard in the specified Logstore.
topic	No	The topic of logs. Note If you do not configure this parameter, double quotation marks ("") are automatically used.
source	No	The source of logs. Note If you do not configure this parameter, the IP address of the host where the producer instance resides is automatically used.
logs	Yes	The logs that you want to deliver to the Logstore.
callback	No	The callback function. This function is called when logs are delivered to Simple Log Service or when logs fail to be delivered after multiple retries and are discarded.

Note

You can call the producer.GenerateLog() method to create a log delivered to a Logstore in a simple manner. However, the generation process is inefficient. We recommend that you call the sls. Log operation to create a log. Example:

	var m sync.WaitGroup
	callBack := &Callback{}
	logs := []*sls.Log{}
	content := []*sls.LogContent{}
	for colIdx := 0; colIdx < 10; colIdx++ {
		if colIdx/2 == 0 {
			content = append(content, &sls.LogContent{
				Key:   tea.String("request_method"),
				Value: tea.String("GET"),
			}, &sls.LogContent{
				Key:   tea.String("status"),
				Value: tea.String("200"),
			})
		} else if colIdx/3 == 0 {
			content = append(content, &sls.LogContent{
				Key:   tea.String("request_method"),
				Value: tea.String("POST"),
			}, &sls.LogContent{
				Key:   tea.String("status"),
				Value: tea.String("500"),
			})
		} else {
			content = append(content, &sls.LogContent{
				Key:   tea.String("request_method"),
				Value: tea.String("POST"),
			}, &sls.LogContent{
				Key:   tea.String("status"),
				Value: tea.String("200"),
			})
		}
	}
	log := &sls.Log{
		Time:     proto.Uint32(uint32(time.Now().Unix())),
		Contents: content,
	}
	logs = append(logs, log)
	shardHash := ""
	for i := 0; i < 10; i++ {
		m.Add(1)
		go func() {
			defer m.Done()
			for i := 0; i < 1000; i++ {
				// GenerateLog  is producer's function for generating SLS format logs
				// GenerateLog has low performance, and native Log interface is the best choice for high performance.
				err := producerInstance.HashSendLogListWithCallBack("gs-log-test", "gstest", shardHash, "topic", "127.0.X.1", logs, callBack)
				if err != nil {
					fmt.Println(err)
				}
			}
		}()
	}
	fmt.Println("Send completion")

4. Shut down a producer instance

Two shutdown modes are supported: limited shutdown and safe shutdown.

Safe shutdown: In this mode, a producer instance is shut down after all cached data in the producer instance is delivered to Simple Log Service.
Limited shutdown: In this mode, a producer instance receives a parameter value that is given in seconds. After the producer instance receives the parameter value, a countdown starts. If the specified value reaches and the producer instance is not completely shut down, the producer instance is forced to quit. In this case, some data may not be delivered and then lost.

producerInstance.Close(60) // The limited shutdown. You must specify a positive integer for this parameter. Unit: seconds.
producerInstance.SafeClose()// The safe shutdown.

References

If the response that is returned by Simple Log Service contains error information after you call an API operation, the call fails. You can fix the error based on the error code description of the related API operation. For more information, see Error codes.
For more information about sample code, see Aliyun Log Go Producer on Github.