How to use a PackId identifier to associate a log with its context - Simple Log Service

To perform contextual query, you must specify a log source such as a log file on a server and a log whose context you want to query. The query operation returns the logs that are generated before or after the specified log and are collected from the log file. For more information, see Contextual query. When you process a large number of logs, you can add a PackId identifier to specific logs to categorize the logs into log groups. You can use a PackId identifier to query a log group in an efficient and complete manner. This helps identify the context of a log. This topic describes how to add a PackId identifier to a log.

How it works

Simple Log Service adds a PackId identifier to associate a log with its context. The PackId identifier is in the Contextual prefix-Log group ID format. Example: 5FA51423DDB54FDA-1E3.

Contextual prefix: A contextual prefix is a hexadecimal number. Example: 5FA51423DDB54FDA. Logs that have the same contextual prefix belong to the context of the same log.
Log group ID: A log group ID is a hexadecimal number. Example: 1E3. In the context of a log, the log group ID is incremental. For example, 1E3 and 1E4 indicate the adjacent log groups in the same log context.

The server that you use generates a PackId identifier and uploads logs and the PackId identifier to Simple Log Service at a time. Logs that have the same contextual prefix belong to the context of the same log.

Automatically generate a PackId identifier

Perform contextual query on logs that are written by using a Producer SDK: Logs that are written to Simple Log Service by the same producer instance have the same context and can be directly used in contextual query. For example, when you use Aliyun Log Java Producer or C Producer to write logs, the system automatically generates a PackId identifier and uploads logs and the PackId identifier to Simple Log Service. For more information, see Use Aliyun Log Java Producer to write log data to Simple Log Service or C Producer SDK.
Perform contextual query on logs that are collected by using Logtail: When you use Logtail to collect logs, the system automatically generates a PackId identifier and adds the PackId identifier to the collected logs. Logs in a log file on a host or a pod have the same context and can be directly used in contextual query. For more information, see Use Logtail to collect logs.

Manually generate a PackId identifier and call the PutLogs operation to upload the related logs

Parameters

You can call the PutLogs operation to upload logs to Simple Log Service. For more information, see PutLogs. After a PackId identifier is manually generated, you must set the Key parameter to __pack_id__ and the Value parameter to the PackId identifier in the LogTags attribute of the LogGroup data structure to call the PutLogs operation.

{
  "Topic": "my-topic",
  "Source": "127.0.0.1",
  "LogTags": [
      {
        "Key": "__pack_id__",
        "Value": "5FA51423DDB54FDA-1"
      },
      {
        "Key": "my_other_tag_key",
        "Value": "my_other_tag_value"
      }
   ],
   "Logs": [
     {
       "Time": 1728961415,
        "Contents": [
            {
              "Key": "hello",
              "Value": "world"
            }
        ]
     }
   ]
}

Sample code

Java

Add the following dependencies to the pom.xml file:

 <dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>27.0.1-jre</version>
</dependency>
<dependency>
    <groupId>com.aliyun.openservices</groupId>
    <artifactId>aliyun-log</artifactId>
    <version>0.6.111</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>2.0.12</version>
</dependency>

Use the following code to manually generate a PackId identifier and call the PutLogs operation to upload logs. Replace the values of the following parameters based on your business requirements: project, logstore, endpoint, accessKeyId, and accessKeySecret.

package org.example;

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.TagContent;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.request.PutLogsRequest;
import com.google.common.base.Charsets;
import com.google.common.hash.Hashing;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.lang.management.ManagementFactory;
import java.net.InetAddress;
import java.net.NetworkInterface;
import java.net.SocketException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Enumeration;
import java.util.List;
import java.util.concurrent.atomic.AtomicLong;

public class Main {
    private static final String TAG_PACK_ID = "__pack_id__";
    private static final int TOKEN_LEN = 4;

    public static void main(String[] args) throws LogException {
        System.out.println("Hello world!");
        // For the same context, specify the same value for the PackIdGenerator parameter.
        PackIdGenerator generator1 = new PackIdGenerator();
        System.out.println(generator1.generateNewPackId());
        System.out.println(generator1.generateNewPackId());
        System.out.println(generator1.generateNewPackId());
        // For different contexts, specify different values for the PackIdGenerator parameter.
        PackIdGenerator generator2 = new PackIdGenerator();
        System.out.println(generator2.generateNewPackId());
        System.out.println(generator2.generateNewPackId());

        // Configure log settings.
        String project = "project";
        String logstore = "logstore";
        String topic = "topic";
        String source = "127.0.0.1";
        Client client = new Client("endpoint", "accessKeyId", "accessKeySecret");
        List<LogItem> logs = new ArrayList<>();
        LogItem log = new LogItem();
        log.PushBack("hello", "world");
        logs.add(log);

        // Send a request to upload logs.
        PutLogsRequest req = new PutLogsRequest(project, logstore, topic, source, logs);
        // Include PackId identifiers in the Tags parameter.
        req.SetTags(Arrays.asList(new TagContent(TAG_PACK_ID, generator1.generateNewPackId())));
        client.PutLogs(req);

        // Send a request to upload logs.
        PutLogsRequest req2 = new PutLogsRequest(project, logstore, topic, source, logs);
        req.SetTags(Arrays.asList(new TagContent(TAG_PACK_ID, generator1.generateNewPackId())));
        client.PutLogs(req2);
    }

    public static class NetworkUtils {
        private NetworkUtils() {
        }

        public static boolean isIpAddress(final String ipAddress) {
            if (ipAddress == null || ipAddress.isEmpty()) {
                return false;
            }
            try {
                final String[] tokens = ipAddress.split("\\.");
                if (tokens.length != TOKEN_LEN) {
                    return false;
                }
                for (String token : tokens) {
                    int i = Integer.parseInt(token);
                    if (i < 0 || i > 255) {
                        return false;
                    }
                }
                return true;
            } catch (Exception ex) {
                return false;
            }
        }

        public static String getLocalMachineIp() {
            try {
                Enumeration<NetworkInterface> networkInterfaces = NetworkInterface.getNetworkInterfaces();
                while (networkInterfaces.hasMoreElements()) {
                    NetworkInterface ni = networkInterfaces.nextElement();
                    if (!ni.isUp()) {
                        continue;
                    }
                    Enumeration<InetAddress> addresses = ni.getInetAddresses();
                    while (addresses.hasMoreElements()) {
                        final InetAddress address = addresses.nextElement();
                        if (!address.isLinkLocalAddress() && address.getHostAddress() != null) {
                            String ipAddress = address.getHostAddress();
                            if ("127.0.0.1".equals(ipAddress)) {
                                continue;
                            }
                            if (isIpAddress(ipAddress)) {
                                return ipAddress;
                            }
                        }
                    }
                }
            } catch (SocketException ex) {
                // swallow it
            }
            return null;
        }
    }

    public static class PackIdGenerator {
        private static final Logger LOGGER = LoggerFactory.getLogger(PackIdGenerator.class);
        private static final AtomicLong GENERATORID = new AtomicLong(0);
        private final String packIdPrefix;
        private final AtomicLong batchId = new AtomicLong(0);

        public PackIdGenerator() {
            packIdPrefix = generatePackIdPrefix(GENERATORID.getAndIncrement()).toUpperCase() + "-";
        }

        public String generateNewPackId() {
            return packIdPrefix + Long.toHexString(batchId.getAndIncrement()).toUpperCase();
        }

        private String generatePackIdPrefix(Long instanceId) {
            String ip = NetworkUtils.getLocalMachineIp();
            if (ip == null) {
                LOGGER.warn("Failed to get local machine ip, set ip to 127.0.0.1");
                ip = "127.0.0.1";
            }
            String name = ManagementFactory.getRuntimeMXBean().getName();
            String input = ip + "-" + name + "-" + instanceId;
            return Hashing.farmHashFingerprint64().hashString(input, Charsets.US_ASCII).toString();
        }
    }

}

Parameters in the sample code

Parameter	Description	Example
project	The name of the Simple Log Service project.	test-project
logstore	The name of the Simple Log Service Logstore.	test-logstore
endpoint	The public Simple Log Service endpoint. For more information, see Endpoints.	cn-hangzhou.log.aliyuncs.com
accessKeyId	The AccessKey ID. For more information, see Create an AccessKey pair.	yourAccessKeyID
accessKeySecret	The AccessKey secret. For more information, see Create an AccessKey pair.	yourAccessKeySecret

Go

Run the following commands to install Simple Log Service SDK for Go and the protobuf dependency package:
```
go get -u github.com/aliyun/aliyun-log-go-sdk
go get google.golang.org/protobuf
```

package main

import (
	"crypto/md5"
	"fmt"
	"os"
	"sync/atomic"
	"time"

	sls "github.com/aliyun/aliyun-log-go-sdk"
	"google.golang.org/protobuf/proto"
)

func main() {
	// For the same context, specify the same value for the PackIdGenerator parameter.
	g1 := NewPackIdGenerator()
	fmt.Println(g1.Generate())
	fmt.Println(g1.Generate())
	fmt.Println(g1.Generate())
	// For different contexts, specify different values for the PackIdGenerator parameter.
	g2 := NewPackIdGenerator()
	fmt.Println(g2.Generate())
	fmt.Println(g2.Generate())

	// Configure log settings.
	project := "project"
	logstore := "logStore"
	topic := "topic"
	source := "source"

	client := sls.CreateNormalInterface("endpoint", "accessKeyId", "accessKeySecret", "")

	logs := []*sls.Log{
		{
			Time: proto.Uint32(uint32(time.Now().Unix())),
			Contents: []*sls.LogContent{
				{
					Key:   proto.String("hello"),
					Value: proto.String("world"),
				},
				{
					Key:   proto.String("hi"),
					Value: proto.String("world"),
				},
			},
		},
	}
	// Call the PostLogStoreLogsV2 operation to upload logs.
	err := client.PostLogStoreLogsV2(project, logstore, &sls.PostLogStoreLogsRequest{
		LogGroup: &sls.LogGroup{
			Topic:  &topic,
			Source: &source,
			Logs:   logs,
			LogTags: []*sls.LogTag{
				{
					Key:   proto.String("__pack_id__"), // Include PackId identifiers in the LogTags parameter.
					Value: proto.String(g1.Generate()),
				},
			},
		},
	})
	if err != nil {
		panic(err)
	}

	// Call the PutLogs operation to upload logs. Call the g1.Generate() operation to generate a new PackId identifier.
	err = client.PutLogs(project, logstore, &sls.LogGroup{
		Topic:  &topic,
		Source: &source,
		Logs:   logs,
		LogTags: []*sls.LogTag{
			{
				Key:   proto.String("__pack_id__"), // Include PackId identifiers in the LogTags parameter.
				Value: proto.String(g1.Generate()),
			},
		},
	})
	if err != nil {
		panic(err)
	}

}

type PackIdGenerator struct {
	prefix string
	id     atomic.Uint64
}

func NewPackIdGenerator() *PackIdGenerator {
	return &PackIdGenerator{
		prefix: generatePackIDPrefix(),
		id:     atomic.Uint64{},
	}
}

func (g *PackIdGenerator) Generate() string {
	return fmt.Sprintf("%s-%X", g.prefix, g.id.Add(1))
}

// make context by (hostname, pid, time)
func generatePackIDPrefix() string {
	m := md5.New()
	m.Write([]byte(time.Now().String()))
	hostName, _ := os.Hostname()
	m.Write([]byte(hostName))
	m.Write([]byte(fmt.Sprintf("%v", os.Getpid())))
	return fmt.Sprintf("%X", m.Sum(nil))
}

Parameters in the sample code

Parameter	Description	Example
project	The name of the Simple Log Service project.	test-project
logstore	The name of the Simple Log Service Logstore.	test-logstore
endpoint	The public Simple Log Service endpoint. For more information, see Endpoints.	cn-hangzhou.log.aliyuncs.com
accessKeyId	The AccessKey ID. For more information, see Create an AccessKey pair.	yourAccessKeyID
accessKeySecret	The AccessKey secret. For more information, see Create an AccessKey pair.	yourAccessKeySecret

View the context of a log

Access your project and click the Logstore that you want to manage. On the Raw Logs > Raw Data tab, find the log whose context you want to view and click the icon.
Note
If logs are uploaded to Simple Log Service by calling the PutLogs operation, logs that have the same contextual prefix belong to the context of the same log. For more information, see PutLogs. In Simple Log Service, logs are categorized by PackId identifier. After you click the Context View icon of a log, you can view the contextual logs of the log, including the logs that have the same contextual prefix and belong to different log groups. The log is highlighted. For more information, see Contextual query.
On the page that appears, scroll up and down to view the contextual logs.

References

You can query the context of a log in the Simple Log Service console. For more information, see Contextual query.
You can use SDKs or APIs to write logs to Simple Log Service. For more information, see Use Aliyun Log Java Producer to write log data to Simple Log Service, C Producer SDK, or PutLogs.