Ingest data and manage data collection - Simple Log Service

Select the correct write location

SessionStore type	Where application data is written	Where collection is managed
Direct write	The SessionStore that you created	The Overview, Data Ingestion, or Logtail configuration management page of the SessionStore.
Reference	The source Logstore that is associated during creation	The collection management page of the source Logstore.

If you need to specify a destination name for direct write ingestion, use the name of the current SessionStore and the project to which it belongs. Do not append a suffix or create another Logstore.

The reference mode does not have a separate Logtail ingestion entry for the SessionStore. This is a normal difference between the two modes and does not indicate a resource fault.

Direct write mode: Create a collection configuration

Prerequisites

A SessionStore in direct write mode is created.
The collected data matches the raw data format of the SessionStore.
If you use Logtail to collect server files, the collector is installed, the machine group is configured, the connectivity is checked, and the collection process can read the log files.

Procedure

Open the destination SessionStore.
Go to the creation entry in the Logtail configuration section on the Overview page, or click Data Ingestion on the Raw Data page. Then, create a Logtail configuration.
In the data ingestion catalog, select a collection method that matches your application and supports SessionStore.
Follow the collection wizard to configure the data source, processing method, and destination information. Make sure that the output destination is the current SessionStore.
Create the configuration and apply it to the corresponding machine group.
Generate a new log and check the result as described in the Verify data ingestion section of this topic.

Note: The ingestion catalog may contain multiple Simple Log Service collection methods. A card that appears in the catalog does not mean that the method supports the SessionStore data format that you select. Refer to the applicability description of the method and the actual wizard.

Check the collection configuration

The wizards vary based on the collection method. Refer to the page of the selected method. For file collection, check the following items before you submit the configuration.

Check item	What to check	Example
Configuration name	Easy to identify the application and environment, and meets the page requirements	`agent-json-collect`
Log directory	The directory on the machine where the collector resides, not the directory on your current computer	`/var/log/agent`
File matching	Actually covers the log files that are generated by the application	`*.json`
Encoding and parsing	Consistent with the actual encoding and content of the files	UTF-8 and JSON format.
Machine group	Contains the machines that actually generate the logs, and the machines are properly connected	Check the actual machine group.
Output destination	The current project and SessionStore	Do not change it to another resource.

JSON parsing only parses the log content and does not automatically add business fields such as the session ID and model inputs and outputs. Make sure that the file records conform to the selected raw data format.

The configuration fails to be created or applied

A successfully created configuration does not necessarily mean that it is successfully applied to all machine groups. If an error occurs, check the existing configurations and application status first, handle the failed items by using the recovery entry that is provided by the current wizard, and make sure that the collector actually receives the configuration.

Do not create a configuration with the same name repeatedly to retry the application to a machine group. After you close or refresh the page, check the existing results before you decide the next step.

Reference mode: Maintain the source Logstore

On the overview page of the SessionStore, check the source project and Logstore.
Open the collection management page of the source Logstore, and create or modify the original collection configuration.
Make sure that new data continuously enters the Logstore.
Return to the SessionStore, query the raw data, and then check the processed trajectories and sessions.

Important

The source Logstore may also be used by other business. Before you modify its collection rules, indexes, permissions, or retention period, evaluate the impact on other consumers.

View and maintain Logtail configurations

For a SessionStore in direct write mode, you can search for configuration names, turn pages, and refresh the configuration list. Click a configuration name or the management entry to go to the configuration details page. Then, modify the configuration and check the application status of machine groups based on the capabilities of the page.

Deleting a collection configuration and deleting a SessionStore are different operations. Before you delete a configuration, confirm its scope of use and machine group dependencies to avoid interrupting collection that is still required. Do not delete a collection configuration as a way to clear stored data.

Verify data ingestion

Record the occurrence time and identifier of a new test interaction.
On the Raw Data page, select a time range that covers the occurrence time, and use * to query data.
Check the actual fields and their values, especially the identifiers that associate executions with sessions.
Query the same interaction on the Single Trajectory and Sessions pages, and allow time for processing.

Symptom	What to check first
Raw data is empty	The write destination, whether new data exists at the source, the file path, the machine group, the collector, the write permissions, and the time range.
Raw data is visible, but trajectories are empty	The data format, the processing wait time, the backend service authorization, and the fields that are required by trajectories.
Trajectories are visible, but sessions are empty	Whether the session ID exists and is consistent, and whether the query time range covers the corresponding records.
Tool spans are visible, but no tool steps appear in the trajectory	Whether the source data contains the associated content that is required for processing. A single tool span is not necessarily sufficient to form a tool step that can be displayed.

Processing results may be generated later than the raw data. For OpenTelemetry Trace pipelines that use silence-based triggering, continuous activity and late-arriving data also affect snapshot generation.