AliyunPipelineConfig parameters - Simple Log Service - Alibaba Cloud Documentation Center

To create a log collection configuration using a ClusterAliyunPipelineConfig, you must define the collection rules in a structured YAML file. This topic describes the structure of the YAML file and its fields.

Important

The Custom Resource (CR) is the source of truth for collection configurations created using a CustomResourceDefinition (CRD). Do not modify these configurations in the Simple Log Service console. Any changes made in the console will be overwritten by the CR, which can cause data format errors or data loss.

How it works

Create a CR: Define collection rules in a ClusterAliyunPipelineConfig YAML file and then submit the file using kubectl.
Controller monitors changes: The loongcollector-operator continuously monitors CRs in the cluster for changes.
Sync configuration: When a change to a CR is detected, the operator converts the CR into a specific configuration and submits it to the specified project.
Collector pulls the latest configuration: The loongcollector-ds periodically sends heartbeats to Simple Log Service to pull the latest collection configuration and then hot-reloads it.
Start collection and reporting: The loongcollector-ds collects logs based on the latest configuration and sends the logs to SLS using the configured endpoint.

Basic fields

All configurations must start by defining the API version and resource type as follows:

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig

Structure example

apiVersion: telemetry.alibabacloud.com/v1alpha1 # Use the default value. Do not modify.
kind: ClusterAliyunPipelineConfig               # Use the default value. Do not modify.
metadata:
  name: test-config                             # Set the resource name. It must be unique within the current Kubernetes cluster.
spec:
  project:                                      # Set the name of the destination project.
    name: k8s-your-project                      
  config:                                       # Set the Logtail collection configuration.
    inputs:                                     # Set the input plugins for the Logtail collection configuration.
      ...
    flushers:                                   # Set the output plugins for the Logtail collection configuration.
      ...

Core parameter descriptions

metadata.name

The name of the collection configuration. This parameter is required. The name must be unique within the project and cannot be changed after creation.

Naming convention:

Can contain only lowercase letters, digits, hyphens (-), and underscores (_).
Must start and end with a lowercase letter or a digit.
Must be 2 to 128 characters long.

Example:

metadata:
  name: nginx-access-log

spec.project

Specifies the target project.

Note

The project field cannot be changed after the CR is created. To switch to a different project, you must create a new CR.

Parameter	Data type	Required	Description
name	string	Yes	The name of the destination project. If the project does not exist, it is automatically created.
description	string	No	The description of the project. This parameter takes effect only when the project is created.
endpoint	string	No	The endpoint of the region where the project is located. The default value is the region of the cluster. This parameter controls only the region where the collection configuration is created. It does not control the data shipping destination of the data collector. For cross-region collection, configure config_server_address and data_server_list.
uid	string	No	The UID of the Alibaba Cloud account to which the destination project belongs. The default value is the UID of the Alibaba Cloud account to which the current cluster belongs. This parameter controls only the account under which the collection configuration is created. To collect logs across accounts, configure the environment variables for the `alibaba-log-controller` component: `ALICLOUD_LOG_ACCOUNT_INFOS={"<uid>":{"accessKeyID":"<your_access_key_id>","accessKeySecret":"<your_access_key_secret>"}}`.

spec.config

Specifies the main body of the collection configuration, which defines the input, processing, and output plugins.

Sub-field	Type	Required	Description
`sample`	string	No	A log sample. You can provide multiple logs. The total length cannot exceed 1,500 bytes.
`global`	object	No	Global configuration.
`inputs`	List of objects	Yes	A list of input plugins. Currently, only one input plugin is allowed.
`processors`	List of objects	No	A list of processing plugins: Native plugins Extension plugins
`flushers`	List of objects	Yes	A list of output plugins. Currently, only one `flusher_sls` plugin is allowed.

spec.logstores

Optional. Declares the Logstores that you want to create automatically. Note the following:

Effective only on creation: All parameters except for name apply only when a Logstore is created. If the Logstore already exists, the system ignores these settings and does not modify the existing properties of the Logstore.
Does not determine the data destination: This list does not determine the destination Logstore for the logs. The actual destination is determined by the output plugin in config.flushers, such as flusher_sls.
Optional configuration: If the destination Logstore already exists, you do not need to include it in this configuration.
Supports only add and delete operations: You can add new Logstores to the list or remove items from it. However, you cannot modify this configuration to change the properties of an existing Logstore, such as its TTL or shard count. To modify these properties, you must use the console or an API operation.

Parameter	Data type	Required	Description
name	string	Yes	The name of the Logstore to create.
queryMode	string	No	Logstore specifications. The default value is `standard`. Valid values: `query`: Query Logstore. `standard`: Standard Logstore.
ttl	int	No	The data retention period in days. Valid values: 1 to 3650. The default value is 30. A value of 3650 indicates permanent storage.
hotTtl	int	No	The hot data storage period in days. The default value is 0. The value must be less than ttl and greater than or equal to 7.
infrequentAccessTTL	int	No	The infrequent access storage period for the target Logstore, in days. The default value is 0. `hotTtl` must be specified, and the value of this parameter must be less than `ttl` and greater than or equal to 30. If `hotTtl` + `infrequentAccessTTL` is not equal to `ttl`, the value of `ttl-(hotTtl+infrequentAccessTTL)` must be greater than or equal to 60. This parameter is supported only when the version of the `loongcollector-operator` component is 1.0.6 or later.
shardCount	int	No	The number of shards. The default value is 2. Valid values: 1 to 100.
maxSplitShard	int	No	The maximum number of shards that can be automatically split. The default value is 64. Valid values: 1 to 256.
autoSplit	bool	No	Specifies whether to enable automatic sharding. The default value is true.
telemetryType	string	No	The type of observable data. The default value is None. Valid values: `None`: Log data. `Metrics`: Time series data.
appendMeta	bool	No	Specifies whether to record the public IP address and the log reception time. The default value is true. `true`: Enables the feature to record the public IP address and log reception time. When enabled, Simple Log Service automatically adds the public IP address of the source device and the time the log arrives at the server to the log's tag field. `false`: Disables the feature to record the public IP address and log reception time.
enableTracking	bool	No	Specifies whether to enable the web tracking feature. The default value is false.
encryptConf	object	No	The encryption configuration data structure. It contains the `enable`, `encrypt_type`, and `user_cmk_info` parameters. This parameter is empty by default.
meteringMode	string	No	Billing mode. For more information, see Manage a Logstore. This field is empty by default. Valid values: `ChargeByFunction`: Pay-by-feature `ChargeByDataIngest`: Pay-by-ingested-data Note A Logstore with `queryMode` set to `query` supports only the pay-by-feature billing mode. If pay-by-ingested-data is not enabled for your account, you cannot set this parameter to ChargeByDataIngest.
index	object	No	The index configuration. This parameter applies only when the Logstore is created. For the format, see the general data format for index. This parameter is available only when the loongcollector-operator component version is 1.0.6 or later.

spec.machineGroups

Specifies the machine groups that can use this collection configuration.

Default behavior: When LoongCollector is installed, the system automatically creates a machine group named k8s-group-${clusterId}. If you do not explicitly specify the machineGroups parameter, this collection configuration is associated with that machine group by default.
Synchronization mechanism: The loongcollector-operator ensures that the machine groups associated with the collection configuration strictly match the list defined in machineGroups. The system automatically disassociates any machine group that is not in this list.
Automatic creation support: If a specified machine group does not exist, the system automatically creates an identifier-based machine group with the same name and attaches it to the current collection configuration.

Parameter	Data type	Required	Description
name	string	No	The name of the machine group to associate.

spec.enableUpgradeOverride

Optional. Specifies whether the system can overwrite previous configurations. You can use this parameter to resolve conflicts between different versions of collection configurations. The default value is false.

true: The loongcollector-operator overwrites and upgrades the existing collection configuration defined by AliyunLogConfig.
false: If a conflict exists in the collection configuration, applying the AliyunPipelineConfig fails.

Scenario: A conflict occurs if an AliyunLogConfig resource that defines a collection configuration already exists in the cluster, and you attempt to apply an AliyunPipelineConfig that references the same collection configuration.

Definition of the same collection configuration:

Same project
- AliyunLogConfig: Uses the cluster's default project or spec.project.
- AliyunPipelineConfig: Uses spec.project.name.
Same collection configuration name
- AliyunLogConfig: spec.logtailConfig.configName
- AliyunPipelineConfig: metadata.name

Upgrade process:

Apply the new configuration: Applying the ClusterAliyunPipelineConfig updates the collection configuration.
Clean up the previous configuration: If the update is successful, the controller automatically deletes the corresponding AliyunLogConfig resource from the cluster.
Complete the migration: This completes the seamless transition from the old method to the new one.