Simple Log Service:[Recommended] Manage Logtail configurations with AliyunPipelineConfig

Prerequisites

• Logtail components v0.5.1 or later are installed. For more information, see Install Logtail components in an ACK cluster and Install Logtail components in a self-managed Kubernetes cluster.

• The target container is actively generating logs.

  Important

  Logtail collects only incremental logs. If a log file is not updated after configuration delivery, Logtail skips it. Read logs.

Procedure

vim cube.yaml

apiVersion: telemetry.alibabacloud.com/v1alpha1 # Fixed value. Do not modify.
kind: ClusterAliyunPipelineConfig               # Fixed value. Do not modify.
metadata:
  name: test-config                             # Specify a resource name. The name must be unique within the current Kubernetes cluster.
spec:
  project:                                      # Set the destination project name.
    name: k8s-your-project                      
  config:                                       # Define the iLogtail collection configuration.
    inputs:                                     # Specify input plug-ins.
      ...
    flushers:                                   # Specify output plug-ins.
      ...

kubectl apply -f cube.yaml

vim cube.yaml

kubectl apply -f cube.yaml

kubectl delete clusteraliyunpipelineconfigs <config_name>

CR parameters

• apiVersion

  The value is fixed as telemetry.alibabacloud.com/v1alpha1.

• kind

  The value is fixed as ClusterAliyunPipelineConfig.

• metadata

  Parameter	Data type	Required	Description
  name	string	Yes	The CR identifier, also used as the iLogtail collection configuration name. Note: Must be unique within the project and cannot be changed after creation. Requirements: It can contain only lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or digit. Length: 2 to 128 characters.

• spec

  Parameter	Data type	Required	Description
  project	PipelineProject	Yes	Destination project information. Note Cannot be changed after CR creation. To switch projects, create a new CR.
  config	PiplineConfig	Yes	iLogtail collection configuration details.
  logstores	PipelineLogstore	No	Logstores to create. Note All parameters except name take effect only when creating a Logstore. This list only creates Logstores; it does not set the destination Logstore. The destination is determined by the flusher plug-in in config. Leave empty if the destination Logstore already exists. You can add or delete Logstore definitions but cannot modify existing ones.
  machineGroups	PipelineMachineGroup	No	Machine groups to associate with the configuration. SLS auto-creates a default group named k8s-group-${your_k8s_cluster_id} when logtail-ds is installed. By default, the list contains only this machine group. Note alibaba-log-controller ensures only listed machine groups are associated. Unlisted groups are dissociated.
  enableUpgradeOverride	bool	No	Default value: false. When an AliyunLogConfig in the cluster points to the same iLogtail configuration as this AliyunPipelineConfig: true: alibaba-log-controller overrides the existing AliyunLogConfig-defined configuration. false: A conflict occurs and the AliyunPipelineConfig fails to apply. Note Definitions that point to the same iLogtail collection configuration: The source and target projects are the same. For AliyunLogConfig, the project is the cluster default project or spec.project. For AliyunPipelineConfig, the project is spec.project.name. Same iLogtail collection configuration name. In AliyunLogConfig, the parameter is spec.logtailConfig.configName. In AliyunPipelineConfig, the parameter is metadata.Name. Definition of an overwrite upgrade: First, AliyunPipelineConfig takes effect and updates the iLogtail collection configuration. Then, if AliyunPipelineConfig succeeds, the system deletes the AliyunLogConfig in the cluster that points to the same iLogtail collection configuration. Finally, the override upgrade completes.

  • PipelineProject

    Note

    Cannot be changed after CR creation. To switch projects, create a new CR.

    Parameter	Data type	Required	Description
    name	string	Yes	Destination project name. Note SLS auto-creates the project if it does not exist.
    description	string	No	Project description. Note Effective only during project creation.
    endpoint	string	No	The service endpoint of the destination project. For more information, see Endpoints. By default, the endpoint for the region where the current cluster resides is used. If the destination project and the cluster are in different regions, configure this parameter. Note This parameter controls only the region of the collection configuration. To deliver iLogtail data to this region, separately configure iLogtail's config_server_address and data_server_list. For more information, see Logtail configuration files and record files.
    uid	string	No	Destination account UID. Defaults to the current cluster's account. For cross-account delivery, set this parameter and the alibaba-log-controller environment variable: ALICLOUD_LOG_ACCOUNT_INFOS={"<uid>":{"accessKeyID":"<your_access_key_id>","accessKeySecret":"<your_access_key_secret>"}}. Note This controls only the project owner account. To deliver iLogtail data to this account, configure the iLogtail user identity separately. Configure a user identity.

  • PipelineConfig

    Parameters follow the CreateLogtailPipelineConfig API. configName uses `metadata.name` and cannot be set separately.

    Parameter	Data type	Required	Description
    sample	string	No	Log sample. Supports multiple logs, up to 1,500 bytes total.
    global	object	No	Global configuration. For more information, see Request parameters.
    inputs	object list	Yes	Input plug-in list, configured per Request parameters. Note Only one input plug-in is allowed.
    processors	object list	No	Processing plug-in list, configured per Request parameters.
    aggregators	object list	No	Aggregation plug-in list, configured per Request parameters.
    flushers	object list	Yes	Output plug-in list, configured per Request parameters. Note Only one flusher_sls plug-in is allowed.
    configTags	map	No	Custom tags for the iLogtail collection configuration.

  • PipelineLogstore

    Note

    All parameters except name are effective only during Logstore creation.

    Parameter	Data type	Required	Description
    name	string	Yes	Logstore name.
    queryMode	string	No	Destination Logstore type. For more information, see Core concepts. Default: standard. Valid values: query: Query Logstore standard: Standard Logstore
    ttl	int	No	Data retention period in days. For more information, see Manage a logstore. Default: 30. Valid values: 1 to 3650. 3650 indicates permanent retention.
    hotTtl	int	No	Retention period for hot data storage in days. Default: 0. Must be less than ttl and greater than or equal to 7.
    infrequentAccessTTL	int	No	The IA storage period in days. Default: 0. Requires hotTtl. Must be less than ttl and greater than or equal to 30. If hotTtl+infrequentAccessTTL is not equal to ttl, then ttl-(hotTtl+infrequentAccessTTL) must also be greater than or equal to 60. Requires loongcollector-operator v1.0.6 or later.
    shardCount	int	No	Shard count. For more information, see Manage a logstore. Default: 2. Valid values: 1 to 100.
    maxSplitShard	int	No	Maximum shard count for automatic sharding. For more information, see Manage a logstore. Default: 64. Valid values: 1 to 256.
    autoSplit	bool	No	Enables automatic shard splitting. For more information, see Manage a logstore. Default: true.
    telemetryType	string	No	Observable data type. Details: Create a Logstore. Default: None. Valid values: None: log type. Metrics: metric type.
    appendMeta	bool	No	Enables public IP address recording. For more information, see Manage a logstore. Default: true.
    enableTracking	bool	No	Enables Web Tracking. For more information, see Manage a logstore. Default: false.
    encryptConf	object	No	Encryption configuration. For more information, see EncryptConf. Default: empty.
    meteringMode	string	No	Billing mode. For more information, see Manage a logstore. Default: empty. Valid values: ChargeByFunction: pay-by-function ChargeByDataIngest: pay-by-ingested-data Note If the Logstore's queryMode is query, only ChargeByFunction is supported. If pay-by-ingested-data billing is not enabled for your account, you cannot set this parameter to ChargeByDataIngest.
    index	object	No	Index configuration, effective only during Logstore creation. Format: index. Requires loongcollector-operator v1.0.6 or later.

  • PipelineMachineGroup

    Parameter	Data type	Required	Description
    name	string	Yes	Machine group name to associate with the configuration. Note If the machine group does not exist, alibaba-log-controller auto-creates an identifier-based machine group. For more information, see Create a custom identifier-based machine group. Effective only during creation.