All Products
Search

This Product

    :Ingest logs from cloud services

    Document Center

    :Ingest logs from cloud services

    Last Updated:Apr 15, 2026

    This enables centralized monitoring and analysis of your alerts and log data across resources. After logs are ingested, Agentic SOC monitors and analyzes the data to identify and construct complete attack chains, generating detailed security events to improve your alert analysis and response efficiency.

    Prerequisites

    Ingest logs from Alibaba Cloud services

    • To ingest logs from a cloud service within your current Alibaba Cloud account, directly select the cloud service and log type on the Service Integration page.

    • To centrally configure log ingestion policies for multiple Alibaba Cloud accounts, first complete the multi-account management setup. Then, log on to the console as a global administrator, switch to the Global Account View on the Service Integration page, and follow the procedure below. For more information about multi-account management, see Multi-account management.

    1. In the left-side navigation pane, choose Agentic SOC > Manage > Integration Settings. In the upper-left corner of the console, select a region where your assets are located: Chinese Mainland or Outside Chinese Mainland.

    2. In the service integration list, find the desired cloud service and click Ingestion Settings in the Actions column.

    3. In the service ingestion settings panel, find the desired log type and click the number in the Associated Accounts column.

      To configure multiple accounts at once, select multiple log types and click the batch integration button at the bottom of the list.

    4. In the ingestion settings panel, select the accounts to integrate.

      Note

      If you are using an individually verified account, the Select Account panel only displays your current account. Only a global administrator can select from all accounts managed by Agentic SOC while in the Global Account View.

      • If a cloud service like Security Center supports only a service-defined Logstore, you only need to select the account. A Logstore selection is not required. Security Center automatically ingests logs into the service-defined Logstore.

      • If the cloud service uses a custom Logstore, you must select the account and then choose the corresponding Logstore from the Logstore (format: regionId.project.logStore) drop-down list or paste the custom Logstore name. The Logstore name must be in the format regionId.project.logStore.

    5. Enable Automatically Associate New Accounts if required.

      If you enable Automatically Associate New Accounts, Agentic SOC automatically ingests the corresponding logs whenever a new Alibaba Cloud account is added.

      Note

      Only a global administrator can configure Automatically Associate New Accounts while in the Global Account View.

    Ingest logs from third-party cloud services

    If your business operates on both Alibaba Cloud and third-party cloud platforms (currently supporting Huawei Cloud and Tencent Cloud), and you need to manage security alerts across these environments, you can connect your third-party cloud accounts to Agentic SOC. This enables centralized alert monitoring and security operations management.

    1. Configure a third-party account

    Huawei Cloud sub-account

    1. Create two custom policies: siemBasePolicy and siemNormalPolicy. For more information, see the Huawei Cloud documentation on Create a custom policy.

      Note

      When creating a custom policy in Huawei Cloud, you cannot select global-level and project-level cloud services simultaneously. Therefore, you must create two separate policies for least-privilege authorization.

      • siemBasePolicy: corresponds to permissions for global-level cloud services. The policy content is:

        {
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:roles:listRoles",
                "iam:roles:getRole",
                "iam:groups:listGroupsForUser",
                "iam:groups:listGroups",
                "iam:users:getUser",
                "iam:groups:getGroup"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "rms:resources:list",
                "rms:resources:summarize"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "obs:object:GetObject",
                "obs:bucket:GetBucketLocation",
                "obs:bucket:HeadBucket",
                "obs:object:GetObjectVersionAcl",
                "obs:bucket:ListAllMyBuckets",
                "obs:bucket:ListBucket",
                "obs:object:GetObjectVersion",
                "obs:object:GetObjectAcl"
            ]
        }
    ]
}

      • siemNormalPolicy: corresponds to permissions for project-level cloud services. The policy content is:

        {
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cfw:ipGroup:list",
                "cfw:acl:list",
                "cfw:ipMember:put",
                "cfw:ipMember:create",
                "cfw:ipGroup:create",
                "cfw:instance:get",
                "cfw:ipGroup:put",
                "cfw:ipMember:list",
                "cfw:ipGroup:get",
                "cfw:ipMember:delete"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "waf:whiteBlackIpRule:list",
                "waf:whiteBlackIpRule:put",
                "waf:ipgroup:get",
                "waf:whiteBlackIpRule:get",
                "waf:ipgroup:list",
                "waf:whiteBlackIpRule:create",
                "waf:whiteBlackIpRule:delete"
            ]
        }
    ]
}

    2. Create two user groups, siemUser and readonlyuser, and grant them the required permissions as shown in the table below. For more information, see the Huawei Cloud documentation on creating a user group and granting permissions.

      User group

      Required permissions

      siemUser

      Custom policy permissions: siemBasePolicy, siemNormalPolicy

      readonlyuser

      • LTS ReadOnlyAccess: Provides read-only permissions for Log Tank Service (LTS).

      • OBS OperateAccess: Provides basic operational permissions for Object Storage Service (OBS), such as viewing the bucket list, getting bucket metadata, listing objects, querying bucket locations, uploading objects, getting objects, deleting objects, and getting object ACLs.

      • OBS ReadOnlyAccess: Provides read-only permissions for Object Storage Service (OBS), including viewing the bucket list, getting bucket metadata, listing objects, and querying bucket locations. No other permissions are granted.

      • CFW ReadOnlyAccess: Provides read-only permissions for Cloud Firewall.

      • WAF ReadOnlyAccess: Provides read-only permissions for Web Application Firewall (WAF).

    3. Create an IAM user and add it to the siemUser user group. For more information, see the Huawei Cloud documentation on creating an IAM user.

    4. Create an AccessKey pair for the IAM user. For more information, see the Huawei Cloud documentation on creating an IAM user access key.

    Tencent Cloud sub-account

    1. Create a custom policy named siemPolicy using policy syntax.

      The policy content for siemPolicy is as follows:

      {
    "statement": [
        {
            "action": [
                "cfw:DescribeAclApiDispatch",
                "cfw:DescribeBorderACLList",
                "cfw:CreateAcRules"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "waf:DescribeDomains",
                "waf:DescribeIpAccessControl",
                "waf:DeleteIpAccessControl",
                "waf:UpsertIpAccessControl",
                "waf:PostAttackDownloadTask"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "ckafka:DescribeDatahubGroupOffsets",
                "ckafka:DescribeGroup",
                "ckafka:DescribeGroupInfo",
                "ckafka:DescribeGroupOffsets",
                "ckafka:CreateDatahubGroup",
                "ckafka:ModifyDatahubGroupOffsets",
                "ckafka:ListConsumerGroup"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        },
        {
            "action": [
                "cam:GetUser",
                "cam:CheckSubAccountName",
                "cam:CheckUserPolicyAttachment",
                "cam:GetAccountSummary",
                "cam:GetPolicy",
                "cam:GetPolicyVersion",
                "cam:ListAllGroupsPolicies",
                "cam:ListAttachedGroupPolicies",
                "cam:ListAttachedRolePolicies",
                "cam:ListAttachedUserAllPolicies",
                "cam:ListAttachedUserPolicies",
                "cam:ListGroupsPolicies",
                "cam:ListPolicies",
                "cam:ListUsers"
            ],
            "effect": "allow",
            "resource": [
                "*"
            ]
        }
    ],
    "version": "2.0"
}

    2. Create a sub-account. For more information, see the Tencent Cloud documentation on creating a sub-account.

    3. Attach the siemPolicy policy to the created sub-account. For more information, see the Tencent Cloud documentation on authorization management.

    4. Create an AccessKey pair for the sub-account. For more information, see the Tencent Cloud documentation on sub-account access key management.

    2. Forward logs to a cloud service

    To enable threat analysis and response, you must forward logs from your cloud services to the vendor's storage or messaging services, such as Object Storage Service (OBS) or CKafka. Agentic SOC can then retrieve and analyze these logs efficiently.

    Cloud provider

    Cloud service log

    Destination service

    Forwarding configuration

    Data collection latency

    Huawei Cloud

    • Cloud Firewall alert log

    • Web Application Firewall (WAF) alert log

    OBS

    You must forward logs stored in Log Tank Service (LTS) to Object Storage Service (OBS). For more information, see the Huawei Cloud documentation on forwarding logs to OBS.

    Configure the key parameters as follows:

    • Custom forwarding path: Enable this option and configure the path with minute-level granularity. For example: /LogTanks/cn-north-4/CFW/lts-topic-cfw-0001//%Y/%m/%d/%H/%M.

    • Forwarding period: Set to 2 minutes.

    Important

    • You must configure these parameters exactly as described. Agentic SOC collects data from OBS based on the file directory structure. If the collection frequency and directory structure do not match, Agentic SOC may pull duplicate data.

    • Agentic SOC cannot ingest data from encrypted buckets, so do not forward logs to one.

    Data collected from OBS is offline, which introduces some latency. The system is designed so that data collection lags behind the current time by three specified collection intervals.

    For example, if the collection period is set to 2 minutes and a collection task starts at 17:58 on September 10, 2024, the system retrieves data from the /2024/09/10/17/52 directory. This data is from 6 minutes ago, which is three intervals earlier.

    This design ensures data integrity. The three-interval delay helps prevent incomplete data or data loss from unfinished write operations, especially in high-volume scenarios.

    Tencent Cloud

    Cloud Firewall alert log (only intrusion prevention logs are supported)

    CKafka

    You must ship intrusion prevention logs to a CKafka topic. For more information, see the Tencent Cloud documentation on log shipping.

    Real-time collection with no latency.

    Web Application Firewall (WAF) alert log

    None

    Agentic SOC collects these logs by calling the WAF API every 10 minutes. No forwarding configuration is required.

    Data collection has a latency of 10 minutes or more.

    3. Add a third-party AccessKey

    Add your third-party cloud account's AccessKey pair to Agentic SOC to allow it to access alert logs from your third-party cloud assets.

    1. Log on to the Security Center console.

    2. Authorize the third-party cloud provider account.

      Security Center uses the AccessKey pair of the third-party account to obtain read permissions and synchronize asset information from the third-party cloud.

      1. In the left-side navigation pane, choose Agentic SOC > Manage > Integration Settings. In the upper-left corner of the console, select a region where your assets are located: Chinese Mainland or Outside Chinese Mainland.

      2. In the Multi-cloud Service Access section, hover over the icon of the desired third-party cloud provider and click Grant Permission.

      3. In the Edit Multi-cloud Configuration panel, select Manual Configuration. In the Permission Description section, select Agentic SOC and click Next.

      4. On the Submit AccessKey Pair page, enter the AccessKey pair information for the sub-account and click Next.

      5. On the Policy Configuration page, select a period for the AK Service Status Check and click OK.

    3. Add the third-party cloud provider account.

      1. In the left-side navigation pane, choose Agentic SOC > Manage > Integration Settings. In the upper-left corner of the console, select a region where your assets are located: Chinese Mainland or Outside Chinese Mainland.

      2. In the Multi-cloud Service Access section, hover over the icon of the third-party cloud provider you want to add and click Add Account.

      3. In the Add Account panel, click Add.

      4. In the Account Association Settings panel, enter the main account name and ID of the third-party cloud provider, select the authorized sub-account AccessKey pair, and click Associate Account and Associate Data Source.

      5. In the Data Source Settings panel, configure the data sources for the cloud services you want to ingest.

        • Huawei Cloud: A single data source can ingest data from one OBS bucket. To ingest data from multiple OBS buckets, you must create multiple data sources. Otherwise, you only need to create one data source.

          1. In the Huawei Cloud panel, fill in the Access Method, Data Source Name, Region, and OBS Bucket Name fields, and then click Save Data Source.

          2. Click Add Log Type, select the Log Type to ingest, enter the OBS File Path, and click Save Log Type.

            The custom forwarding path in the OBS File Path field must be configured with minute-level granularity. For example: /LogTanks/cn-north-4/CFW/lts-topic-cfw-0001//%Y/%m/%d/%H/%M.

            After saving a log type, if both Cloud Firewall and Web Application Firewall logs are forwarded to the same OBS bucket, you must click Add Log Type again to create a configuration for the other log type.

        • Tencent Cloud: Cloud Firewall and Web Application Firewall (WAF) alert logs use different ingestion methods. If you want to ingest both log types, you must create a separate data source for each. These steps use Cloud Firewall alert logs as an example. For WAF alert logs, follow the prompts in the console.

          1. In the Data Source Settings - Tencent Cloud panel, fill in the Access Method, Data Source Name, Public Endpoint, Username, and Password fields, and then click Save Data Source.

          2. Click Add Log Type, fill in the Log Topic and Consumer Group Name fields, select a Log Type, and click Save Log Type.

    4. Ingest logs from third-party cloud services

    1. In the left-side navigation pane, choose Agentic SOC > Manage > Integration Settings. In the upper-left corner of the console, select a region where your assets are located: Chinese Mainland or Outside Chinese Mainland.

    2. In the service integration list, find the third-party cloud service you want to integrate and click Ingestion Settings in the Actions column.

    3. In the ingestion settings panel, find the log type to ingest and click the number in the Associated Accounts column.

    4. In the dialog box that appears, select the account to integrate and click OK.

    5. Enable Automatically Associate New Accounts if required.

      If you enable Automatically Associate New Accounts, Agentic SOC automatically ingests the corresponding logs whenever a new third-party cloud account is added.

      Note

      Only a global administrator can configure Automatically Associate New Accounts while in the Global Account View.

    References

    • After ingesting logs, you can configure detection rules to correlate multiple related security alerts into a single security event with a complete attack chain. This reduces the number of alerts and improves the efficiency of alert analysis and response. For more information, see Configure threat detection rules.

    • Use the Agentic SOC Dashboard to centrally monitor your security posture across platforms, accounts, and services with data visualizations and to evaluate your security operations' effectiveness. For more information, see Dashboard.

    • You can use the log management feature in Agentic SOC to quickly browse and query all collected log data, which simplifies log management in multi-resource environments. For more information, see Log management.

    • You can call log ingestion-related API operations to submit batch jobs for service or log integration, view linked cloud accounts, and more. For a list of API operations, see Log Ingestion.

    • Does Agentic SOC support ingesting logs from on-premises devices?

    • What happens if my log ingestion or storage exceeds the limits of my subscription plan?