Simple Log Service:How do I check the completeness of data that is shipped from Simple Log Service to MaxCompute?

Check completeness using __partition_time__ as the partition key

__partition_time__ is derived from the log's time field. The timestamp is rounded down to the nearest hour to produce the partition value—it is not the time the log was shipped or written to SLS.

For example, a log with a timestamp of 2017-05-19 10:43:00 and a yyyy_MM_dd_HH_mm format, with logs shipped at 1-hour intervals, lands in the 2017_05_19_10_00 partition, regardless of when it was written to SLS. For calculation details, see Ship logs to MaxCompute (legacy).

For real-time log ingestion (no historical backfill), use one of the following approaches:

• Option 1 (recommended): Check shipping task status via the API, SDK, or console

  Use API operations, SDKs, or the SLS console to retrieve shipping tasks for a project or Logstore. The following shows an example API response; the console presents the same data visually.

  {
    "count" : 10,
    "total" : 20,
    "statistics" : {
        "running" : 0,
        "success" : 20,
        "fail" : 0 
    }
    "tasks" : [
        ...
        {
            "id" : "abcdefghijk",
            "taskStatus" : "success",
            "taskMessage" : "",
            "taskCreateTime" : 1448925013,
            "taskLastDataReceiveTime" : 1448915013,
            "taskFinishTime" : 1448926013
        },
        {
            "id" : "xfegeagege",
            "taskStatus" : "success",
            "taskMessage" : "",
            "taskCreateTime" : 1448926813,
            "taskLastDataReceiveTime" : 1448930000,
            "taskFinishTime" : 1448936910
        }
    ]
  }

  taskLastDataReceiveTime is the timestamp of the last data SLS received. Use it to determine whether all data received before a target time T has been shipped to MaxCompute.

  • If taskLastDataReceiveTime is earlier than T plus 300s and every task that finished before T plus 300s has a success status, all data received before T has been shipped. The 300-second offset accounts for transient delivery retries.

  • If any task has a ready or running status, shipping is still in progress. Wait for the task to finish before drawing conclusions.

  • If a task has a failed status, diagnose the failure and retry. If the failure stems from a configuration issue, update the shipping configuration.

• Option 2: Infer completeness from partition existence (approximate)

  If the MaxCompute table is partitioned at 30-minute intervals with 30-minute shipping intervals, and the table contains:

  2017_05_19_10_00
  2017_05_19_10_30

  then the presence of the 2017_05_19_11_00 partition confirms that the 10_00 and 10_30 partitions are complete.

  This approach requires no API calls, but results may be inaccurate. Use Option 1 for authoritative verification.

Check completeness using a custom log field as the partition key

If you configured a custom log field (for example, a date field with values such as 20170518 or 20170519) as the partition key column, note that the field value may differ from the time the log was written to SLS. Account for this difference, then apply the same verification logic from Option 1, using the log write time as your reference point.

For the verification steps, see the section above.

The shipping task succeeded but data in MaxCompute is incomplete

A success status means SLS completed the shipping task—it does not guarantee every log reached MaxCompute. SLS skips logs with partition mapping errors and continues shipping the rest, so a task can succeed with some data silently omitted.

Two partition mapping errors cause this:

• Missing field: The SLS field mapped to the partition key column does not exist in the log. The partition value is null, which MaxCompute rejects.

• Reserved characters: The field value contains forward slashes (/) or other characters that are reserved in MaxCompute and cannot appear in a partition key column.

To fix this, update your partition key column configuration. Using the reserved field __partition_time__ as the partition key avoids both issues. For configuration details, see Limitations.