How to handle empty destination LogStore after data transformation?

Scenario 1: The storage targets in the transformation statement differ from those configured on the Create Data Transformation Job page

The LogStore named website_log contains 5,000 log entries: 1,000 with SourceIP 192.0.2.54, 1,000 with SourceIP 192.0.2.28, 1,000 with SourceIP 192.0.2.136, and 2,000 with other SourceIP values. These entries are now processed and distributed to the following target LogStores: 54_log_target, 28_log_target, and 136_log_target.

• Processing Requirements

  • Distribute logs with SourceIP 192.0.2.54 to the LogStore named 54_log_target.

  • Distribute logs with SourceIP 192.0.2.28 to the LogStore named 28_log_target.

  • Distribute logs with SourceIP 192.0.2.136 to the LogStore named 136_log_target.

  • Discard all logs that do not meet these conditions.

• Transformation statement

  The three target LogStore names used in the statement are 54_log, 28_log, and 136_log.

  e_if(e_search("SourceIP==192.0.2.54"),    
    e_output(name="54-target",
               project="sls-test",
               logstore="54_log"))
  e_if(e_search("SourceIP==192.0.2.28"),
      e_output(name="28-target",
               project="sls-test",
               logstore="28_log"))
  e_if(e_search("SourceIP==192.0.2.136"),
      e_output(name="136-target",
               project="sls-test",
               logstore="136_log"))
  e_drop()

• Storage targets

  The actual destination LogStores are 54_log_target, 28_log_target, and 136_log_target.

  The three storage targets are named 54_target, 28_target, and 136_target. All use the region China (Chengdu), the project sls-test, and the authorization method default role.

• Transformation result

  The LogStores named 54_log_target, 28_log_target, and 136_log_target contain no data.

• Root cause

  In the project (destination project name) and logstore (destination logstore name) parameters of the e_output function in the transformation statement, you specify values that differ from those set for Destination Project and Destination Database in the Create Data Transformation Task panel. In this case, the storage destination defined in the transformation statement takes precedence.

  In this case, the data processing result is distributed to the three Logstores specified in the data processing statement: 54_log, 28_log, and 136_log.

• Solution

  • If you configure only the name parameter in the e_output or e_coutput functions, the output goes to the storage targets defined in the Create Data Transformation Job panel.

  • If you specify the Project and LogStore parameters in e_output or e_coutput, they must exactly match the storage targets configured in the Create Data Transformation Job panel. For more information, see e_outputLogStoreut.

    Note

    When you define a transformation statement with multiple storage targets but do not include an e_drop() statement, all processed logs that do not match any condition—and are not dropped—are sent to the first storage target.

    e_if(e_search("SourceIP==192.0.2.54"),    
      e_output(name="54-target",
                 project="sls-test",
                 logstore="54_log_target"))
    e_if(e_search("SourceIP==192.0.2.28"),
        e_output(name="28-target",
                 project="sls-test",
                 logstore="28_log_target"))
    e_if(e_search("SourceIP==192.0.2.136"),
        e_output(name="136-target",
                 project="sls-test",
                 logstore="136_log_target"))
    e_drop()

Scenario 2: No transformation statement is set, or only an e_drop() statement is used

• Transformation statement

  None

• Storage target: This scenario configures two storage targets: Storage target 1 (54-target) has the destination database 54_log_target, and Storage target 2 (28-target) has the destination database 28_log_target. Both are located in Project sls-test, with the authorization method set to default role.

• Transformation result

  Only the LogStore named 54_log_target contains data.

• Root cause

  • No transformation statement is configured on the data transformation page, but in the Create Data Transformation Task panel:

    • If you configure one storage target, data from the source LogStore is copied to that target.

    • If you configure multiple storage targets, all data is sent only to the LogStore of storage target 1 and is not forwarded to other targets.

  • If you define only an e_drop() statement, all log data is discarded, and the destination LogStores remain empty.

Scenario 3: processing delay

If your transformation job is correctly configured but the destination LogStore still shows no data, the issue may be due to transformation delay.