After you distribute the transformed data to the destination Logstores, the Logstores may contain no data. This topic describes how to troubleshoot the issue in this situation.

Scenario 1: The storage destinations specified in the transformation statements are inconsistent with those configured in the Create Data Transformation Rule panel

The source Logstore is website_log. It contains 1,000 log entries whose SourceIP is 192.0.2.54, 1,000 log entries whose SourceIP is 192.0.2.28, 1,000 log entries whose SourceIP is 192.0.2.136, and 2,000 log entries whose SourceIP is different from these IP addresses. The log entries are transformed. Then, the log entries that meet specific conditions are distributed to the following destination Logstores: 54_log_target, 28_log_target, and 136_log_target.

  • Transformation requirements
    • Distribute the log entries whose SourceIP is 192.0.2.54 to the 54_log_target Logstore.
    • Distribute the log entries whose SourceIP is 192.0.2.28 to the 28_log_target Logstore.
    • Distribute the log entries whose SourceIP is 192.0.2.136 to the 136_log_target Logstore.
    • Discard all other log entries.
  • Transformation statements

    The three destination Logstores are 54_log, 28_log, and 136_log.

    e_if(e_search("SourceIP==192.0.2.54"),    
      e_output(name="54-target",
                 project="sls-test",
                 logstore="54_log"))
    e_if(e_search("SourceIP==192.0.2.28"),
        e_output(name="28-target",
                 project="sls-test",
                 logstore="28_log"))
    e_if(e_search("SourceIP==192.0.2.136"),
        e_output(name="136-target",
                 project="sls-test",
                 logstore="136_log"))
    e_drop()
  • Storage Target

    The three destination Logstores are 54_log_target, 28_log_target, and 136_log_target.

    Storage Target
  • Transformation results

    The 54_log_target, 28_log_target, and 136_log_target destination Logstores contain no data.

  • Cause

    In the e_output functions of the transformation statements, the project and logstore parameters are set differently from Destination Project and Destination Logstore in the Create Data Transformation Rule panel. The project and logstore parameters specify the destination project and Logstores in transformation statements. The transformation statements are domain-specific language (DSL) statements. If the DSL statements use different parameter settings from those specified in the Create Data Transformation Rule panel, the parameter settings in the DSL statements overwrite those in the Create Data Transformation Rule panel. In this situation, only the authorization settings for the storage destinations in the panel are used for data transformation. As a result, the log entries are distributed to the 54_log, 28_log, and 136_log destination Logstores.

    Notice If you use the e_output or e_coutput function, make sure that the storage destinations are consistent between the functions and the Create Data Transformation Rule panel. For more information, see e_output and e_coutput.
  • Solution
    Modify the DSL statements to make sure that the destination Logstores are consistent between the statements and the Create Data Transformation Rule panel.
    e_if(e_search("SourceIP==192.0.2.54"),    
      e_output(name="54-target",
                 project="sls-test",
                 logstore="54_log_target"))
    e_if(e_search("SourceIP==192.0.2.28"),
        e_output(name="28-target",
                 project="sls-test",
                 logstore="28_log_target"))
    e_if(e_search("SourceIP==192.0.2.136"),
        e_output(name="136-target",
                 project="sls-test",
                 logstore="136_log_target"))
    e_drop()

Scenario 2: No transformation statements are entered

  • Transformation statements

    None.

  • Storage TargetNo transformation statements entered
  • Transformation results

    Only the 54_log_target destination Logstore contains data.

  • Cause

    No transformation statements are entered, and multiple storage destinations are configured in the Create Data Transformation Rule panel. In this situation, all log entries are distributed to the destination Logstore in the storage destination that is numbered 1. By default, the storage destination that is numbered 1 is used.

Note
  • If transformation statements are entered without the e_drop() function and multiple storage destinations are configured, the system distributes all the log entries that do not meet the conditions in the statements and are not discarded to the destination Logstore in the storage destination that is numbered 1.
  • If no transformation statements are entered and multiple storage destinations are configured, the system distributes all log entries to the destination Logstore in the storage destination that is numbered 1. Other storage destinations are not used.
  • If transformation statements are entered, the statements include only the e_drop() function, and a single storage destination is configured, the system discards all log entries. The destination Logstore contains no data.
  • If no transformation statements are entered and a single storage destination is configured, the system only replicates the log entries to the destination Logstore.

Scenario 3: Data transformation encounters latency

If the settings in the transformation rule are correct but the destination Logstores contain no data, data transformation may encounter latency.

In this situation, we recommend that you perform the following operations:
  • Modify the numbers of shards in the source and destination Logstores. For more information, see Performance guide.
  • Optimize the logic of the transformation statements. For example, optimize regular expressions.