All Products
Search
Document Center

Simple Log Service:What do I do if the destination Logstores contain unexpected data?

Last Updated:Aug 25, 2023

After you distribute the transformed data to the destination Logstores, the Logstores may contain unexpected data. This topic describes how to troubleshoot the issue in this situation.

The source Logstore is website_log. It contains 1,000 log entries whose SourceIP is 192.0.2.54, 1,000 log entries whose SourceIP is 192.0.2.28, 1,000 log entries whose SourceIP is 192.0.2.136, and 2,000 log entries whose SourceIP is different from these IP addresses. The log entries are transformed. Then, the log entries that meet specific conditions are distributed to different destination Logstores.

  • Transformation requirements

    • Distribute the log entries whose SourceIP is 192.0.2.54 to the 54_log Logstore.

    • Distribute the log entries whose SourceIP is 192.0.2.28 to the 28_log Logstore.

    • Distribute the log entries whose SourceIP is 192.0.2.136 to the 136_log Logstore.

  • Expected results

    • The 54_log Logstore contains 1,000 log entries whose SourceIP is 192.0.2.54.

    • The 28_log Logstore contains 1,000 log entries whose SourceIP is 192.0.2.28.

    • The 136_log Logstore contains 1,000 log entries whose SourceIP is 192.0.2.136.

  • Transformation statements

    e_if(e_search("SourceIP==192.0.2.54"),    
      e_output(name="54-target",
                 project="sls-test",
                 logstore="54_log"))
    e_if(e_search("SourceIP==192.0.2.28"),
        e_output(name="28-target",
                 project="sls-test",
                 logstore="28_log"))
    e_if(e_search("SourceIP==192.0.2.136"),
        e_output(name="136-target",
                 project="sls-test",
                 logstore="136_log"))
  • Storage TargetStorage Target

  • Transformation results

    • The 54_log Logstore contains 3,000 log entries whose SourceIP is 192.0.2.54 or an IP address other than 192.0.2.26 and 192.0.2.136. This does not meet the expectation.

    • The 28_log Logstore contains 1,000 log entries whose SourceIP is 192.0.2.28. This meets the expectation.

    • The 136_log Logstore contains 1,000 log entries whose SourceIP is 192.0.2.136. This meets the expectation.

  • Cause

    When Simple Log Service distributes the log entries after transformation, the log entries that meet the condition for each e_output function are distributed to the specified destination Logstore. The log entries that do not meet the conditions and are not discarded are all distributed to the destination Logstore in the storage destination that is numbered 1. In this example, these log entries are distributed to the 54_log Logstore. By default, the storage destination that is numbered 1 is used.

  • Solution

    Include the e_drop() function in the transformation statements to discard the log entries that do not meet the conditions specified in the statements.

    e_if(e_search("SourceIP==192.0.2.54"),    
      e_output(name="54-target",
                 project="sls-test",
                 logstore="54_log"))
    e_if(e_search("SourceIP==192.0.2.28"),
        e_output(name="28-target",
                 project="sls-test",
                 logstore="28_log"))
    e_if(e_search("SourceIP==192.0.2.136"),
        e_output(name="136-target",
                 project="sls-test",
                 logstore="136_log"))
    e_drop()