Unexpected data in a destination Logstore

A Logstore named website_log contains 5,000 log entries. Of these, 1,000 entries have a SourceIP of 192.0.2.54, 1,000 have a SourceIP of 192.0.2.28, and 1,000 have a SourceIP of 192.0.2.136. The remaining 2,000 entries have other SourceIP values. You want to transform these entries and send them to different destination Logstores.

• Transformation requirements

  • Distribute log entries with a SourceIP of 192.0.2.54 to a Logstore named 54_log.

  • Distribute log entries with a SourceIP of 192.0.2.28 to a Logstore named 28_log.

  • Distribute log entries with a SourceIP of 192.0.2.136 to a Logstore named 136_log.

• Expected results

  • The 54_log Logstore contains 1,000 log entries, all of which have a SourceIP of 192.0.2.54.

  • The 28_log Logstore contains 1,000 log entries, all of which have a SourceIP of 192.0.2.28.

  • The 136_log Logstore contains 1,000 log entries, all of which have a SourceIP of 192.0.2.136.

• Transformation statement

  e_if(e_search("SourceIP==192.0.2.54"),    
    e_output(name="54-target",
               project="sls-test",
               logstore="54_log"))
  e_if(e_search("SourceIP==192.0.2.28"),
      e_output(name="28-target",
               project="sls-test",
               logstore="28_log"))
  e_if(e_search("SourceIP==192.0.2.136"),
      e_output(name="136-target",
               project="sls-test",
               logstore="136_log"))

• Storage target

  You configure three storage targets: 54-target (default), 28-target, and 136-target. These targets route data to the 54_log, 28_log, and 136_log Logstores, respectively, in the sls-test project.

• Actual results

  • (Unexpected) The 54_log Logstore contains 3,000 log entries. In addition to the expected logs with a SourceIP of 192.0.2.54, it also contains logs with other SourceIP values.

  • (As expected) The 28_log Logstore contains 1,000 log entries, all of which have a SourceIP of 192.0.2.28.

  • (As expected) The 136_log Logstore contains 1,000 log entries, all of which have a SourceIP of 192.0.2.136.

• Cause

  Logs that match a condition in an e_output function are sent to the specified destination Logstore. Logs that do not match any condition are automatically sent to the default storage target. In this example, the default target (54-target) routes unmatched logs to the 54_log Logstore.

• Solution

  Add the e_drop() function to the end of your transformation statement to discard all unmatched logs.

  e_if(e_search("SourceIP==192.0.2.54"),    
    e_output(name="54-target",
               project="sls-test",
               logstore="54_log"))
  e_if(e_search("SourceIP==192.0.2.28"),
      e_output(name="28-target",
               project="sls-test",
               logstore="28_log"))
  e_if(e_search("SourceIP==192.0.2.136"),
      e_output(name="136-target",
               project="sls-test",
               logstore="136_log"))
  e_drop()