edit-icon download-icon

LogHub Writer configuration

Last Updated: Mar 21, 2018

Overview

LogHub Writer uses the Java SDK of Log Service to push data from Reader to a specific Log Service LogHub so that the data can be consumed by other applications.

Note: Given that LogHub cannot implement idempotence, re-running the task upon failover results in data duplication.

LogHub Writer acquires the data generated by Reader by means of the CDP framework and individually converts the data types supported by CDP to the String type. When the specified batchSize is reached, LogHub Writer pushes the data to LogHub in one operation using the Java SDK of Log Service. By default, 1,024 entries of data is pushed at a time.

LogHub Writer supports most LogHub data types. Check whether your data type is supported.

The following list shows the conversion between LogHub Writer and LogHub data types:

Category LogHub data type
Integer string
Floating point string
String string
Date and time string
Boolean string
Binary string

Parameter description

  • endpoint

    • Description: Log Service address.

    • Required: Yes

    • Default value: None

  • accessID

    • Description: Log Service access ID.

    • Required: Yes

    • Default value: None

  • accessKey

    • Description: Log Service AccessKey.

    • Required: Yes

    • Default value: None

  • project

    • Description: Project name of the target Log Service.
    • Required: Yes
    • Default value: None
  • logstore

    • Description: LogStore name of the target Log Service.

    • Required: Yes

    • Default value: None
  • topic

    • Description: Selected topic.
    • Required: No
    • Default value: Null string
  • batchSize

    • Description: number of data entries pushed at a time.
    • Required: No
    • Default value: 1024, and 4096 at the maximum
  • column

    • Description: Column names in each data entry. Also, the data is considered as dirty data if the number of columns is inconsistent with that of the data source.

    • Required: Yes

    • Default value: None

Development in wizard mode

Currently, the wizard mode is not supported for LogHub.

Development in script mode

  • Configure the data synchronization job to write data to LogHub:
  1. {
  2. "type": "job",
  3. "version": "1.0",
  4. "configuration": {
  5. "reader": {},
  6. "writer": {
  7. "plugin": "loghub",
  8. "parameter": {
  9. "endpoint": "http://cn-hangzhou.sls.aliyuncs.com",
  10. "accessId": "accessId",
  11. "accessKey": "accessKey",
  12. "project": "project",
  13. "logstore": "store",
  14. "batchSize": 1024,
  15. "topic": "",
  16. "column": [
  17. "col0",
  18. "col1",
  19. "col2",
  20. "col3",
  21. "col4",
  22. "col5",
  23. "col6",
  24. "col7",
  25. "col8",
  26. "col9"
  27. ]
  28. }
  29. }
  30. }
  31. }

Performance report

Currently unavailable.

Thank you! We've received your feedback.