edit-icon download-icon

User guide

Last Updated: Mar 15, 2018

This topic introduces how to use the LogHub Shipper for Table Store (known as Shipper Service). For more information, see Concept and configuration information and Environment preparations. This topic assumes you have already set up the Shipper Service.

View service details

  1. Log on to the Container Service console.

  2. In the left-side navigation pane, select Swarm > Services.

  3. Click the service name under the service name column (in this example, loghub-shipper is created), to go to the service details page. On this page you can:

    • View service log

      Click Logs under the Action bar, to view the real-time Shipper Service log.

    • View operation status

      Click Monitor under the Action bar, to view the operation status of the container.

Usage example

Example configurations:

  1. loghub:{"endpoint":"https://lhshipper-test.cn-hangzhou.log.aliyuncs.com", "logstore":"test-store","consumer_group":"defaultcg"}
  2. tablestore:{"endpoint":"https://lhshipper-test.cn-hangzhou.ots.aliyuncs.com", "instance":"lhlshipper-test", "target_table":"loghub_target", "status_table":"loghub_status"}
  3. exclusive_columns: ["__source__", "__time__"]
  4. transform: {"rename":"original", "trans-pkey":"(->int 10 original)"}

Write data

Use the Log Service API to write a line of data to the Log Service, as follows.

  1. LogItem log = new LogItem();
  2. log.PushBack("original", "12345");
  3. log.PushBack("keep", "hoho");
  4. ArrayList logs = new ArrayList<LogItem>();
  5. logs.add(log);
  6. loghub.PutLogs("lhshipper-test", "test-store", "smile", logs, "");

The preceding example has:

  • Two user fields.
  • The original field value, 12345
  • The keep field value, hoho.
  • Three fields added by Log Service. The topic field is set to smile, and the __source__ and __time__ fields change with the environment.

View data

Using your preferred tool, you can view data have already been entered in the Table Store table (the following example is converted to JSON format for easy interpretation).

  1. [{"rename": "12345", "trans-pkey": 12345, "keep": "hoho"},
  2. {"__topic__": "smile", "original": "12345"}]

In the preceding example, rename, trans-pkey, and keep are primary key columns, while __topic__ and original are attribute columns.

Based on the environment variable configurations:

  • The transform field defines "rename": "original". In the log data, the original value is 12345. Therefore, the rename value for this row in Table Store is also 12345.

  • The transform field defines "trans-pkey": "(->int 10 original)". Thus, the Shipper Service converts the original value to an integer to be entered in the Table Store trans-pkey column, using base 10 conversion.

  • keep does not have any conversion rules. In Table Store, the attribute class keep value remains consistent with the value in the log data.

  • In Log Data, the __source__ and __time__ fields are not written to the data table because exclusive_columns is set to [“__source__“, “__time__“ ].

  • The __topic__ and original fields are written to the data table as attribute columns.

Query the sync status

Each Shipper Service process adds a row of status data to the status table at intervals of 5 minutes. A status data example, converted to JSON format for easy interpretation, is as follows:

  1. [{"project_logstore": "lhshipper-test|test-store", "shard": 0, "target_table": "loghub_target", "timestamp": 1469100705202},
  2. {"skip_count": 0, "shipper_id": "fb0d62cacc94-loghub-shipper-loghub-shipper-1", "cu_count": 1, "row_count": 1, "__time__": 1469100670}]

In the preceding example, the function “shipper_id”: “fb0d62cacc94-loghub-shipper-loghub-shipper-1” is a Shipper Service worker . At 2016-07-21 T11:31:45.202000Z, the status “timestamp”: 1469100705202 was added.

1 log from the shard number 0 (“shard”: 0) was then processed in the log store test-store of the log project named lhshipper-test (“project_logstore”: “lhshipper-test|test-store”). During this time, the worker skipped 0 log (“skip_count”: 0) and wrote 1 log to the data table (“row_count”: 1), consuming 1 CU (“cu_count”: 1).

Log alert (incorrect formatting)

Logs are skipped when a system exception, system upgrade, or erroneous event forces the output of incorrectly formatted log data. In these cases, the Shipper Service cannot convert the logs, and skips them.

For example, the following log data is entered:

  1. LogItem log = new LogItem()
  2. log.PushBack("original", "abcd")
  3. log.PushBack("keep", "hoho")
  4. ArrayList logs = new ArrayList<LogItem>()
  5. logs.add(log)
  6. loghub.PutLogs("lhshipper-test", "test-store", "smile", logs, "")

In the environment settings, the log’s original field is converted to an integer and written to the trans-pkey column in the data table. However, in this log, the original value is not a number. Therefore, the status table shows the following data:

  1. [{"project_logstore": "lhshipper-test|test-store", "shard": 0, "target_table": "loghub_target", "timestamp": 1469102805207},
  2. {"skip_sample": "{\"__time__\":1469102565,\"__topic__\":\"smile\",\"__source__\":\"10.0.2.15\",\"original\":\"abcd\",\"keep\":\"hoho\"}", "skip_count": 1, "shipper_id": "fb0d62cacc94-loghub-shipper-loghub-shipper-1", "cu_count": 0, "row_count": 0, "__time__": 0}]

In the status table, “skip_count” is 1 and the “skip_sample” attribute column shows the original log data that was skipped.

You can also view the following log information in the Container Service log:

  1. loghub-shipper_loghub-shipper_1 | 2016-07-21T12:02:56.113581003Z 12:02:56.111 [pool-4-thread-3] ERROR shipper.error - abcd is not 10-based int
  2. loghub-shipper_loghub-shipper_1 | 2016-07-21T12:02:56.114039933Z 12:02:56.111 [pool-4-thread-3] INFO shipper.core - skip 1 rows
  3. loghub-shipper_loghub-shipper_1 | 2016-07-21T12:02:56.139854766Z 12:02:56.139 [pool-4-thread-3] INFO shipper.core - skip: {"__time__" 1469102565, "__topic__" "smile", "__source__" "10.0.2.15", "original" "abcd", "keep" "hoho"}

The reason the log was skipped is displayed as “abcd is not 10-based int”.

Change configurations

  1. Log on to the Container Service console.

  2. Select Services to go to the services page.

  3. Find the Shipper Service and click Update in the Action bar to the right.

  4. Change your expected configuration, and then click Update.

Image upgrades

  1. Log on to the Container Service console.

  2. Select Applications to go to the applications page.

  3. Find the Shipper Service and click Redeploy in the right-side Action bar.

  4. Click OK.

Scaling

The Shipper Service is highly scalable and automatically performs shard allocation based on the number of container instances.

Scale up

Two effective methods are available to scale up the service. If you want to frequently scale up and down, we recommend that you use Auto Scaling and Resource Orchestration.

  • Adding nodes:

    1. Go to the “Clusters” page on the Container Service console and find the Shipper Service’s cluster.

    2. Click More > Expand to add Pay-As-You-Go ECS instances.

      Or, click More > Add existing instance to add an existing ECS instance to the cluster.

  • Adding Shipper Service Containers:

    Adding nodes to the cluster does not add a new Shipper Service container. To do so, you must change the Shipper Service configuration. The procedure is as follows:

    1. Log on to the Container Service console.

    2. Select Services to go to the services page.

    3. Find the Shipper Service and click Update in the right-side Action bar.

    4. Modify the container quantity and then click Update.

Scale down

  1. Log on to the Container Service console.

  2. On the left side, select Services.

  3. In the right-side Action bar, click Update.

  4. Decrease the Scale.

  5. Click Update.

  6. Find the Shipper Service and, on the right side, click Reschedule.

  7. Click OK.

  8. On the left side, select Nodes.

  9. Find the ECS instance of the container to be removed and, in the right-side Action bar, select More > Remove.

Note: Unlike the scale up action, the Container Service does not automatically release the idle ECS instances. You must manually release them in the ECS console.

Thank you! We've received your feedback.