:User guide

Check service details

1. Log on to the Container Service console.
2. In the left-side navigation pane, choose Container Service - Swarm > Services to go to the Service List page.
3. Click a service name in the Name column such as loghub-shipper to go to the Details page. On the Containers tab page, you can perform the following operations:
   • View service logs.

     Click Logs in the Action column to view real-time logs of LogHub Shipper.

   • Check the running status and alarm rules of the container.
     1. Click Monitor in the Action column to check the running status of the container.
     2. Click View History Monitoring Data/Set Alarm Rules to view monitoring charts and alarm rules. On this page, you can create alarm rules.

Example

loghub:{"endpoint":"https://lhshipper-test.cn-hangzhou.log.aliyuncs.com", "logstore":"test-store","consumer_group":"defaultcg"}
tablestore:{"endpoint":"https://lhshipper-test.cn-hangzhou.ots.aliyuncs.com", "instance":"lhlshipper-test", "target_table":"loghub_target", "status_table":"loghub_status"}
exclusive_columns: ["__source__", "__time__"]
transform: {"rename":"original", "trans-pkey":"(->int 10 original)"}

Write data to Log Service

LogItem logItem = new LogItem();
log.PushBack("original", "12345");
log.PushBack("keep", "hoho");
ArrayList logs = new ArrayList<LogItem>();
logs.add(log);
loghub.PutLogs("lhshipper-test", "test-store", "smile", logs, "");

This log entry contains two fields related to users. One is the original field whose value is 12345, and the other is keep whose value is hoho. Log Service also adds three fields to the log entry. The topic field is set to smile, and the __source__ and __time__ fields change with the environment.

By using a tool, you can see that one data item appears in the data table of Table Store. The following example uses the JSON format.

[{"rename": "12345", "trans-pkey": 12345, "keep": "hoho"},
{"__topic__": "smile", "original": "12345"}]

In this example, rename, trans-pkey, and keep are primary key columns, and __topic__ and original are attribute columns.

Based on environment variables, follow these configuration rules:

• Define "rename": "original" in the transform field. The original field value is 12345 in the log data, so the rename field value is also 12345 in Table Store.
• Define "trans-pkey": "(->int 10 original)" in the transform field. Thus, LogHub Shipper converts the original field value to a decimal integer, and writes the value to the trans-pkey column of Table Store.
• The keep field value does not require conversion. This value is the same in both Table Store and log data.
• LogHub Shipper does not write data in the __source__ and __time__ fields to the data table, because the exclusive_columns environment variable has [“__source__“, “__time__“ ] configured.
• LogHub Shipper writes the __topic__ and original fields as attribute columns to the data table.

The status table updates the status of each process of LogHub Shipper at five-minute intervals. The following example shows the status data in the JSON format.

[{"project_logstore": "lhshipper-test|test-store", "shard": 0, "target_table": "loghub_target", "timestamp": 1469100705202},
{"skip_count": 0, "shipper_id": "fb0d62cacc94-loghub-shipper-loghub-shipper-1", "cu_count": 1, "row_count": 1, "__time__": 1469100670}]

A worker (”shipper_id”: “fb0d62cacc94-loghub-shipper-loghub-shipper-1”) of LogHub Shipper adds a status item (”timestamp”: 1469100705202) at 2016-07-21 T11:31:45.202000Z.

During the five minutes before the timestamp, this worker consumed one log entry from shard 0 (“shard”: 0) of the test-store Logstore (“project_logstore”: “lhshipper-test|test-store”) in the lhshipper-test log project. The worker skips no log (“skip_count”: 0), writes one log (“row_count”: 1), and consumes one Capacity Unit or CU (“cu_count”: 1).

The status table shows the number of log entries skipped in the past five minutes. System exceptions or system upgrades cause some log entries that do not follow the format requirement. LogHub Shipper cannot convert these log entries, and has to skip them.

For example, LogHub Shipper processes the following log data:

LogItem log = new LogItem()
log.PushBack("original", "abcd")
log.PushBack("keep", "hoho")
ArrayList logs = new ArrayList<LogItem>()
logs.add(log)
loghub.PutLogs("lhshipper-test", "test-store", "smile", logs, "")

Based on environment settings, LogHub Shipper converts the original field of log data to an integer, and writes the value to the trans-pkey column in the data table. But the value of original is not a number in the preceding log data, so the status table shows the following data:

[{"project_logstore": "lhshipper-test|test-store", "shard": 0, "target_table": "loghub_target", "timestamp": 1469102805207},
{"skip_sample": "{\"__time__\":1469102565,\"__topic__\":\"smile\",\"__source__\":\"10.0.2.15\",\"original\":\"abcd\",\"keep\":\"hoho\"}", "skip_count": 1, "shipper_id": "fb0d62cacc94-loghub-shipper-loghub-shipper-1", "cu_count": 0, "row_count": 0, "__time__": 0}]

In the status table, skip_count is 1. The skip_sample attribute column indicates that the original log data has been skipped.

Also, the log of Container Service shows the following data:

loghub-shipper_loghub-shipper_1 | 2016-07-21T12:02:56.113581003Z 12:02:56.111 [pool-4-thread-3] ERROR shipper.error - abcd is not 10-based int
loghub-shipper_loghub-shipper_1 | 2016-07-21T12:02:56.114039933Z 12:02:56.111 [pool-4-thread-3] INFO  shipper.core - skip 1 rows
loghub-shipper_loghub-shipper_1 | 2016-07-21T12:02:56.139854766Z 12:02:56.139 [pool-4-thread-3] INFO  shipper.core - skip: {"__time__" 1469102565, "__topic__" "smile", "__source__" "10.0.2.15", "original" "abcd", "keep" "hoho"}

The log indicates the cause of the skipped data: abcd is not 10-based int.

Modify the configuration

1. Log on to the Container Service console.
2. In the left-side navigation pane, click Services to go to the Service List page.
3. Click Update in the Action column next to the LogHub Shipper service to go to the edit page.
4. Click OK.

Upgrade the image

1. Log on to the Container Service console.
2. In the left-side navigation pane, click Applications to go to the Application List page.
3. Click Redeploy in the Action column next to the LogHub Shipper service to go to the edit page.
4. Click OK.

Scale the service

LogHub Shipper is highly scalable and automatically allocates shards based on the number of container instances. When the data transmission capability does not meet the service requirements, you can easily scale the service.

You can scale out the service in two ways. If you frequently scale out and in the service, you can use Auto Scaling and Resource Orchestration Service.

• Add a node.
  1. Log on to the Container Service console.
  2. In the left-side navigation pane, click Clusters to go to the Cluster List page.
  3. Click Manage in the Action column next to the cluster where LogHub Shipper is deployed.
  4. Click Expand on the right of the cluster page to add a Pay-As-You-Go instance, or click Add Existing Instances to add existing instances to the cluster.
     Note For more information about adding existing instances, see Add existing ECS instances to an ACK cluster.
• Add containers to LogHub Shipper: the nodes in a cluster do not increase with the containers of LogHub Shipper. To modify the configuration of LogHub Shipper and then increase the number of LogHub Shipper containers, follow these steps:
  1. Log on to the Container Service console.
  2. In the left-side navigation pane, click Services to go to the Service List page.
  3. Click Update in the Action column next to the LogHub Shipper service to go to the edit page.
  4. Change the number of containers, and click OK.

To scale in the service, follow these steps:

1. Log on to the Container Service console.
2. In the left-side navigation pane, click Services to go the Service List page.
3. Click Update in the Action column next to the LogHub Shipper service to go to the edit page.
4. Reduce the value of Container Quantity.
5. Click OK to return to the Service List page.
6. Click Reschedule in the Action column next to the LogHub Shipper service.
7. In the left-side navigation pane, click Nodes to go to the Node List page.
8. Choose More > Remove in the Action column next to the instance that you want to remove.
9. Different from scaling out, Container Service does not release the removed ECS instance. You have to manually release the instance in the ECS console.