edit-icon download-icon

Collect multi-channel data

Last Updated: May 08, 2018

The Log Service LogHub function supports real-time data collection and consumption. The real-time collection feature supports over 30 collection methods.

Data is usually collected in two different ways as described in the following table. This document primarily discusses collecting data by using LogHub streaming import (real-time).

Method Pros Cons Example
Batch import Large throughput, focusing on historical data Poor real-time performance FTP, OSS uploads, mailing hard drives, and SQL data export
Streaming import Real-time, WYSIWYG (what you see is what you get), focusing on real-time data Higher requirements on collection terminals Loghub, HTTP upload, IOT, and Queue

Background

“I Want Take-away” is an e-commerce website with a platform involving users, restaurants, and couriers. Users can place their take-away orders using the website, App, WeChat, or Alipay. Once order is received, a merchant starts preparing the ordered dish item, and the nearest couriers are auto-notified. Then, one of the couriers accepts the order and delivers it to the buyer.

1

Operational requirements

During operation, the following issues are identified:

  • Difficulty in getting users. Despite a hefty advertising investment in various marketing channels (webpages ads and WeChat push messages), even when obtain new users, but the effectiveness of each channel cannot be evaluated.
  • Users often complain about the slow delivery. Delay may have the following reasons: order taking, delivery, or preparing the order. How to identify the main cause and improve on the services?
  • User operation teams often organize promotions, such as giving away coupons, but cannot get expected results.
  • In terms of scheduling, how to help merchants stock up food for the peak hours, and send more couriers to a specific area?
  • As for customer service, when users reported that they failed to place an order, what operations had they performed? Did the system error occur?

Data collection challenges

In digital operations, the first step is to figure out how to centrally collect the distributed log data, the following challenges are encountered:

  • Multiple channels: advertisers, street promotions (leaflets)
  • Multiple terminals: webpages, official social media accounts, mobile phones, web browsers (desktop and mobile websites)
  • Heterogeneous networks: VPC, user-built IDC, Alibaba Cloud ECS
  • Various development languages: Java for the core system, Nginx server for the front end, and C++ for the backend payment system
  • Devices: merchants’ devices are running on different platforms (X86, ARM)

Therefore, you must gather the logs distributed externally and internally for unified management. In the past, it took massive and diversified work. Now you can use LogHub’s collection feature for unified access.

multi-logs

Unified log management and configuration

  1. Create a log management project, for example, MyOrder.
  2. Create the log library LogStore for logs generated from different data sources, for example:
    • WeChat-server (for storing WeChat server access logs)
    • WeChat-app (for storing WeChat server application logs)
    • WeChat-error (error logs)
    • Alipay-server
    • Alipay-app
    • Deliver-app (courier app status)
    • Deliver-error (error logs)
    • Web-click (H5 webpage clicks)
    • Server-access (service-side Access-Log)
    • Server-app (application)
    • Coupon (application coupon logs)
    • Pay (payment logs)
    • Order (order logs)
  3. If it is necessary to cleanse and run ETL jobs on the raw data, you can create intermediate LogStores.

For more operations, see Quick Start/Management Console.

Collection of user promotion logs

The two common methods to attract new users are:

  • Directly give away coupons upon website registration
  • Offer coupons for scanning QR codes using other channels
    • QR codes on leaflets
    • Scan QR codes on a webpage to log on

Implementation

Define the following registration server link and generate QR codes (for leaflets and webpages). When the user scans a QR code on a webpage to sign up, you can identify the user as being from a specific source, and create logs accordingly.

  1. http://examplewebsite/login?source=10012&ref=kd4b

When receiving a request, the server outputs the following logs:

  1. 2016-06-20 19:00:00 e41234ab342ef034,102345,5k4d,467890

In the preceding figure:

  • time: Time of registration.
  • Session: The current browser session, used for behavior tracking.
  • Source: Source channels. For example, Campaign A is labelled as “10001”, leaflets “10002”, and elevator ads “10003”.
  • Ref: Referral account, whether someone recommends the user to sign up; left blank if there is none.
  • Params: Other parameters.

Collection method:

  • Application outputs logs to the hard drive using Logtail collection.
  • Application writes the logs using SDK, see SDK.

Service-side data collection

Alipay/WeChat official account programming is a typical web-side model that generally utilizes three types of logs:

  • Nginx/Apache access logs are used for monitoring and real-time statistics:

    1. 10.1.168.193 - - [01/Mar/2012:16:12:07 +0800] "GET /Send?AccessKeyId=8225105404 HTTP/1.1" 200 5 "-" "Mozilla/5.0 (X11; Linux i686 on x86_64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2"
  • Nginx/Apache error logs:

    1. 2016/04/18 18:59:01 [error] 26671#0: *20949999 connect() to unix:/tmp/fastcgi.socket failed (111: Connection refused) while connecting to upstream, client: 10.101.1.1, server: , request: "POST /logstores/test_log HTTP/1.1", upstream: "fastcgi://unix:/tmp/fastcgi.socket:", host: "ali-tianchi-log.cn-hangzhou-devcommon-intranet. sls.aliyuncs.com"
  • Application-layer logs. The application layer logs capture details about events that occurred, such as time, location, result, delay, method, parameter, and usually end with extended fields:

    1. {
    2. "time":"2016-08-31 14:00:04",
    3. "localAddress":"10.178.93.88:0",
    4. "methodName":"load",
    5. "param":["31851502"],
    6. "result":....
    7. "serviceName":"com.example",
    8. "startTime":1472623203994,
    9. "success":true,
    10. "traceInfo":"88_1472621445126_1092"
    11. }
  • Application-layer error logs. Time when the error occurred, code line, error code, reason:

    1. 2016/04/18 18:59:01 :/var/www/html/SCMC/routes/example.php:329 [thread:1] errorcode:20045 message:extractFuncDetail failed: account_hsf_service_log

Implementation

  • Logs are written to local files using the Logtail, and the configuration regular expressions are written to a specified LogStore.
  • Logs generated in Docker can be collected using the Container Service integrated log service.
  • For Java programs, use the Log4J Appender (without saving logs on the hard drives), LogHub Producer Library (for high-concurrency client-side write).
  • For C #, Python, Java, PHP, and C, use the SDK Write.
  • For Windows Server, use the LogStash collection.

Access to end user logs

  • Mobile client: Use the mobile terminal SDK IOS, Android or MAN (mobile analytics) for log access.
  • ARM devices: The ARM platform can use the Native C for cross-compiling.
  • Merchant platform devices: Devices running on an X86 platform can use SDK, and those running on the ARM platform can use Native C for cross-compiling.

Desktop/mobile website users’ page actions

The users’ page actions for collection are divided into two types:

  • Page actions with backend server interaction, including placing an order, logging on, and logging out.
  • Page actions without backend server interaction, including requests that are processed directly at the front end, such as scrolling, and closing a page.

Implementation

  • For the first type, see the service-side collection method.
  • For the second type, use Tracking Pixel/JS Library to collect page actions, see Tracking Web Interface.

Server log maintenance

For example:

  • Syslog logs

    1. Aug 31 11:07:24 zhouqi-mac WeChat[9676]: setupHotkeyListenning event NSEvent: type=KeyDown loc=(0,703) time=115959.8 flags=0 win=0x0 winNum=7041 ctxt=0x0 chars="u" unmodchars="u" repeat=0 keyCode=32
  • Application debug logs

    1. __FILE__:build/release64/sls/shennong_worker/ShardDataIndexManager.cpp
    2. __LEVEL__:WARNING
    3. __LINE__:238
    4. __THREAD__:31502
    5. offset:816103453552
    6. saved_cursor:1469780553885742676
    7. seek count:62900
    8. seek data redo
    9. log:pangu://localcluster/redo_data/41/example/2016_08_30/250_1472555483
    10. user_cursor:1469780553885689973
  • Trace logs

    1. [2013-07-13 10:28:12.772518] [DEBUG] [26064] __TRACE_ID__:661353951201 __item__:[Class:Function]_end__ request_id:1734117 user_id:124 context:.....

Implementation

See the service-side collection method.

Data collection in different network environments

LogHub provides access points in each region with three following access methods:

  • Intranet (classic network): For service access within the current region, providing the best bandwidth link quality (recommended).
  • Internet (classic network): Since it can be accessed by anyone, the access speed may vary with link quality, and we recommend that you use the HTTPS for security and protection.
  • Private network (VPC): For accessing VPC network within the current region.

For more information, see Network access. You can always find a suitable solution.

Additional information

Thank you! We've received your feedback.