Internet of Things (IoT) is enjoying high growth. More and more IoT devices are being applied to our daily life, such as smart routers, various TV dongles, Tmall Genie, and robot vacuum cleaners, bringing us the convenience of intelligence. According to Gartner's prediction, 20 billion smart devices will be put into service by the end of 2020, which gives a glimpse of the huge market in this field. The embedded development model in the traditional software field is faced with great challenges in the IoT field. In addition to the large number and wide distribution, IoT devices are difficult to debug and restricted in hardware. As a result, traditional device log solutions cannot meet the demands perfectly.

Based on years of Logtail development experience and the characteristics of IoT devices, the Log Service team has customized a log data collection solution for IoT devices: C Producer Library.


log data collection solution

Embedded development requirements

The IoT or embedded development engineers must have profound knowledge and experience in development and the capability to manage, monitor, and diagnose a large number of black boxes. Embedded development mainly has the following requirements:

  • Data collection: How can the engineers collect data from millions or even tens of millions of devices distributed all around the world in real time?
  • Debugging: How can the engineers use one solution to meet the requirements of both online data collection and real-time debugging in development?
  • Online diagnosis: When an error occurs on an online device, how can the engineers locate the device quickly and check the error context?
  • Monitoring: How many devices are currently online? How is the working status distribution? How is the geographic distribution? How does a device send alerts in real time when an error occurs?
  • Real-time data analysis: How is the data generated by devices integrated with real-time computing and big data warehouses to build user profiles?

Embedded development requirements

Major challenges in the IoT field

When thinking about the solutions to the preceding problems, we find that the approaches in the traditional software field are ineffective in the IoT field. The main challenges arise from the following characteristics of IoT devices:

  • Large numbers of devices: In the traditional O&M field, a company managing 10,000 servers is qualified for a large one. However, managing 100,000 online devices is only a small threshold in the IoT field.
  • Wide distribution: The deployed hardware devices are usually distributed around the country or even the world.
  • Black box: IoT devices are mostly in unknown states. It is difficult to log on to and debug these devices.
  • Restrictions in resources: IoT devices are relatively restricted in hardware to reduce costs, for example, the total memory size may only be 32 MB. As a result, traditional log collection approaches for PCs do not work in the IoT field.

C Producer Library: a log data collection solution customized by Log Service

Logtail, the client of Log Service, is deployed on millions of x86 servers. For more information, see Logtail technology sharing: and . In addition, Log Service provides a variety of collection solutions:

  • Mobile SDK: It collects data from Android or iOS platforms with tens of millions of daily active user (DAUs).
  • Web Tracking (JS): Similar to Baidu Tongji and Google Analytics, it uses a lightweight collection mode without signature.

In the IoT field, based on years of Logtail development experience and the characteristics of IoT devices in terms of CPU, memory, disk, network, and application mode, we have developed a log data collection solution for IoT devices: C Producer Library.


C Producer Library

Features of C Producer Library

Like a lightweight Logtail, C Producer Library offers high stability, high performance, and low resource consumption. Although it does not have the feature of real-time configuration management in Logtail, C-Producer Library has 70% of the features of Logtail, including:

  • Multiple tenants: C Producer Library can process various types of logs (such as Metric, DebugLog, and ErrorLog) according to their priorities. Multiple clients can be configured and each client can be separately configured with the collection priority and target project or Logstore.
  • Context query: Logs generated by the same client are in the same context, and the relevant logs before and after a log can be viewed.
  • Concurrent sending and resumable upload: The upper limit of cache can be set. Logs fail to be written when the upper limit is exceeded.

In addition, C Producer Library provides the following features specific to IoT devices, including:

  • Local debugging: Logs can be exported to local devices. You can set the rotation, log quantity, and rotation size.
  • Fine-grained resource control: Different cache upper limits and aggregation modes can be set for different types of data or logs.
  • Log cache compression: The cache of the data failed to be sent can be compressed to reduce the memory usage of devices.

Features of C Producer Library

Advantages of C Producer Library

As a custom solution for IoT devices, C Producer Library has obvious advantages in the following aspects:


Advantages of C Producer Library
  • Highly concurrent write on the client: Hundreds of thousands of logs can be written every second with a configurable sending thread pool. For more information, see "Performance test" in this topic.
  • Low resource consumption: Only 70% of the CPU is occupied when 200,000 log entries are written every second. The resources are not affected when up to 100 log entries are generated on low-performance hardware (for example, Raspberry Pi) every second.
  • No data copies on disks of client logs: Data is directly sent to the server through the network after being generated.
  • Client computing separated from I/O logic: Logs are generated asynchronously, without blocking the working thread.
  • Multiple priorities: Different clients can be configured with different priorities to make sure that logs with higher priorities are sent first.
  • Local debugging: Local debugging can be configured to facilitate you to test the applications locally when the network is unavailable.

In the preceding scenarios, C Producer Library simplifies application development. You do not have to consider log collection details or worry about the impact of log collection on your business operations. This makes data collection significantly easier.

To make it distinct, we made a comparison between C Producer Library and other embedded collection solutions. The following table lists the comparison results.

Type C Producer Library Other solutions
Programming Platform Mobile + Embedded Mobile-based
Context Supported Not supported
Multiple logs Supported Not supported (one type of logs)
Custom format Supported Not supported (several limited fields are provided)
Priority Supported Not supported
Environment parameter Configurable Configurable
Stability Concurrency High Medium
Compression algorithm LZ4 (balance between efficiency and performance) + Gzip Optimized
Low resource consumption Optimized Medium
Transmission Resumable upload Supported By default, resumable upload is not supported. Secondary development is required.
Access point 8 (in China) + 8 (outside China) Hangzhou
Debugging Local log Supported Supported in manual mode
Parameter configuration Supported Not supported
Real-time performance Visible on the server side 1 second (99.9%) to 3 seconds (maximum) 1 to 2 hours
Custom processing More than 15 interconnection modes Customized real-time and offline solution

C Producer Library + Log Service solution

C Producer Library integrates with Alibaba Cloud Log Service to form a complete set of log solutions for IoT devices.

  • Large scale
    • Supports writing hundreds of millions of log entries on the client in real time.
    • Supports writing petabytes of data every day.
  • High speed
    • Fast data collection: Data can be consumed after being written without any latency.
    • Quick query: Billions of data records can be processed and queried within 1 second by using a complex query statement (with five conditions).
    • Rapid analysis: Hundreds of millions of data records can be aggregated and analyzed within 1 second by using a complex analysis statement (aggregated with five dimensions and the GroupBy statement).
  • Wide interconnection
    • Seamlessly integrated with various Alibaba Cloud products.
    • Compatible with various open-source storage, computing, and visual systems.

C Producer Library + Log Service solution

Download and use

Download URL: GitHub

One application can create multiple producers, and each producer can include multiple clients. Each client can be independently configured with the target address, log level, local debugging, cache size, custom identifier, and topic.

For more information about installation methods and operation steps, see README.


Download and use

Performance test

Environment configuration
  • High-performance scenario: traditional x86 servers.
  • Low-performance scenario: Raspberry Pi (low power consumption environment).

The following figure shows the configurations.


Environment configuration
C Producer Library configuration
  • ARM (Raspberry Pi)

    • Cache: 10 MB
    • Aggregation time: 3 seconds (If any of the conditions is met, namely, aggregation time, aggregation data package size, and aggregation log quantity, the data is packaged and sent.)
    • Aggregation data package size: 1 MB
    • Aggregation log quantity: 1,000
    • Sending thread: 1
    • Custom tag: 5
  • x86

    • Cache: 10 MB
    • Aggregation time: 3 seconds (If any of the conditions is met, namely, aggregation time, aggregation data package size, and aggregation log quantity, the data is packaged and sent.)
    • Aggregation data package size: 3 MB
    • Aggregation log quantity: 4,096
    • Sending thread: 4
    • Custom tag: 5
Sample log
  1. The total data volume is approximately 600 Bytes for 10 key-value pairs.
  2. The total data volume is approximately 350 Bytes for 9 key-value pairs.
__source__: 11.164.233.187
__tag__:1: 2
__tag__:5: 6
__tag__:a: b
__tag__:c: d
__tag__:tag_key: tag_value
__topic__: topic_test
_file_: /disk1/workspace/tools/aliyun-log-c-sdk/sample/log_producer_sample.c
_function_: log_producer_post_logs
_level_: LOG_PRODUCER_LEVEL_WARN
_line_: 248
_thread_: 40978304
LogHub: Real-time log collection and consumption
Search/Analytics: Query and real-time analysis
Interconnection: Grafana and JDBC/SQL92
Visualized: dashboard and report functions

Test results

Test results on the x86 platform

  • C Producer Library sends up to 90 MB data every second. It consumes only 70% of CPU and 140 MB of memory to upload 200,000 log entries every second.
  • When the sending speed of the server is 200 entries every second, data sending basically does not have any impact on CPU (reduced to less than 0.01%).
  • The average time it takes the client thread to send an entry of data (or generate a log entry) is 1.2 μs.


x86 platform

Test results on the Raspberry Pi platform

  • In the Raspberry Pi test, the frequency of CPU is only 600 MHz. Therefore, the performance is approximately 10% of that of the server. The highest sending speed is 20,000 log entries every second.
  • When the sending speed of Raspberry Pi is 20 entries every second, data sending basically does not have any impact on CPU (reduced to less than 0.01%).
  • Raspberry Pi uses a USB to connect to a PC shared network. The average time it takes the client thread to send an entry of data (or generate a log entry) is about 12 μs.

 Raspberry Pi platform