All Products
Search
Document Center

Simple Log Service:Collect IoT or embedded development logs

Last Updated:Aug 02, 2023

The global market for Internet of Things (IoT) solutions is growing fast. More and more intelligent IoT devices are applied to daily life, such as smart routers, TV dongles, Tmall Genie, and robot vacuum cleaners. However, the embedded development model in the traditional software industry poses big challenges to the IoT industry. For example, traditional device logging solutions cannot meet requirements because a large number of IoT devices are widely distributed in different places and are difficult to debug or scale.

To address these challenges, Simple Log Service provides C Producer Library based on years of experience in developing Logtail. C Producer Library is a log collection solution that is customized for IoT devices.

Log collection solution

Requirements in embedded software development

IoT or embedded software developers must have profound knowledge and rich experience to manage, monitor, and troubleshoot a large number of IoT devices. Embedded software development has the following requirements:

  • Data collection: Collect data from millions or tens of millions of devices distributed around the world in real time.

  • Debugging: Use a set of solutions to collect data online and debug software in real time.

  • Online troubleshooting: Locate an online IoT device and identify the error cause if an error occurs on the device.

  • Monitoring: Monitor the number, statuses, and locations of online IoT devices, and configure alert rules for the devices.

  • Real-time data analysis: Visualize the data generated by IoT devices to create user profiles by connecting the devices with real-time computing platforms and big data warehouses.

Requirements in embedded software development

Major challenges in the IoT field

Traditional software development solutions cannot meet the preceding requirements in the IoT industry due to the following challenges:

  • A large number of IoT devices are deployed. In traditional O&M, managing 10,000 servers is a big challenge. However, in the IoT industry, managing 100,000 online IoT devices is only a basic requirement.

  • IoT devices are widely distributed in different places within a country or around the world.

  • IoT devices are in unknown statuses in most cases and are difficult to access and debug.

  • IoT devices are equipped with low-end hardware to reduce costs. For example, an IoT device may have a total memory size of 32 MB. As a result, traditional log collection solutions for PCs do not apply to IoT devices.

C Producer

A log data collection solution customized by Simple Log Service

Logtail is the agent of Simple Log Service and is deployed on millions of x86 servers. In addition, Simple Log Service provides a variety of collection solutions:

  • Mobile SDK: You can use the SDKs to collect data from Android or iOS platforms with tens of millions of daily active users (DAUs).

  • Web tracking (JavaScript): Web tracking is a lightweight solution that is similar to Baidu Tongji and Google Analytics. You can use web tracking to collect data without a signature.

In these solutions, C Producer Library is a solution that is customized based on Logtail to collect log data from IoT devices. The solution is compatible with the CPU, memory, disk, network, and application mode of IoT devices. The following figure shows the features of these solutions.

Log collection solutions

Features of C Producer Library

C Producer Library serves as a lightweight Logtail to offer high stability, high performance, and low resource consumption. Compared with Logtail, C Producer Library does not support real-time configuration management. However, C Producer Library inherits the following features of Logtail:

  • Multiple tenants: C Producer Library can process multiple types of logs such as Metric, DebugLog, and ErrorLog based on their priorities. You can configure multiple clients for C Producer Library. You can also configure a collection priority, destination project, and destination Logstore for each client.

  • Contextual query: Logs generated by the same client are in the same context. You can view the relevant logs before and after a specified log.

  • Concurrent sending and resumable upload: You can set the maximum cache size. If the size of cached logs reaches the upper limit, no more logs can be written to the cache.

In addition, C Producer Library provides the following features that are specific to IoT devices:

  • Local debugging: You can export logs to local devices. You can configure the log rotation, log quantity, and rotation size.

  • Fine-grained resource control: You can set cache sizes and aggregate modes for different types of data or logs.

  • Log cache compression: If log data fails to be sent to Logstores, the log data can be compressed to reduce the memory usage of IoT devices.

Features of C Producer Library

Advantages of C Producer Library

As a custom solution for IoT devices, C Producer Library has the following advantages:

Advantages of C Producer Library
  • Highly concurrent write traffic: You can configure the thread pool of clients to write hundreds of thousands of logs per second to Simple Log Service. For more information, see the "Performance test" section in this topic.

  • Low resource consumption: Only 70% of the CPU resources are required to write 200,000 logs per second. For low-performance hardware such as Raspberry Pi, the CPU resources are not affected even if 100 logs are generated per second.

  • Direct log shipping: After logs are generated, they are directly sent to Simple Log Service.

  • Logic isolation between computing and I/O: Logs are generated in an asynchronous manner without blocking worker threads.

  • Multiple priorities: You can configure different priorities for your clients to ensure that logs with higher priorities are sent first.

  • Local debugging: You can configure local debugging to test your applications if the network connection is unavailable.

C Producer Library simplifies application development. You do not need to consider the implementation of log collection or the impact on your business. This makes data collection much easier.

The following table compares C Producer Library and other embedded collection solutions.

Type

C Producer

Other solutions

Programming

Platform

Mobile-based and embedded

Mobile-based

Context

Supported

Not supported

Multiple log types

Supported

Not supported

Custom Format

Supported

Not supported (Several limited fields are provided)

Priority

Supported

Not supported

Environment parameter

Configurable

Configurable

Stability

Concurrency

High

Medium

Compression algorithm

LZ4 (balance between efficiency and performance) and Gzip

Optimized

Low resource consumption

Optimized

Medium

Transmission

Resumable upload

Supported

By default, resumable upload is not supported. Secondary development is required to support resumable upload.

Endpoint

8 endpoints in China and 8 endpoints outside China

Hangzhou

Debugging

Local log

Supported

Supported in manual mode

Parameter setting

Supported

Not supported

Real-time collection

Visible on the server side

1 second (99.9%) to 3 seconds (maximum)

1 to 2 hours

Custom processing

More than 15 connection modes

Custom real-time and offline solutions

C Producer Library + Simple Log Service solution

C Producer Library can be integrated with Simple Log Service to provide a full range of log collection solutions for IoT devices.

  • Large scale

    • Writes hundreds of millions of logs from clients to Simple Log Service in real time.

    • Writes petabytes of data per day.

  • High speed

    • Fast collection: Logs can be consumed immediately after being written to Simple Log Service.

    • Fast query: Billions of data records can be processed and queried within 1 second by executing a complex query statement in which you specify five conditions.

    • Fast analysis: Hundreds of millions of data records can be aggregated and analyzed within 1 second by executing a complex analysis statement that includes five aggregate functions and the GROUP BY clause.

  • High compatibility

    • Seamlessly integrated with various Alibaba Cloud services.

    • Compatible with various open source storage, computing, and visualization systems.

C Producer Library solution

Download and use C Producer Library

Download URL: GitHub

You can create multiple producers for each application and create multiple clients for each producer. You can configure the destination IP address, log level, local debugging, cache size, custom identifier, and topic for each client.

For more information about how to install C Producer Library, see README.

Download and use C Producer Library

Performance test

Environment configuration

  • High-performance scenarios: traditional x86 servers.

  • Low-performance scenarios: Raspberry Pi (environments with low power consumption).

The following figure shows the configuration details.

Environment configuration

C Producer Library configuration

  • ARM (Raspberry Pi)

    • Cache size: 10 MB

    • Aggregation time: 3 seconds. If the aggregation time, size of the aggregated packet, or number of aggregated logs reaches the specified thresholds, the data is encapsulated and sent to Simple Log Service.

    • Size of the aggregated packet: 1 MB

    • Number of aggregated logs: 1,000

    • Number of sending threads: 1

    • Number of custom tags: 5

  • X86

    • Cache size: 10 MB

    • Aggregation time: 3 seconds. If the aggregation time, size of the aggregated packet, or number of aggregated logs reaches the specified thresholds, the data is encapsulated and sent to Simple Log Service.

    • Size of the aggregated packet: 3 MB

    • Number of aggregated logs: 4096

    • Number of sending threads: 4

    • Number of custom tags: 5

The following sample log contains nine key-value pairs:

__source__: 192.0.2.1
__tag__:1: 2
__tag__:5: 6
__tag__:a: b
__tag__:c: d
__tag__:tag_key: tag_value
__topic__: topic_test
_file_: /disk1/workspace/tools/aliyun-log-c-sdk/sample/log_producer_sample.c
_function_: log_producer_post_logs
_level_: LOG_PRODUCER_LEVEL_WARN
_line_: 248
_thread_: 40978304
LogHub: Real-time log collection and consumption
Search/Analytics: Query and real-time analysis
Interconnection: Grafana and JDBC/SQL92
Visualized: dashboard and report functions

Test results

  • Test results on the x86 servers

    • C Producer Library can send up to 90 MB of data per second. C Producer Library can also upload 200,000 logs per second, and occupies only 70% of CPU resources and 140 MB of memory.

    • When C Producer Library sends 200 logs per second, it occupies less than 0.01% of the CPU resources.

    • Each sending thread consumes 1.2 μs to send a log on average.

    Test results on the x86 servers
  • Test results on Raspberry Pi

    • The CPU of Raspberry Pi reaches a speed of only 600 MHz. Therefore, the performance of Raspberry Pi is approximately 10% of that of an X-86 server. In this case, C Producer Library can send a maximum of 20,000 logs per second.

    • When Raspberry Pi sends 20 logs per second, it occupies less than 0.01% of the CPU resources.

    • When Raspberry Pi is connected to a PC shared network by using a USB port, each sending thread consumes 12 μs to send a log on average.

    Test results on Raspberry Pi