Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog High-Speed Querying with Confluo from RISElab

High-Speed Querying with Confluo from RISElab

In this article, we will learn how Confluo can help to solve the challenges of high-speed writes, low-latency online query, and low-overhead offline query.

By Jiao Xian

Challenge

More and more applications can collect tens of millions of data points per second in many scenarios, such as terminal IoT network monitoring, smart home, and data centers. The collected data is used for online query display, monitoring, offline root cause analysis, and system optimizations. These scenarios require high-speed writes, low-latency online query, and low-overhead offline query. However, existing data structures can barely meet these requirements. Some data structures focus on high-speed writing and simple queries, while others focus on complex queries, such as ad hoc queries, offline queries, and materialized views. These data structures increase the maintenance overload and reduce the writing performance.

1

Confluo is designed to solve these challenges.

Prerequisites and Typical Application Scenarios

Confluo can solve several challenges at the same time because it has made some trade-offs in some specific scenarios. A typical scenario is telemetry data.

2

Telemetry data has the following important features:

  • Write-once: append data; no updates and deletes
  • Fixed-length type
  • No transactions in concurrent scenarios; only atomicity guaranteed

Based on these data features, Confluo implements an innovative data structure to support high throughput and online or offline query.

Features

Confluo is designed for real-time monitoring and data stream analysis scenarios, such as network monitoring and diagnosis frameworks, time series databases, and a pub/sub message systems. The main features of Confluo include:

  1. high-throughput concurrent writes of millions of data points;
  2. online queries at millisecond timescale;
  3. ad-hoc queries using minimal CPU resources.

Implementation Overview

The basic storage abstraction in Confluo is an Atomic MultiLog (hereinafter referred to as "AM"), which relies on two critical technologies:

  • An AM is a collection of lock-free concurrent logs that stores raw data, aggregate statistics and materialized views. Each log records the concurrent read/write: writeTail and readTail.
  • A set of atomic directives supported on modern CPU hardware are used to update logs: AtomicLoad, AtomicStore, FetchAndAdd, and CompareAndSwap.

An AM is similar in interface to database tables. Therefore, applications need to create an AM with a pre-specified schema, and write data streams that conform to the schema. Applications then create indexes, filters, aggregates and triggers on the Atomic MultiLog to enable various monitoring and diagnosis functionalities.

3

Confluo Data Type System

  • Confluo uses a strictly typed system.
  • Primitive data types: BOOL, CHAR, SHORT, INT, LONG, FLOAT, DOUBLE, and STRING.
 {
 timestamp: ULONG,
 op_latency_ms: DOUBLE,
 cpu_util: DOUBLE,
 mem_avail: DOUBLE,
 log_msg: STRING(100)
 }
- Confluo also requires each record to have an 8-byte timestamp. If an application does not assign write timestamps, Confluo internally assigns one.
- The metric data is of double and string types.
  • Custom data types. After a custom data type has property fields defined, gets registered through the corresponding interface and obtains the reference through the corresponding interface, that data type can be used as a member of the schema and appending data along with operations like filters and triggers, will work out of the box.

Write

  • Create a Store File for storing data

  • Create an AM with a specified schema. Three storage modes are available for an AM: IN_MEMORY, DURABLE, and DURABLE_RELAXED.

    • IN_MEMORY: All data is written purely in memory.
    • DURABLE: The raw data is persisted to secondary storage for each write.
    • DURABLE_RELAXED: The raw data is buffered in memory and only persisted periodically.

  • Perform basic operations on the AM. Define indexes, filters, aggregates, and triggers for the AM.

    • Add indexes. The application layer can create a K-d tree index for each metric.
    • Add filters. A filter consists of relational and boolean operators and filters metrics.

4

  • Add aggregates: This operation is applicable to filtered record aggregates, such as SUM, MIN, MAX, COUNT, and AVG.
  • Add triggers: A trigger is a boolean conditional and applies to operations on a result set, for example, MAX(latency_ms) > 100.

5

Query

Queries in Confluo can either be online or offline, the difference being that online queries require pre-defined rules while offline queries do not.

  • Offline queries are mainly for diagnostic analysis. If the fExpressio has been defined, directly see the FilterLog. Otherwise, query the raw data by using the IndexLog.
  • Real-time stream queries are mainly for real-time monitoring and alerting, and require predefined rules. For example, alerting can be achieved by defining triggers, like the alerting rule SUM(pktSize)>1GB.

6

  • Parser: Spirit is used to implement syntax parsing. Spirit can flexibly process small data models that are compliant with the EBNF syntax specification. Confluo defines several syntax rules, including exp, term, factor, predicate, identificate, value, and quoted_string.
  • Execution plan: After the parser generates an expression, the query scheduler is used to generate the execution plan as response-> filter-> index. Confluo has a simple built-in evaluator that determine whether to use indexes or full scans based on its rough count of the cost.

7

Archive and Compress Data

In addition to raw data, Confluo also needs to store objects like indexes, pre-defined filters, and aggregations. Therefore, This increases the storage overhead. This problem can be solved by archiving some data to cold devices. Currently three data archiving methods are supported: periodic archival, forced archival, and allocator-triggered archival (based on memory).

  • Periodic archival By default, if archival is on, the archiver will run periodically every 5 minutes. The background archival management task periodically detects the size configured for an AM log. Once the limit is exceeded, DataLogs, IndexLogs, and FilterLogs are archived to the cold device and stored there.
  • Forced archival. Regardless of whether archival is turned on for a particular multilog, a user can force archival by calling the interface. The interface supports both full archival and incremental archival based on offset.
  • Allocator-triggered archival (based on memory). In cases where the periodic archiver cannot keep up with write pressure and the maximum memory of the system is reached, the allocator will block until Confluo makes more memory available. After the pre-defined threshold is reached, memory allocation is blocked until all AMs are archived as cold data.
  • Encoding. By default, the LZ4 compression algorithm is used for HeaderLog and the Delta compression algorithm is used for IndexLogs and FilterLogs. Decompression is done by the underlying engine, and the reference count is used to avoid concurrent access between archiving threads and read threads.

Core Technologies

Atomic MultiLog

Atomic MultiLogs are the core of the entire system. Atomic MultiLogs mainly include DataLogs, IndexLogs, FilterLogs, AggregateLogs, and logs that describe how the entire collection of logs are updated as a single atomic operation.

8

  • A DataLog consists of two parts: offset and raw data points. An offset is the unique indentifier of the raw data.
  • An IndexLog is the index of a DataLog and organizes an index by using a radix tree, which is a general-purpose data structure like a dictionary. For example, many prefixes of IP addresses and network addresses can be shared in monitoring scenarios.
  • A FilterLog stores the offsets of the raw data partitioned by windows of time and index filters and windows according to a radix tree.
  • Similar to other logs, an AggregateLog indexes aggregates based on time slices. Because an AggregateLog requires write after read, a collection of thread-local objects are designed to ensure secure access.

Integration Methods

Confluo is an open-source C ++ project. Confluo is available in two modes:

  • It can be used as an embedded dependent database to support online and offline analysis.
  • It can be used as an independent service to expose RPC interface communication.

Summary

The core innovation of Confluo released by RISELab lies in its data structure: Atomic MultiLogs. Confluo supports concurrent read and write at high speed. A single core can run 1,000 triggers and 10 filters. An excellent capability of Confluo is that the atomic operations on new hardware and lock-free logs can support real-time, offline and fast writing.

Database Big Data Internet of Things Data Transmission Service Time Series Database
0 0 0
Share on

Read previous post:

Analysis of the Storage Mechanism in InfluxDB

Read next post:

Prometheus: The Unicorn in Metrics

Alibaba Clouder

2,605 posts | 747 followers

You may also like

Comments

Alibaba Clouder

2,605 posts | 747 followers

Related Products

More Posts by Alibaba Clouder

See All