Real-time OLAP, from 0 to 1

1. Business background

1.1 Business Introduction - ABCD

BTC.com is a provider of blockchain technology solutions. Our business is mainly divided into four parts. In summary, it is ABCD: A is artificial intelligence machine learning, B is blockchain, C is cloud, and D is data. These modules are not only independent of each other, but also can be combined with each other. In recent years, the accelerated development of artificial intelligence and blockchain is closely related to the support provided by big data.

1.2 Business Introduction - Blockchain Technology Solution Provider

Generally speaking, the blockchain can be understood as an irreversible distributed ledger. Our role is to enable everyone to better browse the ledger and mine the information and data behind the ledger. At present, the data volume of Bitcoin is about billions to tens of billions, and the data volume is about tens of terabytes. Of course, we also have some other businesses, such as Ethereum currency and smart contract analysis services.

Overall, we are a provider of blockchain technology solutions, providing mining services. Like banks in the financial industry, we also have a lot of OLAP needs.Tracking, statistics, etc., to provide assistance to the police.

2. Opportunities and challenges

2.1 Previous architecture

Around 2018, there were fewer competitors, and our overall structure was as above. The bottom layer is the nodes of the blockchain, which are continuously parsed to MySQL through Parser, and then extracted from MySQL to Hive or Presto, run various scheduled tasks from Spark to analyze data, and then obtain reports or data through visual queries. The problem with the architecture is also obvious:

Cannot process data in real time

There is a single-point problem, for example, a link suddenly hangs up, and the whole link will have problems at this time

2.2 Needs and challenges encountered

Efficiency, efficiency issues are very common. Our tables are on the order of billions. It may take a long time to run this kind of SQL. SQL queries are relatively slow, which seriously affects the statistical efficiency.

Real-time, the data is not real-time, it needs to wait for a certain period of time to update, such as yesterday's data can only be seen today.

Monitoring, real-time requirements, such as real-time risk control, every time a block appears in the blockchain, we have to analyze it, but the time when the block appears is random. Lack of complete monitoring, sometimes the homework is suddenly broken, or the target is not reached, we cannot know in time.

2.3 What do we need to consider in technology selection

What do we need to consider when choosing a technology? The first is capacity reduction. The market in 2020 is not very good. Everyone is trying their best to reduce costs and live better. In the case of limited cost, how can we do more things? We must improve our own efficiency and at the same time ensure quality. So we need to find a balance between cost efficiency and quality.

3. Architecture Evolution

3.1 Technology selection

As the saying goes, if you choose a good tool, you get off work early. We have thought about whether to introduce Flink for a long time. What are its advantages compared with Spark?

After our actual research, we found that Flink still has many advantages, such as flexible windows, precise semantics, low latency, and support for second-level, real-time data processing. Because the team itself is more proficient in Python, we chose PyFlink at that time. With the support of a professional development team, the changes in recent versions have been relatively large, and many functions have been realized. In terms of real-time OLAP, we use ClickHouse for the database.

3.2 Why use ClickHouse

Why use ClickHouse? The first is fast, and the query efficiency is high. Big companies such as ByteDance, Tencent, and Kuaishou are all using it. At the same time, we also have accumulated technology in C++, which is relatively easy to use and the cost is not too high.

3.3 Real-time OLAP architecture

Based on the above technical selection, we have formed the architecture in the above figure. The bottom layer is the data source, including the nodes of the blockchain, which are parsed to Kafka through Parser. Kafka is responsible for connecting Flink and Spark tasks, and then Flink outputs the data to MySQL and ClickHouse supports report export, data statistics, data synchronization, OLAP statistics, etc.

In terms of data governance, we refer to the industry's layering, which is divided into the original layer, detail layer, summary layer, and application layer. We also have machine learning tasks, which are deployed on the K8s platform.

3.4 Architecture evolution process

The evolution process of our architecture is shown in the figure below. From Spark and Hive in 2018 to Tableau visualization later, we came into contact with Flink this year and started using ClickHouse in the second half of the year. Later, there were more Flink tasks. We developed a simple scheduling platform. Developers You only need to upload the task, and the task will be run regularly or in real time.

3.5 Thoughts on Architecture Evolution

Why is the evolution so slow, because the development of the blockchain has not yet reached a certain level, and it cannot have hundreds of millions or PB-level data volumes like some large companies. Our data volume is not that large, and the blockchain is a new thing without a certain history. Another problem is the problem of resources. Due to the shortage of personnel, the cost of personnel has also been controlled.

We have summarized what kind of enterprise it is suitable for the architecture just mentioned. First of all, there is a certain data scale. For example, if an enterprise MySQL only has tens of millions of data, MySQL, Redis, and MongoDB can be used, but this architecture is not suitable. Secondly, a certain amount of cost control is required, and the whole set of costs will be much lower than that of Spark. There must be a technical reserve, and it is necessary to develop and understand related things.

Characteristics of blockchain data. The amount of data is relatively large, historical data is basically unchanged, real-time data is relatively more valuable, and there is a certain relationship between data and time.

3.6 Value generated by real-time OLAP

After the real-time OLAP is launched, it basically meets the business needs, and the cost is also within a controllable range.

What is suitable is the best. Don't blindly pursue new technologies, such as data lakes. Although they are good, the magnitude of our data is actually not used.

We do not consider building a technology center. Our company is small and medium-sized, and it is relatively easy for departments to communicate without too many barriers. We have not developed to a certain organizational scale, so we have no plans to develop a technology center or a data center. Blindly follow the trend to the middle stage.
The effect we achieve is to shorten the development time and reduce the running time of the job.

4. Architecture optimization

4.1 Flink and ClickHouse

There are some linkages between Flink and ClickHouse, and we have customized three jobs.

Custom sinks.

ClickHouse needs to insert a lot of data at one time, so it needs to control the frequency of writing, and write to the local table first, which takes a lot of time.

We mainly use it in the transaction analysis of smart contracts. The new data is relatively large and frequent, and there is a lot of data every few seconds. There are many associations in the data.

4.2 Problems encountered by ClickHouse

Failure and error tolerance when bulk importing.

Upsert optimization.

Commonly used UDFs have been developed. Do you know that ClickHouse officially does not support UDFs? It can only be patched to ensure that ClickHouse will not hang.

We are also doing some open source follow-up, making some patch attempts, and bringing together UDFs commonly used in our business and technology.

4.3 Batch import strategy

Historical data can be considered as a kind of cold data, which is relatively infrequently changed. When importing, it is segmented according to size and sorted according to the primary key, similar to bitcoind, the underlying Checker and Fixer work, and timely alarm and repair during the import process. For example, if a certain data fails to be imported, how to better detect it in time, before it could only be monitored by human flesh.
Real-time data, we need to constantly analyze real-time data. You may not be familiar with the concept of reorganization and 51%. Let me briefly talk about it here. The longest chain in the above picture is also the most important chain. The chain above it is a reorganization and division A chain that is forked, when an attacker or a miner digs the above chain, the final result will lead to the abandonment of the chain and no rewards.

If it exceeds 51% of the computing power, it will achieve such an effect and become the longest chain. This is a relatively high cumulative difficulty. At this time, we will think that the data import has failed. Roll back and reorganize until the most complete chain is satisfied. Of course, we will also set some records and CheckPoints. The concept of CheckPoint here is also different from Flink's CheckPoint.

It is the CheckPoint of the blockchain. The blockchain has a currency called bch, which defines the CheckPoint. When a certain length is met, it cannot be rolled back, avoiding the attack of the attacker. We mainly use CheckPoint to record information to prevent rollback. At the same time, we also record the failure or success of batch insertion according to level/table. If it fails, we will retry and alarm rollback.

4.4 Optimization of Upsert

ClickHouse does not support Upsert. It is mainly compatible with the SDK. It used to write data directly to MySQL. The goal is to modify the corresponding SDK through the SQL statement to add the join of the temporary small table, and perform the Upsert operation by joining the temporary small table.

For example, blockchain address account balances, like bank account balances, must be very precise.

4.5 Kubernetes optimization

Optimizations in Kubernetes. Kubernetes is a very complete platform.

Highly available storage. In the early days, we deployed services on Kubernetes as much as possible, including Flink clusters, basic business components, currency nodes, and ClickHouse nodes. ClickHouse does a better job in this regard, is convenient and compatible, and supports high Actions available.

Support horizontal expansion.

In terms of service discovery, we made some customizations.

4.6 How to ensure consistency?

Use Final to query and wait for the data merge to complete.

In terms of data, achieve idempotency, ensure uniqueness, sort out a set of data by primary key sorting, and then write.

When an exception is written, it will be repaired and backfilled in time to ensure final consistency.

4.7 Monitoring

Use Prometheus as monitoring tool. Easy to use and low cost.

5. Future Outlook

5.1 From 1 to 2

Expand more business and data. In the past, our business model was relatively simple, with only data statistics, and more information will be mined later, including on-chain tracking and financial audits.

Only by earning more money and living as long as possible can we do more things and explore more profit models.

Follow up the ecology of Flink and PyFlink, actively participate in open source work, and optimize related operations. Explore the work of multiple sinks, the practice of native Kubernetes.
5.2 From 2 to 3

Specifications, means and operations of data modeling.

Flink and machine learning combined.

Strive to get the business of real-time online training, and Flink is a very good choice for real-time monitoring. Large companies already have related practices. Including alarm and other operations.