2016 US QCon: the future of IoT has come. What should be our concerns?
Created#More Posted time:Jan 19, 2017 9:44 AM
Editor's Note: Internet of Things (IoT) is always a “bark worse than his bite”. However, at this year’s conference, the length of the speech and disclosed progresses all mean that the era of IoT will arrive very soon. All we need is the final ball, just like the eve before the outburst of smart phones. IoT's technical reserves including standards, hardware and software ecosystems are in place. IoT is also slowly penetrating various areas such as logistics, healthcare, smart hardware, and automobiles. But the value of IoT is not yet fully demonstrated. The core of IoT still lies in the data. IoT will show its true power and status only when the data is generated, accumulated, and circulated and produces value.
The topic of IoT has only started to heat up in recent years, but IoT has been around for a long time. For example, some sensor networks in industrial facilities, networks used for meteorological and seismic data collection and NFC used by supermarkets or parking lots. The difference is that all this previous data is isolated and each network is an information islet. Only when these networks are connected, the data is circulated and thus generates value and remodels each industry, can we assert the final arrival of the era of IoT.
At the conference, IoT also received much attention. Riak shared their understanding and ideas about IoT and some exploration results in IoT data processing. Riak is also working actively for IoT data processing. IoT-related industry includes manufacturing, transportation, defense, agriculture, mineral exploration and production, insurance, smart home, catering, infrastructure, retail, logistics, banking, medical, health, intelligent building and public facilities, according to the keynote speaker. Based on the disclosed data, the market size of IoT in 2016 reached US$6.4 billion and the number will reach US$21 billion in 2020, with an average annual increase around 30 percent. An average of 5.5 million devices will be activated daily. By 2020, 50 percent of our production and life are IoT-related. About US$6 trillion investment will be spent on IoT solutions, among which hardware and application development will account for the major share:
We need to pay attention to the following aspects of IoT projects:
A: IoT-related protocols (such as CoAP, MQTT and HTTP) and data generated (temperature, humidity, location, speed and so on). Make investment after this information is clear and ready.
B: Implement the data collection protocol and the parsing process, and then have this data stored/pooled in the “Data Lake”.
C: After data is stored in the “Data Lake”, conduct data analysis and mining in order to predict and discover the unknown.
The overall structure is shown in the figure:
But as the author says, the whole process is time-consuming and the work load is huge.
IoT ecology is also complex, as shown in the figure:
The author provides a wishlist in the entire data chain of IoT:
A: Network reliability, such as fiber-optic network.
B: Data delivery (specific channel required).
C: End-to-end low latency (strong enough signal required).
D: Quality service (anti-interference and anti-intersection).
E: Engineering topology (high signal-to-noise ratio required).
F: Committed information rate and low bit error ratio.
But the real life situation is much worse:
A: Low-cost hardware is used for the wireless technology.
B: Shared transmission medium uses low-power radio transmission.
C: Limited bandwidth due to very short antennas.
D: Mesh or Ad-hoc topology technology uses custom firmware.
E: The signal interference odds are good and the CoAP protocol is used.
F: Out of order or packet loss due to the use of “shoot and forget” model.
IoT is indeed big data and its characteristics are:
Speed: a large number of small writes, lots of low-latency reads, and streaming update;
Diversity: sensor data, user and device data, timing data;
Scale: At the beginning it is very small, but it keeps growing every day and may take off with the business surge. Thus we need a good retention strategy;
Accuracy: Since the cost and accuracy of equipment are low, it cannot be presumed to be reliable. Data may be damaged or lost during transmission.
Value: Profiles and summaries data is more valuable than raw data. Recent raw data is more valuable than historical data. Aggregated data is more valuable than raw data.
Complexity: The data schema is relatively simple. For speed and flexibility concerns, there will be some unstructured data.
The technical capabilities required to handle IoT big data include:
Raw timing data processing: high throughput, and fast access to the latest data.
Aggregated timing data processing: SQL queries, aggregation, arithmetical operations, bulk queries for analysis and secondary indexing.
Efficient storage: automatic data expiration, compression and hot standby.
Profiles and metadata processing: parallel low-latency reads, reliable writes, structured and semi-structured data, secondary indexing and text indexing.
Scalability and availability: distributed, and linear scale up/down.
Easy operation: no-master architecture, automatic balancing and rolling upgrade.
Databases that can be used to store this IoT data are:
The complete technology stacks for handling IoT data are:
Solutions from ApsaraDB are:
It is worth mentioning that the PetaData product of ApsaraDB is a distributed relational database supporting PB-level storage. Its bottom layer adopts the TokuDB engine, the compression ratio is around 10:1, and PetaData supports retention by time. PetaData is a timing database in nature. Its search function supports rich text searches. PetaData is the integration of OLTP and OLAP and therefore it is a powerful tool in the IoT era. It is also ApsaraDB's investment in the future of IoT.
Summary: the long-awaited IoT is finally coming. But IoT is too large, much larger and more inclusive than the mobile internet. Each person and every industry will become one or more links in the IoT ecology actively or passively. In the IoT era, the difference will come from the depth of understanding of the data. In front of data, the status of the traditional company and the so-called internet companies is the same. The future belongs to those who have an insight into the data. Now what we can do is to do a good job in technical and human reserves, and then wait for it.