the code is on the riak-ts branch of github riak!
Riak KV is built on the Riak kernel and provides a high-elasticity and high-availability key-value database. Riak KV products are currently being continuously improved, focusing on data correctness, prevention of data loss and destruction, and other features.
Riak TS product originates from Riak KV database and is a product specially built for time series data warehouse. It integrates all the powerful functions of Riak KV products and uses these functions to solve problems encountered by users in processing time series data. What features have we really achieved in this product? Here I list some of them:
- the fast data writing path;
- create a model for the data bucket;
- query planning and query subsystem;
- parallel data extraction from virtual nodes;
- flexible composite key value;
we also checked the market situation of Time series database products. At that time, only a few solutions were found, and the quality of all these solutions was not enough to bear the production workload of enterprise-level. Existing time series data solutions are lack of scalable clusters or elasticity, or complex management and operation. All these make them bad choices.
To discuss the idea of solving this problem, we then held an architecture meeting. Finally, one of our engineers came up with an interesting idea, that is, using quantum (time range) to distribute the data around the hash loop, and based on this idea, a concept verification prototype that seems to work well is constructed. . Based on this, we started the development process of Riak TS products, trying to solve many more difficult problems in time series data processing.
IoT databases should be as flexible as required by the application. NoSQLdatabases -- especially key-value, document and column family databases -- easily accommodate different data types and structures without the need for predefined fixed schemas. NoSQL databases are good options when an organization has multiple data types and those data types will likely change over time. In other cases, applications that collect a fixed set of data -- such as data on weather conditions -- may benefit from a relational model. In-memory SQL databases, such as MemSQL, offer this benefit.
Managing a database for IoT applications in-house
For those organizations choosing to manage their own databases, DataStax Cassandra is a highly scalable distributed database that supports a flexible big table schema and fast writes and scales large volumes of data. Riak IoT is a distributed, highly scalable key-value data store which integrates with Apache Spark, a big data analytics platform that enables stream analytic processing. Cassandra also integrates with Spark as well as other big data analytics platforms, such as Hadoop MapReduce.
OpenTSDB is an open source database capable of running on Hadoop andHBase. The database is made up of command line interfaces and a Time Series Daemon (TSD). TSDs, which are responsible for processing all database requests, run independently of one another. Even though TSDs use HBase to store time-series data, TSD users have little to no contact with HBase itself.
MemSQL is a relational database tuned for real-time data streaming. With MemSQL, streamed data, transactions and historical data can be kept within the same database. The database also has the capacity to work well with geospatial data out of the box which could be useful for location-based IoT applications. MemSQL supports integration with Hadoop Distributed File System and Apache Spark, as well as other data warehousing solutions.
you've heard the hype, the Internet of Things (IoT) is going to connect more people to devices, more devices to the Internet and generate more data than any major IT shift in history. IoT is going to be bigger than the web, mobile and the cloud, right? It's still too early to tell for sure, but at InfluxData we are helping startups and enterprises everyday bring an interconnected world closer to reality.
What does time-series have to do with IoT? Everything, actually. Sensors and devices used in IoT architectures emit time-series data, and a lot of it.
Why are companies building IoT and sensor data solutions?
Whether it's pH and humidity readings from an agri-sensor, depth and fluid readings from a geo-sensor or voltage and temperature from a power control sensor, these metrics are forming the basis of intelligent businesses. Common use cases we run across are:
- Agro industries are monitoring and trying to control environmental conditions for optimal plant growth.
- Power and utility companies are building smart solutions to reduce resource wastage for residential and commercial customers.
- Research labs and heavy industries are tracking the resources, usage and health of millions of tiny valves and instruments that go into their massive production plants, factories and manufacturing facilities.
- Smart cars are now powerful computers making runtime decisions based on data collected by 100s of sensors on every vehicle.
Challenges in building IoT and sensor data solutions
The key challenges organizations face while building an IoT solution are:
- Bandwidth - As sensors are generally deployed on-premise and need to communicate over wireless networks, bandwidth constraints prevent sending large packets of data in real-time
- Horsepower - Compute power on sensors are generally limited. Hence analytics software - programs or databases or even processing logic needs to have a tiny footprint.
- Concurrency - In case of industrial IoT, number of sensors could easily range in 100s of 1000s, each transmitting metrics every minute or so. Anticipating backend database's concurrency limits is crucial in the design of such solutions
- Protocol - As this space is rapidly evolving, there aren't any definitive standards for communication protocols. MQTT, AMQPP, CoAP etc are being used based on use cases. Hence IoT analytics solutions need to support many communication protocols.
- Scale - Data retention, compression and visualization has it's own challenges in such a large data footprint solution. Businesses want to plot trends (WoW, MoM, YoY) and aggregation of massive data sets can be very compute heavy.
NoSQL Database: The NoSQL database is typically used to address the fast data ingest problem for device data. In some cases, there may be a stream processor-e.g. Storm, Samza, Kinesis, etc.-addressing data filtering and routing and some lightweight processing, such as counts. However, the NoSQL database is typically used because unlike most SQL databases, which top out at about 5,000 inserts/second, you can get up to 50,000 inserts/second from NoSQL databases. However, NoSQL databases are not designed to handle the analytic processing of the data or joins, which are common requirements for Internet of Things applications. NoSQL effectively provides a real-time data ingest engine for data that is then moved to Hadoop using an extract, transform and load (ETL) process.-- NOSQL writes fast, but data analysis and joint query are inconvenient!
This article is forwarded from Zhang Xinhua-sky blog, original link: http://www.cnblogs.com/bonelee/p/6265106.html, if you need to reprint, please contact the original author.