Alibaba Cloud Tech Share -IoT and Big Data Analysis Platform

Throughout the history of mankind, technology innovation has provided the catalyst that fueled the evolution of society by delivering methods and systems to better serve our ecosystem, continuously improving in the pursuit of a model that can sustain a perpetual increase in productivity, correlated with the creation of wealth and consequently a more prosperous future. While technology and its applications can take many forms and contribute in many meaningful ways to our society's development there is one unifying factor that has proven, since the beginning of time, to be intrinsic to the rate at which progress can be achieved.


The ultimate goal for a society in order to perpetuate progress is to facilitate and increase the rate at which the exchange of information happens between participating entities; beginning with the early stages of human evolution when upon a decision to cooperate, ideas for improvements have emerged, continuing with the concept of money, the common symbol of monetary value leading to economic developments by incentivizing trade and the movement of commodities, that provided a shared medium and method of levelling the uneven distribution of resources throughout the globe. Increasing the rate at which financial transactions can happen, by virtue of scriptural currency, lead to further developments powered by technology advancements.


The internet and its exponential growth, triggered primarily by technological innovations, changed the way companies and people communicate. Companies like Google, Facebook, Twitter reinforced the relevance of communication and information exchange since their primary source of income is contingent on people exchanging information or structuring that information in a relevant way and presenting it to interested parties.
Email is something we use every day, Facebook, or other social network website and applications, have hundreds of millions of members some of which are not individuals but companies, corporations or education institutions looking to engage, communicate and share information with their audience, public or customers. Communication changed the way we interact with companies as consumers, the speed at which we expect information to arrive or the speed we expect companies to respond to our requests has increased dramatically in the last decade. In the financial sector we trade stock from in front of our
computers and react to news that we receive in real time, as a consequence the fundamental concepts that used to define a stock market have changed.


The power of global communication has made it possible for companies and individuals alike to engage in trading activities in over the counter markets, like the foreign exchange, twenty-four hours per day, seven days per week, something that would be unimaginable a couple of years ago considering trading activities used to be settled in days, if not weeks, not seconds or minutes. This communication breakthrough allowed individuals to access these types of markets regardless of their location on the globe, in real-time.
In the past these types of advantages were only available to major banks and investors, proving yet again that communication is not only enhancing our lives but can become the potential driving force behind economic growth, diversified revenue streams, and together with its innovating characteristics, unforeseen profitability.


Communication speed reached a point where humans can be augmented by computers in numerous activities, for instance by using machine learning techniques on stock market data and/or fundamental news in real time the whole trading/investment process can be left in the hands of a computer which has retrospectively proven to not lose investors' money. We reached the technological advancements where the generation of data based on empirical observation automated via sensors and machines permit the collection of information at an unprecedented rate, data that can be later correlated and applied in innovative, potentially revenue generating ways or expose new opportunities in the automation of activities where human effort is no longer required.


Developing a centralized platform that could provide meaningful insight by combining machine and human generated data, from all available sources within an economic environment, can have a significant impact on society. The healthcare industry can benefit from medical sensor data and the patience's browsing history, current location and publicly available discussions on social platforms, all relevant data in providing a diagnosis or preventive healthcare. A city's transportation network can be augmented temporarily based on information received not only from city spread sensors but social-media expressed desired to travel to a certain location on the onset of a global event such as the Olympics. Sales and Marketing departments of consumer oriented companies can correlate information on an unprecedented scale while taking in account all available data inside the platform. There are numerous examples and applications to be discovered, while ensuring all security standards and privacy laws are being followed with a major advantage of decreasing platform operational costs through centralization. An economy where financial players together with government institutions are willing to cooperate in an auspicious political context can have the potential of greatly improving the economic output of said entity.


A milestone in providing for the medium to allow the free information flow and integration of data from multiple sources was reached upon the development of data formatting standards (i.e XML, JSON) that power the interoperability of disperse systems not originally envisaged as providing value in a joint capacity. Large multinational companies have benefitted from the these standards improving internal communication and exposing opportunities for growth while deduplicating efforts, minimizing costs and increasing their departments profitability. (see IBM ca. 1992).


In order to meet the stringent forward looking requirements in terms of speed of service instantiation and the immediate availability of business and technical capability to sustain innovative revenue models, apply machine learning algorithms and account for the exponential growth patterns on every network level, we will have to leverage pioneering technology paradigms that account for and seamlessly expose flexible and scalable components exchanging information in order to achieve self organisation and adaptability driven by software definitions, with near instant applicability in response to automatically gathered business and technical KPI data.


The platform's role is to provide a dependable medium for data to be correctly measured, cross-referenced with participating entities, correctly monitored, tracked and safeguarded from entry to the rendering and presentation interface. Towards achieving this milestone, starting from our existing deployment, a number of services inside our network will act on said data, save it, analyse it, carry it forward to deliver the functionality of a data analysis exchange. Any system or service interacting with our data flow can be replaced in our modular design with an equivalent that will deliver the same value to the network or have a similar effect on our data. In other words, as long as we enforce rules and principles of interactions between services and data, we will never be tied to a specific implementation. The change in our service or software providers will be a matter of choosing the vendor that can be delivered most effectively and bring the most value to our network, backed by technical and/or financial considerations.


Implementing a system of the magnitude described above is outside the scope of this article, as it stands today Alibaba Cloud offers the features required to instantiate a proof of concept deployment showcasing the potential expandability of the platform. A few of the software initiatives that were considered for this demo are: Lumify, Talend Open Studio for Big Data, HPCC Systems Big Data, Apache Storm, Apache Hadoop , Apache Drill, Apache Samoa, Ikanow. As most the aforementioned projects are available as prebuilt AMIs on alternative cloud offerings I strongly encourage Alibaba to proceed with establishing partnerships with vendors that can deliver such AMIs or alternatives and open programmatic interfaces for importing VM images of other formats.