Elasticsearch: Let the data speak for itself

With the explosive growth of Internet data scale, how to quickly obtain useful information from massive historical and real-time data has become more and more challenging. Among them, search, as one of the most efficient ways to obtain information, has been increasingly favored by people.

An excellent search engine, which connects ordinary users and website pages, users can easily and free search for the websites and content they want to see, and the content of these websites is retrieved by search engines and presented to users through search engine technology.

On November 13, at the 2017 Hangzhou Computing Conference, Elasticsearch and Alibaba Cloud announced a strategic cooperation to jointly develop and release Elasticsearch hosted on Alibaba Cloud to provide a new user experience for the Chinese market. What are the opportunities and challenges for Elasticsearch to enter the Chinese market? What new services does Alibaba Cloud Elasticsearch provide Chinese users? To that end, InfoQ interviewed Shay Banon, founder and CEO of Elasticsearch.

After just an hour of communication, it is obvious that Shay Banon has a keen sense of business. He has been working in the search field for 18 years and founded Elasticsearch almost 8 years ago. He said that the most important thing about starting a business is to find what you are good at and maintain your passion and love. Starting a business means you have to find problems in your life. Then use creative thinking to solve them.

Elasticsearch started as a cookbook application
When talking about the original intention of getting in touch with Lucene and developing Elasticsearch, Shay Banon thought that his participation in Lucene was completely accidental. He was an unemployed engineer and came to London with his new wife, who wanted to study in London to be a I am a chef, and I want to develop an application for my wife to easily search for recipes, so I came into contact with Lucene. Directly using Lucene to build search has many problems, including a lot of repetitive work, so Shay Banon continues to abstract on the basis of Lucene, making it easier to embed search in Java programs. After a period of polishing, he was born. The first open source work "Compass", which means "compass" in Chinese. Afterwards, he found a new job in a high-performance distributed development environment. During his work, he gradually discovered the need for an easy-to-use, high-performance, real-time, and distributed search service, so he decided to rewrite Compass. Built it from a library to a stand-alone server, and created an open source project.

The first public release appeared in February 2010, and since then Elasticsearch has become one of the most popular projects on Github.

Elasticsearch's success stems from open source
After eight years, Elasticsearch is also popular among engineers in China. Shay Banon said that the key factor for Elasticsearch's success is that it is open source and has different use cases besides search, such as log management, security and analysis.

He believes that the open source search engine provides an excellent way and material for people to learn, research and master search technology, promotes the popularization and development of search technology, and enables more and more people to understand and promote the use of search technology. Using an open source search engine can greatly shorten the cycle of building search applications, and can create personalized search applications according to application requirements, and even build search engine systems that meet specific needs. The open source of search engines is a boon for both technicians and ordinary users.

Shay Banon has a vision to empower every developer in the world to use search as a foundation to simply solve their most complex use cases. Serving data in real time and at scale, Elastic's products have been downloaded more than 150 million times cumulatively and are used to build modern search, logging, security, metrics, and analytics applications.

Technology Boost
In today's world, the rapid changes in technology have intensified the process of market competitiveness. Enterprises are paying more and more attention to the enhancement of competitiveness brought about by technological innovation and the short-term and long-term market benefits created by it. core development strategy. The competition among enterprises is not only the competition of scale, but more importantly, the competition of technological innovation strength among enterprises.

When Ma Yun gave a speech at the Yunqi Conference, he talked about the importance of technology for the future, "We are all children in front of the future, and there are no experts in the future." He believes that companies that develop well in the future must be able to make the best use of Internet technology company of.

Any emerging technology must go through a spiral development trajectory, and must also conform to the development law of the technology life cycle, that is, from conception, bubble, burst, calm, maturity, application rise, and then rebirth and re-innovation. For enterprises, in terms of enterprise direction and R&D strategy, they must grasp and respect the development laws of the technology industry.

Shay Banon introduced the trends of several key technologies in Elasticsearch.

640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy

Shay Banon said that he and his team will continue to innovate on the road of technology, so that Elasticsearch can solve various problems for different users.

Elasticsearch and Alibaba Cloud cooperate to make great strides into the Chinese market
When talking about Elasticsearch's strategy to enter the Chinese market, Shay Banon said: "China is a growing market for us. In the past few years, we have seen the community of Elasticsearch expand to more than 5,000 developers. China is also In one of the world's largest markets, there are almost 190 million developers. I hope that these 190 million developers can use the open source Elasticsearch products and succeed. Today Elasticsearch chooses to cooperate with Alibaba Cloud and cooperate with Elasticsearch's real-time processing capabilities , powerful X-Pack functions, such as security, alerting and machine learning, together accelerate the pace of innovation of China's vast developer ecosystem, build, host and manage more different applications." In addition, Shay Banon believes that Elasticsearch will It will vigorously promote its commercial product X-Pack for the Chinese market, so that more and more people can understand and use it.

"Alibaba Cloud Elasticsearch" is now officially launched, and it can be added to the customer's cloud computing service with simple configuration. Combine the real-time search capability of Elasticsearch with customer applications, import data into Alibaba Cloud Elasticsearch with Logstash or Beats, use the Kibana dashboard to visualize real-time and historical data, and add a series of functions of X-Pack such as security, alerting, and monitoring , reporting, Graph analysis and machine learning, providing developers with a one-stop product experience.

Alibaba Cloud Elasticsearch's new service allows Alibaba's customers to use Elasticserach's powerful real-time search, collection and data analysis functions as they wish, and is a one-stop and leading solution.

In addition, Alibaba Cloud and Elasticsearch will focus on technological improvement to ensure that Alibaba Cloud Elasticsearch keeps pace with the times and has the latest functions. In the future, the log import function and other services will also be available.

Data Mining Advantages of Search Engines
The era of big data is also the era of information explosion. It is no longer important to have information, but how to quickly find the required information. Search engines have natural advantages in this regard, and data mining of search engines will produce more obvious results. Effect.

The improvement of many search technologies is inseparable from big data technology. In essence, a search engine is a typical big data application. At present, search has taken a big step forward in the field of big data, and people can search for the information they want in real time.

According to the latest database engine rankings, Elasticsearch, Solr and Splunk occupy the top three database search engines respectively.

640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=

From the trend point of view, Elasticsearch and Splunk have risen significantly, and Elasticsearch has shown a very strong momentum.

640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=

It has become a common practice to record application running logs in the production environment, but the logs need to be processed and analyzed to be meaningful. The emergence of third-party log management tools is aimed at solving this problem. The current representative log management tools are Splunk and Logstash (Note: Logstash is used to insert data into Elasticsearch and Kibana to visualize logs).

Shay Banon said that in the field of log analysis, Elasticsearch's biggest competitor is Splunk, choosing between paying for commercial software and free open source software. Elasticsearch is the most popular open source solution in the world, and it will use flexibility, real-time capabilities and Large-scale processing of large amounts of data, so if you ask developers in the mainland, most developers tend to Elastic Stack.

For example, companies like Netflix, Facebook, Microsoft, and LinkedIn choose to run large Elasticsearch clusters on their log infrastructure. In addition, the Elastic Stack can be used in different areas, such as fraud detection and domain-specific business analysis, which will allow Elastic to continue to expand.

Machine learning empowers users to solve complex problems
The development of cloud computing has made data collection, processing and analysis easier, and big data can exist in various data systems in all walks of life. Therefore, artificial intelligence has become a hot field.

Among them, machine learning is established based on search technology, and the massive data accumulation brought by search can build a set of data statistical analysis based on massive data, which can provide guidance and support for key decisions in some application scenarios .

Shay Banon emphasized the important value of machine learning in the field of data search: "In the future, instead of telling the data what we want, the data will actively tell us what is here. This is the power of machine learning.

little worry
Search engines know our travel routes, geographical locations, work information, daily behavior patterns and social circles. It knows our risk profile better than any insurance company or bank. With the rise of wearable smart devices, it may also know better than doctors. Know our own physical condition. In other words, search engines will come to know us better than we do.

This is the unique background of the information age. The pursuit of efficiency makes us inevitably enjoy the information services brought to us by services such as Internet search engines, and at the same time inevitably enjoy the distress of personal information leakage. The machine learning of search engines is bound to require more and more user information, which inherently conflicts with our right to privacy. Perhaps, we are already aware of this, but there is nothing we can do about it in the face of efficiency.

Advice to engineers:
The speed at which the computer world is changing is astounding. Programmers are considered to be the profession closest to the computer world, and almost all new technological products have to be coded by programmers.

Shay Banon advises programmers to constantly learn new skills and keep in mind the experience gained when using those skills in the past. Have passion and love this profession, always evaluate the software you write from the perspective of end users, instead of writing code in a closed space.

In addition, programmers must also be good at using tools, and choose the tools that suit themselves and project development during the development process. As the saying goes, if a worker wants to do a good job, he must first sharpen his tools.

Write at the end:
Shay Banon likes Jack Ma's words very much, "Help the young, help the weak, because a small sapling may grow into a towering tree. You plant the seeds in the brains of these young people, and when they grow up, can change the world.”

Help others, make others strong, you can be stronger. This is the meaning of life.