×
Community Blog Definition and Benefits of Message Queuing Notification Service

Definition and Benefits of Message Queuing Notification Service

Message Service is a message queuing and notification service that facilitates smooth transfer of messages between applications.

Message Queue vs. Message Service, which is better?

You might have already come across the term message queues, if not I suggest you take a quick read over how they can greatly improve application reliability and scalability in a previous post Improving Reliability with Alibaba Cloud Message Service.

There are very many message queues out there, from RocketMQ, RabbitMQ, Apache Kafka, ZeroMQ, MosquitoMQ, and many more. Many Cloud providers also offer managed message queues as a service and Alibaba Cloud has two of them namely the Message Queue and the Message Service.

From their online descriptions.

Message Queue

Message Queue is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.

Message Service

Message Service is a message queuing and notification service that facilitates smooth transfer of messages between applications

They almost sound like a paraphrase of each other, don't they? Well, as we'll see shortly, it's mostly because they are very similar products. To understand this, let's briefly look at the architecture of a message queue.

message queue

At the very heart of both the Message Queue and Message Service is the design above. They both use and can execute this model with similar efficiency. However, the major difference I've found is in how the message can be sent by the producer and consumed by the receiver.

The Architectural Difference
The older, battled tested Message Queue has been integrated into the infrastructure of Alibaba Group for over 11 years and servicing Alibaba's 11-11 e-commerce sales event for over nine years. In September 2017, RocketMQ, the kernel engine of Alibaba Cloud Message Queue, became an Apache top-level project.

With those many years of development, the Message Queue had more community experience and expertise behind it, but it also comes with more legacy methods and restrictions. For example, the Message Queue can only speak over TCP or MQTT protocols. This means you'll most likely require specific extensions enabled on your server to access these protocols. It also means you will most likely require an officially supported SDK to use it. Currently, Alibaba Cloud supports Python, PHP, .NET and Java programming languages. This means that both your producers and consumers must be written in one of those languages if you want official support.

The newer Message Service product differs from the Message Queue in that it can speak HTTP. This is a crucial differentiator implying that, so long as you can send a HTTP request, your application can use the Message Service. This opens the door to almost each and every programming language out there. There are official SDKs of course, but these are mostly to add synthetic sugar to make using it easier. Another decisive differentiator is that the Message Service has the ability to push out message to consumers instead of waiting for them to pull the messages out of the queue. The consumer can be any HTTP server, a browser via websockets, a mobile phone, an email address or even another Message Queue service!

What this means is that the newer Message Service removes the restrictions on what programming languages or frameworks you can use. Which can be vital for the competitiveness of any application, for example Uber rewrote their Geolocation component from Python to Go while One Signal rewrote their entire application in Rust. Both saw great gains, companies in similar shoes would find that difficult or impossible to do if those components depended on the Message Queue. On the contrary, it would be a rather trivial affair if they were using the Message Service.

The Performance Difference

Both the Message Queue and the newer Message Service are blazing fast and globally scalable across the many Alibaba Cloud data centers. They both provide 99.9999999% reliability and ensure successful message delivery within the valid period.

However, for pure performance comparisons, the Message Queue probably wins here, due to its maturity and that fact that its protocols operate at a lower level. To quote Alibaba Cloud, "From years of experience servicing Alibaba's annual 11-11 e-commerce event, the technical team at Alibaba Cloud have developed methods to resolve slow request response time jitter problems caused by system overload due to super-high concurrency. Among the flow of the trillion messages sent during the 11-11 event held in 2017, the recorded latency of 99.6% of message write requests was less than 1ms, and less than 10ms for the recorded latency of 99.996% message write requests."

That said, it should be noted that by using the Message Service, application developers can now be free to rewrite critical components in the language most suited to it as with the examples above. Therefore, while the Message Queue is probably a slightly faster queuing system, the Message Service has greater potential to help develop faster applications.

You can refer to Alibaba Cloud Message Queue vs. Message Service and get the following conclusion.

Related Blogs

Decoupling Application Components Using Alibaba Cloud Message Notification Service

In this tutorial, we will investigate how you can achieve more resilience by decoupling using the queues and topics of Alibaba Cloud's Message Notification Service.

When you have an application with lots of moving parts, ensuring the availability of your services can be a real challenge. If one of the components goes down, a tightly coupled architecture might cause the error to snowball. This can quickly take down your full application and sometimes requires several hours to fix. Decoupling the components in your application can help prevent the snowball effect and increase availability. In this post, we will investigate how you can achieve more resilience by using the queues and topics of the Alibaba Cloud Message Notification Service.

Unravelling a Coupled Architecture
Let us say you run a webshop that sells shoes. Logically, you have an order processing component that receives incoming orders and checks the stock levels. If the shoes are in stock, a payment is triggered. Finally, a delivery is scheduled with a third party logistic service provider. As the project had a tight schedule, the team decided to put the workflow in a single component that handles these tasks from top to bottom. The application went live and people started ordering. On Friday afternoon that week, you get a phone call: the servers of the logistic service provider are down. People have ordered shoes and paid for them, but the application was unable to schedule any deliveries. This means the customers will never get their new shoes. There are two ways to recover from this. You can either undo the payment and fail the order, or wait until the logistic service provider resolves the issues and try to manually fabricate the needed messages. All in all, you are in a pickle.

How could these issues have been prevented? By decoupling the coupled architecture! In a more decoupled approach, the workflow will be processed by multiple individual components. Every component receives a message, does its work and triggers the next step in the workflow. If a step fails, that component can retry the processing for as long as it sees fit. The failure will not influence any other steps in the process, although it might take a little bit longer before orders can be finalized. But a customer would probably be happy to wait for a few minutes, rather than not being able to order at all.

Decoupling an application using messages like this is usually done using queues and topics, that we will look at next.

Queues and Topics

When you know that only a single logical component should receive a message, you can use a queue. A queue is a stack of messages that a component can receive one by one. The receiving component is sometimes called a consumer. If a message was processed successfully, it is removed from the queue. If something unexpected happens during the processing of a message, it is put back on the queue. The message will then be received again at a later point in time. Normally, consumers can decide for themselves how many messages they will process concurrently, so the throughput is at an acceptable rate. It is also very common to see multiple consumers for any given queue.

Queues are ideal for batch calculations, generating invoices or images, sending emails or doing any kind of work that takes a fair amount of time. The beauty of it is that the component that sends the message, the producer, can keep on sending messages even if the consumer is down. The queue will fill up with messages that will be processed when the consumer comes up again. In this way, not a single message gets lost if an outage occurs. That makes an application more resilient and ensures a good user experience.

Topics can be used if multiple components should receive a message. These consumers are called subscribers in the context of a topic. The messages on a topic are often events triggered by the application. An example might be that an order was processed. Multiple subscribers might be interested in that event, like the shipment service and the payment service. Theoretically, a topic differs from a queue because the publishing side does not know anything about the consumers. It has no idea how many components will pick up the message, nor does it know what subscribers might want to do with it. This is a very powerful concept and sometimes referred to as a Publish-Subscribe pattern.

The best thing about queues and topics is that you can combine them to create an extremely flexible setup. When adding subscribers to a topic, you can specify the transport protocol of the message. An HTTP call can be made whenever a message is published on the topic, but you could also have the message delivered into a queue. That way, the fault tolerance of queues is combined with the fan-out kind of architecture topics provide.

Implementation of Message Push and Storage Architectures of Modern IM Systems

This article describes the architecture of an IM system and introduces a Table Store-based message synchronization and storage system architecture.

Preface

IM is the abbreviation of instant messaging. In the highly information-based mobile Internet era, IM products have become a must-have item in our life. The most well-known IM products in China are DingTalk, WeChat, and QQ. Among them, WeChat has already developed into an ecosystem, but its core function is still an IM. IM is an inseparable module of some applications that do not take IM as the core business. The most typical ones are online games and social networking applications. The IM feature is essential to applications with social attributes.

The IM system came into being in the early days of the Internet. Its basic technology architecture has been updated many times during the last dozen of years—from the early CS and P2P architectures to the current distributed systems in the backend. It involves all aspects of technologies, such as mobile clients, network, security, and storage. The daily active users that an IM product serves have increased from a small number in the early days to up to 900 million as announced by WeChat recently.

The core of an IM system is the messaging system, and the core function of the messaging system is the synchronization and storage of messages:

  1. Message synchronization: Transmitting integrate messages from the sender to the recipient quickly. The most important metrics of a message synchronization system are the instantaneity and integrity of transmitted messages, and the size of messages that can be supported. In terms of functionality, an IM system must at least support online and offline message push. Advanced IM systems also support "multi-terminal synchronization".
  2. Message storage: The persistent storage of messages. This does not mean the local storage of messages at the client-side, but the storage on the cloud. This is the so-called "message roaming" function. The advantage of "message roaming" is that you can log on to your account at any terminals to view all historical messages. This is also one of the unique features of the advanced IM system.

The article mainly describes the messaging system architecture in the IM system. We will introduce the implementation of a Table Store-based message synchronization and storage system architecture. It supports advanced features of the messaging system, such as "multi-end synchronization" and "message roaming". In terms of performance and scale, it supports full message storage on the cloud, and message synchronization with millions of TPS and millisecond delays.

Architecture Design

This section mainly describes the architecture design of Table Store-based modern IM systems. Before describing the architecture design in detail, we will introduce the timeline logic model to abstract and simplify the understanding of the IM synchronization and storage models. After understanding the timeline model, we will talk about the modeling methods for the synchronization and storage of messages based on the timeline model. We also have to make technical trade-offs in various aspects when implementing message synchronization and storage. For example, we have to compare and choose common pull and push modes for message synchronization, and choose the underlying database based on the characteristics of the timeline model.

Comparison between Conventional and Modern Architectures

The above diagram is a simple comparison between the conventional and modern architectures of a messaging system.

Under a conventional architecture, messages are synchronized first before they are stored. For online users, messages are directly synchronized to online recipients in real time. After a message is successfully synchronized, it will not be persisted. For offline users or messages that cannot be synchronized in real time, the messages will be persisted to an offline database. The recipient may pull all unread messages from the offline database after reconnecting to the Internet. A message will be deleted from the offline database after it is successfully synchronized to the recipient. Main tasks of the conventional messaging system server is to maintain the connection between the sender and the recipient, and to provide online message synchronization and offline message caching. This design ensures messages can be transmitted from the sender to the recipient under all circumstances. The server does not persist the message, so it does not support message roaming.

Under the modern architecture, messages are stored first and then synchronized. The advantage of storing messages before synchronization is that when a recipient receives a message, the message must have already been saved on the cloud. In addition, the messages are saved in two databases—the message storage database and the message synchronization database. The message storage database stores messages of all sessions to support message roaming. The message synchronization database is mainly used for multiple-terminal synchronization of the recipient. After a message is sent by the sender, it is forwarded by the server. The server subsequently saves the message to the message storage database and the message synchronization database. After a message is persisted, if the recipient is online, the message is pushed to recipient directly. However, online push is not the only option. It's just the preferred one. For messages that failed to be pushed online, or when the recipient is offline, there is another unified message synchronization method. The recipient will actively pull all unsynchronized messages from the server. However, the time of synchronization, and from which terminals the recipient may send the message synchronization requests are unknown to the server. So, the server must save all messages that need to be synchronized to the recipient. This is what the message synchronization database is designed for. Users of an IM product may have message roaming needs when they use new devices. The message storage database is designed to meet such needs. From the message storage database, you can pull all historical messages for any session.

Using Message Service to Schedule Jobs in Batch Compute

In this article, we will use a queue based model of Alibaba Cloud's Message Service to receive message and automatically submit jobs in Batch Compute.

Overview

Batch Compute is a distributed cloud service suitable for processing massive volumes of data concurrently. The system automates resource management, job scheduling, and data loading and supports billing on a Pay-As-You-Go basis. This tutorial demonstrates how you can submit or schedule jobs automatically simply by sending message from Alibaba Cloud Message Service.

Alibaba Cloud Message Service is a high performance, reliable, safe, extensible distributed message and notification service that supports massive messages, concurrent operations. It help facilitate message transfer between applications and system decoupling.

We will use Queue based model of messaging service to receive message and automatically submit job in batch compute.

There are various real world applications of this idea. For example,

  1. You can run various system maintenance jobs depending on the alerts generated from system health monitoring applications
  2. Trigger data load job as soon as file arrives from external system. File watcher application can send file arrival message to Message Service queue and consumer of the message will submit the file load job in batch compute.
  3. Analysis jobs of call-center calls can be triggered as soon as the recorded audio file is uploaded into cloud storage. File watcher program can send job-triggering message into the message queue for analysis job to start.

Architecture

Architecture

In this tutorial, we will simulate the message alert using send message function in Message Service web console. From the console, job names will be passed as message text or body to consumer script. Consumer is a custom python script, which will monitor messages from Message Queue in certain frequencies and depending on the message text received, consumer will submit the job corresponds to the job name mentioned in each message.

Related Courses

An Introduction of Alibaba Cloud MQ

This course is designed to help businesses that want to use Message Queue services, as well as cloud computing engineers and enthusiasts who want to understand and learn Message Queue technology. By learning this course, you can fully understand what is Message Queue and how to use Alibaba Cloud MQ, which can provide reference for practical application.

How to Manage Ultra Large Application with EDAS

Manage Ultra-Large-Scale application has many challenges, such as application management and monitoring, high availability, performance optimization, and system expansion .etc. This course will show you how to manage the large applications on Alibaba Cloud using EDAS which is the core product supports 99 percent of Alibaba Cloud's large-scale application systems.

Related Documentation

What is Message Queue for Apache RocketMQ?

Message Queue for Apache RocketMQ provides asynchronous decoupling and load shifting for distributed application systems, and supports features for Internet applications, including massive message accumulation, high throughput, and reliable retry.

Network access

In addition to internet access, Message Queue for Apache RocketMQ supports private networks for application infrastructure, namely Virtual Private Cloud (VPC) access. You have full control over your VPC, which you can define and customize by specifying the IP address range and configuring route tables and network gateways. You can also launch Alibaba Cloud resources such as Elastic Compute Service (ECS), Relational Database Service (RDS), and Server Load Balancer (SLB) in your own VPC.

Scenarios

  1. Load shifting
    Large activities, such as flash sales, red envelope snatching, and enterprise success, may cause high traffic pulses, and the system may become overloaded or even crash due to lack of proper protection, or the user experience may be affected because too many requests fail due to too many limits. Message Queue for Apache RocketMQ provides the load shifting feature to solve this problem.
  2. Asynchronous decoupling
    As the core system of Taobao/Tmall primary sites, the transaction system can attract the attention of hundreds of downstream business systems, including logistics, shopping cart, credits, and stream computing analysis when each transaction order is created. The overall business system is large and complex. Message Queue for Apache RocketMQ supports asynchronous communication and application decoupling, ensuring the continuity of services on the primary site.
  3. Sending and subscription of ordered messages
    Several scenarios need to ensure the sequence in daily life, such as the time-first principle of securities trading, order creation, payment, refund, and other processes in the trading system, as well as the handling of boarding messages of passengers on flights. Ordered messages in Message Queue for Apache RocketMQ are sent and received in the first-in-first-out (FIFO) order.
  4. Consistency of distributed transactions
    Final data consistency must be ensured in scenarios such as transaction systems and payment envelopes. Message Queue for Apache RocketMQ distributed transactions can implement decoupling between systems and ensure final data consistency.
  5. Big data analysis
    Data creates values during movement. Traditionally, data analysis is mostly based on the batch computing model and cannot be performed in real time. Message Queue for Apache RocketMQ together with Stream Compute of Alibaba Cloud allows you to conveniently analyze your service data in real time.
  6. Distributed cache synchronization
    During the Tmall Double 11 Shopping Festival, the changes to the commodity prices in different activities need to be perceived in real time. A large number of concurrent accesses to the database results in slow page response. The centralized cache restricts the traffic for commodity changes due to bandwidth bottlenecks. To solve this problem, Message Queue for Apache RocketMQ provides a distributed cache to notify commodity data changes in real time.

Message query - Message Queue

In case of message consumption problems, you can query specific message content for troubleshooting. Message Queue for Apache RocketMQ provides you with three methods of querying messages: by message ID, by message key, and by topic.

Query methods

Messages are stored in Message Queue for Apache RocketMQ for three days by default, and we recommend that you do not modify the value. That is, only the messages that are generated within three days before the current query time can be queried. For example, if the current time is 15:09:48 on June 10, 2019, messages that were generated under a topic as early as 15:09:48 on June 7, 2019 can be queried.

The following table compares the characteristics of the three methods.

Related Products

AlibabaMQ for Apache RocketMQ

AlibabaMQ for Apache RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.

Message Service

A message queuing and notification service that facilitates smooth transfer of messages between applications

0 0 0
Share on

Alibaba Clouder

2,183 posts | 504 followers

You may also like

Comments