RABBITMQ

RABBITMQ: UNDERSTANDING MESSAGE BROKER

A message broker acts as an intermediary platform when it comes to processing communication between two applications. RabbitMQ is one such open-source enterprise messaging system modeled on the Advanced Message Queuing Protocol (AMQP) standard. In this blog post, I will describe a scenario that led us to get to the bottom of a development problem and understand the real need behind using such a service.

The Problem Explained – Real World Example

One of our new customers just started development using sprints, and the first story which we were delivering for this customer was on a registration module. In a nutshell, what we were doing was re-writing this module from scratch.

Our developers were putting in their best effort to finish this entire module in the given time-frame. The following change requests that came in did, however, slow us down and lead us eventually to RabbitMQ:

1. Welcome email message must be sent after each successful registration. Third-party SMTP server must be used to send the required messages.
2. Registration must be made possible through affiliate links and tracking must be made possible for those registrations through third-party APIs to track the performance of the affiliate registration process. Log audit information about user registration like time, location, etc. also needed to be gathered.

Once everything was done, tested and deployed, your code will look like:

public void RegisterUser(User user)
{
    //Register user
   userService.Register(user);
    //Send welcome email
   emailService.SendWelcomeEmail(user)
    //Call third party api for affiliate tracking
   someThirdPartyWrapper.log(user)
    //audit log
   auditLogger.log(user)
}

A new registration module went live and did work like a charm initially. One of the early problems we encountered was that the registration module showed an error once a user finished the registration process. Subsequent investigations revealed that the third-party SMTP APIs were down for some time. It was during this downtime that a user registered but never received confirmation emails.

Keeping the above in mind, we also had to take into account the following scenarios: – Third-party API stops working – Audit log table gets locked due to high traffic – User information is sent to other internal systems which are written in a different language – And many more such cases where direct communication is required between applications.

Can our code scale to meet new requirements or remove dependencies on third-party APIs?

We assumed that if our RegisterUser method will only register the user, other tasks can be performed by other applications without impacting the user registration process.

The code was changed to:

public void RegisterUser(User user)
{
   //Register user
   userService.Register(user);
   //Send message about user's registration
  var message = new Message()
   {
   //user's registration information
  }
  messagingService.SendMessage(message)
}

These messages can now be stored in a central place so that other applications can have quick access. Now email services, third-party APIs, and audit log service can run independently and perform their tasks without impacting the user registration module.

Even in case of any issues encountered in email or third-party services, our registration module will keep working, and once these services are up again they can continue their tasks without losing any data.

What is Messaging?

It is a way of exchanging messages from point A to point B or many points C. Messaging enables distributed communication that is loosely coupled. A message sender sends a message to a destination, and the message recipient can retrieve the message from the destination. However, the sender and the receiver do not have to be available at the same time in order to communicate. To understand better, email is a great example of messaging which enables people to communicate.

You may want to use messaging in the following scenarios:

• Send data to many applications without calling their API directly into your application
• Want to do things in certain order like a transactional system
• Monitor data feeds like the number of registrations in any application

The component which receives the message from the sender and recipient retrieves the message from a message broker or messaging middleware.

Message Broker/Middleware

According to Wikipedia:

“Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems. MOM allows application modules to be distributed over heterogeneous platforms and reduces the complexity of developing applications that span multiple operating systems and network protocols.”

Message brokers do many things such as:

• Decouple message publisher and consumer
• Store the messages
• Routing of messages
• Monitoring and management of messages

Such message broker services typically used to lack standards, and existing commercial implementations had proprietary implementation and API issues to look at. This was up until recently an issue which can now be addressed through AMQP. AMQP is an open standard application layer protocol for message-oriented middleware.

From the AMQP website:

“AMQP is an Open Standard for Messaging Middleware.

By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organizations in a timely manner.

AMQP enables complete interoperability for messaging middleware; both the networking protocol and the semantics of broker services are defined in AMQP.”

In null-set, AMQP defines:

• Where to send messages (Routing)
• How to get there (Delivery)
• What goes in must come out (Fidelity)

AMQP is a standard wire level protocol (communicating with a remote machine or getting data from point to point) and has many implementations.

RabbitMQ is one such open source message broker software that implements AMQP.

Hello RabbitMQ

RabbitMQ is a message broker that takes messages and sends them to other places in a pretty smart way. AMQP is the protocol that RabbitMQ implements. It is completely language-neutral and while using it you can write and read to them in any language just like you would while using TCP or HTTP.

Another great advantage is that RabbitMQ runs on all major operating systems and supports a large number of developer platforms like Java, .NET, Python, PHP, Erlang and many more.

The RabbitMQ server is written in the Erlang programming language and is built on the Open Telecom Platform (OTP) framework for clustering and failover.

As defined on the Erlang website:

“Erlang is a programming language used to build massively scalable soft real-time systems with requirements on high availability. Some of its uses are in telecoms, banking, e-commerce, computer telephony and instant messaging. Erlang’s runtime system has built-in support for concurrency, distribution and fault tolerance.”

“OTP is set of Erlang libraries and design principles providing middle-ware to develop these systems. It includes its own distributed database, applications to interface towards other languages, debugging and release handling tools.”

Example: WhatsApp uses Erlang to run messaging servers, achieving up to 2 million connected users per server.

RabbitMQ has the following advantages:

• Fast
• Polyglot
• Simple management
• No Erlang knowledge needed
• Great documentation and community

Installation Steps:

Erlang: Download and install Erlang (OTP R16B03-1 Windows 64bit Binary Release) from http://www.erlang.org/download.html

RabbitMQ: Download and install RabbitMQ server for Windows from https://www.rabbitmq.com/download.html

RabbitMQ will be installed as a Windows service

RabbitMQ also comes with web-based management plugin which is quite handy to manage RabbitMQ.

Steps to install the management plugin:

• Open command prompt
• Run the following command from /sbin

rabbitmq-plugins.bat enable rabbitmq_management

• Restart the RabbitMQ by following command

rabbitmq-service.bat stop rabbitmq-service.bat start

By default management plugin will run on http://localhost:15672/ Default user name and password is guest & guest

Basic operation

• To stop RabbitMQ along with Erlang

rabbitmqctl stop

• Stop/Start RabbitMQ only

rabbitmqctl stop_app rabbitmqctl start_app

Elements

Producer creates message and sends (publishes) into the message broker (RabbitMQ). A message must have two parts: a payload and a label. Payload is data and it can be anything from a simple JSON to MPEG-4 file. Label describes the payload and how RabbitMQ will determine who should get a copy of the message. The communication between publisher and RabbitMQ is one directional and fire and forget.

Consumer on the other hand attaches to the broker and subscribes to a queue to get the message.

Your app can be producer when it needs to send messages to other applications, or it can be consumer when it needs to receive the message.

Connect to RabbitMQ

Applications will connect to RabbitMQ by creating TCP connections and getting authenticated. Setting up and tearing down TCP session is a time-consuming process for OS. AMQP also uses what is called a channel which is nothing but a virtual connection inside a real TCP connection. The Publisher/Consumer apps use channel to issue AMQP command to the broker. A single TCP connection can be used to establish multiple communication paths between the application and the broker. Publisher writes to channel and consumer reads through the channel.

Exchanges, Queues and Binding

Exchange: Producer publishes messages in exchange. Producer never directly communicates through consumer applications.

Queue: Messages end up in the queue and are received by the consumer.

Binding: Rule to route the message into one or more queue. This is a relationship between exchange and a queue.

When producer publishes a message in exchange it contains two parts; a payload and a label. Label is called the routing key. Each queue is bound to a routing key or a pattern of routing keys. This routing key binding with queue are rules that allow the exchange to put messages into the queue.

The exchange receives messages from producer and from the other side it pushes them to the queues. The exchange must know exactly what to do with the messages it receives. Should it be appended to a particular queue? Should it be appended to many queues? Or should it get discarded? The rules for determining that are defined by the exchange type.

There are mainly four type of exchanges available: direct, topic, headers and fanout.

Fanout Exchange: Fanout exchange brodcasts all the messages it receives to all the queues it knows.

Direct exchange: As we have seen, Fanout exchange broadcasts all the messages to all queues. In direct exchange, message goes to the queues whose binding key exactly matches the routing key (label) of the message. In direct exchange it is also perfectly legal to bind multiple queues with the same binding key.

In the below example Q1 & Q2 are binded with the routing key “Orange,” Q3 with yellow, and Q4 with Orange, black, and green.

• Exchange receives message with “Orange” routing key and it will deliver message to Q1, Q2, and Q4
• Exchange receives message with “Yellow” routing key and it will deliver message to Q3.
• Exchange receives message with “Green” routing key and it will deliver to Q4.

Topic exchange: Direct exchange gives flexibility to bind routing key with queues, but still it lacks binding based upon pattern. In topic exchanges will route messages to one or many queues based on the pattern that was used to bind a queue to an exchange. Messages sent to a topic exchange can’t have an arbitrary routing_key – it must be a list of words, delimited by dots. A valid routing key examples are “stock.usd.nyse”, “nyse.vmw“, “quick.orange.rabbit“. Queues can bind with exchange bu using patterns like “.usd.” to get all the messages where “usd” is a middle part of the message.

In the above example:

1. Q1 will get all the messages which start with a log. E.g. log.debug, log.error, log.info, and log.warn etc.
2. Q2 will get message with routing key log.error.
3. Q3 will only get log.info message.
4. Q4 will get all the messages.

Summary

Messaging enables software applications to connect and scale. RabbitMQ is fast, reliable, and a flexible messaging solution and therefore my preferred choice for enterprise-level message broker service. More about rabbitmq please visit this page https://www.rabbitmq.com/features.html