Discovering AMQP and RabbitMQ

Message queuing is a one-way communication style that provides asynchronous interaction between systems. As this chapter continues to describe how message queues work, the benefits will become clear. Some background on the request-response message exchange pattern will shed light on how RabbitMQ works.

The request-response message exchange pattern

There are many types of message exchange patterns, but the request-response style is the most common. A system, acting as a client, interacts with another remote system, which is acting as a server. The client sends a request for data, and the server responds to the request, as shown in the following diagram:

The request-response style is used when the client must have an immediate response or wants the service to complete a task without delay, such as being placed on hold when calling a restaurant to reserve a table:

Whether it takes the form of a remote procedure call, a web service invocation, or consumption of a resource, the model is the same: one system sends a message to another and waits for the remote party to respond. Systems communicate with each other in a point-to-point manner, where events and processes occur simultaneously or have dependencies or events related to time; the interaction between the client and server is synchronous.

One on hand, this request-response style gives developers a simple programming model as everything happens procedurally. On the other hand, the tight coupling between both parties has a deep impact on the architecture of the whole system as it is hard to evolve, hard to scale, and hard to ship in independent releases.

Message queuing exchange pattern

Message queuing is a one-way style of interaction where one system asynchronously interacts with another system via messages, generally through a message broker. A requesting system in asynchronous communication mode does not wait for an answer or require return information; it continues processing no matter what. The most common example of such an interaction is an email. The point is, asynchronous communication does not involve waiting for a response in order to continue processing. In fact, there may be no response or it may take some time for a response to be sent. Whatever the case, the system does not rely on a response to continue the process.

Messages flow in one direction, from the publisher to the broker and finally to the consumer:

Systems and applications play both the role of message publishers (producers) and message consumers (subscribers). A publisher publishes a message to a broker that they rely on to deliver the data to the intended consumer. If a response is required, it will arrive at some point in time through the same mechanism, but reversed (the consumer and producer roles will be swapped).

A loosely coupled architecture

One big advantage of the messaging queuing approach is that systems become loosely coupled with each other. They do not need to know the location of other nodes on the network; a mere name is enough to reach them. Systems can, therefore, be evolved in an independent manner with no impact on each other as the reliability of message delivery is entrusted to a broker.

The following diagram illustrates a loose coupling between the publisher and the consumer:

If one system is down for any reason, the other part of the system can still operate, and messages that are supposed to be sent between them wait in the queue.

The architecture represented via message queuing allows for the following:

• The publishers or consumers can be updated one by one, without them impacting each other.
• The performance of each side leaves the other side unaffected.
• The publishers or consumers are allowed to fail without impacting each other.
• The number of instances of publishers and consumers to scale and to accommodate their workload in complete independence.
• Technology mixing between consumer and publishers.

The main downside of this approach is that programmers cannot rely on the mental model of procedural programming where events occur one after another. In messaging, things happen over time. Systems must be programmed to deal with this.

If all this is a little blurry, use the example of a well-known protocol, Simple Mail Transfer Protocol (SMTP). In this protocol, emails are published (sent) to an SMTP server. This initial server then stores and forwards the email to the next SMTP server, and so on until the recipient email server is reached. At this point, the message is queued in an inbox, waiting to be picked up by the consumer (typically, via POP3 or IMAP). With SMTP, the publisher has no idea when the email will be delivered or whether it will eventually be delivered at all. In the case of a delivery failure, the publisher is notified of issues later down the line.

The only sure fact is that the broker has successfully accepted the message that was initially sent. This entire process can be seen in the following diagram:

Furthermore, if a response is needed, it will arrive asynchronously using the same delivery mechanism but with the publisher and consumer roles reversed.

With these fundamental notions established, it is the perfect time to delve into the messaging protocol that will be used in this book, which is AMQP.

Meet AMQP

AMQP is an open standard protocol that defines how a system can exchange messages. The protocol defines a set of rules that needs to be followed by the systems that are going to communicate with each other. In addition to defining the interaction that happens between a consumer/producer and a broker, it also defines the representation of the messages and commands being exchanged. AMQP is truly interoperable as it specifies the wire format for messages, leaving nothing open to interpretation by a particular vendor or hosting platform. Since it is open source, the AMQP community is prolific and has broker and client implementations in a wide range of languages.

RabbitMQ is built upon the AMQP 0-9-1 specification, but plugins are available that support AMQP 1.0.

The following is a list of the core concepts of AMQP, which will be explained in detail in upcoming chapters:

• Broker or message broker: A broker is a piece of software that receives messages from one application or service, and delivers them to another application, service, or broker.
• Virtual host, vhost: A vhost exists within the broker. It's a way to separate applications that are using the same RabbitMQ instance, similar to a logical container inside a broker; for example, separating working environments into development on one vhost and staging on another, keeping them within the same broker instead of setting up multiple brokers. Users, exchanges, queues, and so on are isolated on one specific vhost. A user connected to a particular vhost cannot access any resources (queue, exchange, and so on) from another vhost. Users can have different access privileges to different vhosts.
• Connection: Physical network (TCP) connection between the application (publisher/consumer) and a broker. When the client disconnects or a system failure occurs, the connection is closed.
• Channel: A channel is a virtual connection inside a connection. It reuses a connection, forgoing the need to reauthorize and open a new TCP stream. When messages are published or consumed, it is done over a channel. Many channels can be established within a single connection.
• Exchange: The exchange entity is in charge of applying routing rules for messages, making sure that messages are reaching their final destination. In other words, the exchange ensures that the received message ends up in the correct queues. Which queue the message ends up in depends on the rules defined by the exchange type. A queue needs to be bound to at least one exchange to be able to receive messages. Routing rules include direct (point-to-point), topic (publish-subscribe), fanout (multicast), and header exchanges.
• Queue: A queue is a sequence of items; in this case, messages. The queue exists within the broker.
• Binding: A binding is a virtual link between an exchange and a queue within the broker. It enables messages to flow from an exchange to a queue.

The following diagram illustrates an overview of some of the concepts in AMQP:

The open source broker shown in detailin this book has been built from the ground up to support AMQP, but many other protocols are also supported by RabbitMQ, such as MQTT, HTTP, and STOMP.

Now, it's time to turn the focus to RabbitMQ.

The RabbitMQ broker

RabbitMQ is an Erlang implementation of an AMQP broker. It implements Version 0-9-1 of AMQP with custom extensions, as allowed by the protocol. Erlang has been chosen because of its intrinsic support for building highly reliable and distributed applications. Indeed, Erlang is used to run telecommunication switches in several large telecommunication systems, and a total system's availability of nine nines has been reported (that's only 32 milliseconds of downtime per year). Erlang is able to run on any operating system.

For data persistence, RabbitMQ relies on Mnesia, the in-memory/file-persisted embedded database of Erlang. Mnesia stores information about users, exchanges, queues, bindings, and so on. The queue index stores message positions and information on whether a message has been delivered or not. Messages are stored either in the queue index or in the message store, a key-value store shared among all queues.

For clustering, it mainly relies on Erlang's ingrained clustering abilities. RabbitMQ can easily be extended with the addition of plugins. For example, a web-based administration console can be deployed on RabbitMQ thanks to this mechanism.

RabbitMQ can be set up on a single, standalone instance, or as a cluster on multiple servers:

RabbitMQ brokers can be connected together using different techniques, such as federation and shovels, in order to form messaging topologies with smart message routing across brokers and the capacity to span multiple data centers.

The following screenshot shows federation between RabbitMQ brokers located in different places around the world:

So, when or where is RabbitMQ used? The next section describes some common use cases for RabbitMQ.