How to use RabbitMQ with Rails
Service-oriented architecture is used everywhere and those services need to communicate with each other, a way to do that is to use messages and a message bus to handle them, this allows the decoupling of the system with asynchronous communication, and the only thing in common between services is the structure of the messages. One way to implement this architecture pattern is by using RabbitMQ, a top-rated message broker with a ton of functionality and support. In this article, we are going to take a look at what is RabbitMQ, its components, functionality, and how to use it in a Rails application.
Let’s start with some basic definitions. RabbitMQ is a message broker that uses the messaging protocol AMQP 0–9–1, the main idea is that RabbitMQ receives messages from producers and routes them to one or more consumers that use the information inside those messages to perform actions on their end. To achieve this the producers create a connection with RabbitMQ and inside that connection, they also create a channel that is used to send messages to RabbitMQ to be handled. On the consumer side, they also create connections and channels to receive those messages from RabbitMQ when certain criteria are met. This functionality comes out of the box with RabbitMQ but can be modified and expanded with plugins that allow it for example to be able to use AMQP 1.0 as well.
Let’s continue with some crucial elements of RabbitMQ, the first and most important one is the message. A message is a string with relevant data to the applications that are part of the system. The data it conveys can be anything, from a command or a query to just information about an event that occurred somewhere in the system. A message is made of several attributes (some are required and some are optional) that are going to determine how it is handled by RabbitMQ, these are the attributes:
- Headers: These are a collection of key-value pairs that can be used for routing or sending additional information like who is the original publisher of a message.
- Routing Key: This is the most used attribute to route a message, it is a string that contains one or more words that are going to be used to create the routing logic inside RabbitMQ.
- Payload: This is the actual data that an application wants to share with other applications in the system.
- Publishing Timestamp: It is an optional attribute set by the publisher to indicate when it was sent to RabbitMQ.
- Expiration: This is set by the publisher to indicate the lifetime of the message in the queue, after this time lapses the message is considered dead and is undeliverable. This time is expressed in milliseconds.
- Delivery Mode: This attribute tells RabbitMQ how to handle the message. Persistent messages are written to disk and if the server restarts they are reloaded. Transient messages are lost in a server restart.
- Priority: Indicates how important the message is for the system, the publisher sets this on the message and it is an integer between 0 and 255, with 0 as default.
- Message ID: This is an optional attribute set by the publisher to track the message.
- Correlation ID: It is an optional attribute that is set by the publisher to match a request to a response when using a pattern called request-reply or a remote procedure call (RPC).
- Reply To: When using the beforementioned request-reply pattern, it is used to indicate which queue the application has to send the reply message to.
That was a lot and we are not even halfway through but hang in there it gets more interesting. Another part of the RabbitMQ system is the producers (publishers), which are applications that send messages to RabbitMQ to be distributed to the interested apps based on the routing logic implemented by using the message attributes like the routing key or the headers.
On the other side are the consumers, which are applications listening to messages they are interested in (subscribed) to process them.
Now that we have some basic knowledge let’s dive into another important component of the RabbitMQ system, the queue. A queue is a place with a unique name where the messages are stored, managed, and delivered to the consumers. A messaging system has many queues depending on the number of consumers and the complexity of the system. Same as the messages, queues also have many attributes:
- Name: It is a string that identifies the queue, it can have a maximum of 255 characters and has to be unique. Many organizations already have a standard to name the queues.
- Durable: This attribute tells RabbitMQ if the queue can survive a server restart.
- Auto Delete: It tells RabbitMQ to delete the queue if no one is subscribed to it.
- Exclusive: It is used by one connection and it gets deleted when that connection is closed.
- Max Length: Indicates the maximum number of messages that the queue can hold
- Max Priority: It is the maximum number for the priority that the queue supports.
- Message TTL: It is the lifetime for each message in the queue. If the queue and the message both have that value set, RabbitMQ picks the lowest one.
- Dead Letter Exchange: It is the name of the exchange that all the dropped or expired messages will be sent automatically.
- Binding Configurations: These are logic rules that create associations between queues and exchanges, for a queue to be able to receive messages it has to be bound to an exchange.
Let’s continue with the system components definitions and dive into the last one. An Exchange is a place where messages enter the system and are distributed to bonded queues according to the implemented routing logic, for this reason is also known as a routing element. Like the other components of the system, it has several important attributes:
- Name: A string identifying the exchange
- Type: Describes how the exchange will route the messages and can be fanout, direct, topic, and headers. Later we will touch on how each of these types works.
- Durable: Same as in the queue, a durable exchange survives a server restart.
- Auto Delete: It tells RabbitMQ that the exchange will be deleted when no more queues are bound to it.
- Internal: If this attribute is set to true the exchange can only receive messages from other exchanges.
- Alternate Exchange: It indicates the name of the exchange where unroutable messages will be sent.
- X-Arguments: These are other named headers that can be provided to the exchange to have additional information, these headers always start with “x-”
We have mentioned a concept that is very important to the messaging system and now it is time to have more information on it. A binding defines the relationship (routing rules) between the exchange and the queue, taking into account that a queue can be bound to one or more exchanges. This relationship can be defined in two ways, through the routing key or the headers, and the exchange is going to use it as a filter to select the queue where the message should go, with an exchange checking the routing information of a message and comparing it to the routing information of a binding and then selecting the queues that match that routing information. This is the secret sauce of the system and is where its success lies, bad bindings will create bad and inefficient routing so pay attention to them.
Now for selecting the right type of exchange for your system needs let’s take a look at them:
- Fanout: It is the simplest exchange and it sends the incoming message to all the bounded queues without checking any routing information. This is used when several users need to get the same message like for a notification of some sort.
- Direct: Uses the whole routing key to select the bounded queue that matches it exactly. This is used for targeted messages for specific users.
- Topic: This kind of exchange takes sections of the routing key, usually separated by dots, and selects the queues based on the rules that can contain wildcards to set its logic, for this the wildcard * means one or more words have to be present and for the wildcard # means zero or more words have to be present from that point on.
- Headers: As the name suggests it uses the headers instead of the routing key for the distribution logic. The rules use the x-match pseudo-header to match all or any of the provided headers to make the binding.
There is another type of exchange that RabbitMQ creates by default and every queue that is created is bound to until any other binding is explicitly defined between that queue and an exchange, and it also handles the messages that are not referencing an exchange directly. It has no name but it is known as the default exchange and its type is Direct.
There is also the possibility of doing bindings between exchanges, these bindings are configured the same way as queues and they help a different exchange to route the message to the actual exchange that has the needed queues bound to it.
The final piece for exchange configuration is something called alternate exchange, which when defined will handle all the messages that could not be routed by that exchange. This configuration is set by the key alternate-exchange when creating or editing any exchange and can be bound to any other existing exchange. It is worth noting that the preferred type for an alternate exchange is fanout so it doesn’t filter any messages and send them to all its queues.
From the consumer’s perspective, there are two ways of getting the messages into your application. The push method is when the consumer subscribes to a queue and if there are any new messages in it they are automatically sent to the consumers, this is the recommended way of consuming messages from a queue. On the other hand, the pull method is when a consumer checks the queue for messages in intervals (polling) and if there are any new messages they are manually pulled from the queue by the consumer, this method is not recommended because you are going to have many wasted processing cycles where there are no new messages, but it can be useful when the consumer doesn’t have a live connection with the broker.
Now that we have seen the main elements of a RabbitMQ messaging system and how they work is time to start talking about some architecture patterns that they can be used for, here are some of the most common ones:
- Work Queues: This pattern is used to distribute tasks of the same nature between multiple workers for processing. It is very useful for tasks that are independent of each other like sending emails, generating a pdf, or processing an image.
- Publish-Subscribe: This is the most used pattern for this kind of system, and it is used when the same message has to be delivered to all the subscribers, and this is achieved by having a queue for each subscriber. This si commonly used to send notifications to users.
- Request-Reply: This pattern is used when the publisher of a message that can contain a query or a command needs to get the response of the operation performed. It is implemented using at least two queues, one for requesting and one for responding, and the messages for the request use the reply-to attribute to indicate which queue the publisher will be expecting the response. It is used mostly for remote procedure calls (RPC).
- Priority Queues: As the name says, this pattern handles messages based on their priority attribute, based on the assumption that some messages are urgent and need immediate processing and others are not that important and can wait to be processed. An example of this pattern is when you have transaction and promotional data going in the same message bus, the transaction data has to be processed immediately while the promotional data can wait, so when a transaction message enters the system it jumps to the front of the queue to be processed faster.
We have been exposed to a lot of needed theoretical concepts for RabbitMQ and message buses, and now is the time to put it all in an application that helps to put them in a practical perspective. At the moment of the writing of this article, I have a GitHub repo that has all the code for the publishing part and I expect to add the consumer side by the time you are reading this, that code is the one that I am going to be using in the rest of the article. For this practical application, I will use Rails, but there is a problem, there is no RabbitMQ-supported gem and the gems that have RabbitMQ on the name are outdated or not supported anymore, so I’m going to use a gem called Bunny that is very good in handling RabbitMQ operations and it is still supported.
To create the system we are going to need three main parts: the RabbitMQ server, the publisher application, and the consumer application. For all of them, we are going to use Docker to make our lives a lot easier when it comes to deploying and testing the applications.
The RabbitMQ server is the easier part because we already have a ready-to-go image in Dockerhub that we can pull and run, it is important that for a better experience, I recommend using the image that has the admin web interface enabled, but if you like a challenge you can use the other one. The admin interface uses the URL http://localhost:15672/ for accessing it and uses the default username guest and password guest. When you log in, it will have a default host(/) and you will be able to create the exchanges and the queues from it.
In order to run the publisher and consumer applications we will have to create a Docker file that will help with the simplification of the environment for them. For this, I have the following Docker file.
And for running everything together use the following docker compose file.
Now that we have everything running we can start doing some publishing to the messaging system. The first exchange that we are going to publish is a fanout exchange, remember that for this type of exchange, we only need the message, so we only have to define the connection, create the channel, and connect to the existing exchange, note that the name of the exchange has to already exist in RabbitMQ, if it doesn’t exist the code will create it. Then we just create the message and publish it to the connected exchange. It is as simple as that.
For the direct and topic exchanges the procedure is very similar to the fanout, the only thing that changes is that now we are going to be sending a routing key when we publish the messages, so pay attention to the code to see what kind of routing key is being used for each exchange.
Now for our final type of exchange again the code is very similar but instead of setting the routing key when sending the messages we are going to set the headers instead as shown in the following image.
This is all that I had to share about RabbitMQ. Remember, I owe you the consumer part but hopefully, you can find it in the GitHub repo by the time you are reading this. Thank you for reading this article and I hope you found it helpful. If you liked it give it a clap, check my other articles on various topics about development, and consider subscribing so you can get my new articles when they get published.
