Flux7 Flux7
  • Digital Innovation
      • Digital Innovation
      Enable Software Innovation
      • CI/CD: Accelerate Deployments Through Pipelines
      • Containers Infrastructure:Improve Agility with Containers
      • Build:Custom Toolchain Deployment
      • Microservices: Speed Application Development
      • HPC:Product Design & Simulation
      • Renovate:Application Migration to the Cloud
      • Serverless:Innovate at the Speed of the Market
      Scale Enterprise DevOps
      • AWS DevOps Consulting:  Refactor Large Quantities of Apps to AWS
      • Cloud-Native Architectures: Accelerate Business with Cloud-Native Services
  • Operational Excellence
      • Services
      Cloud Implementation Services
      • Cloud Migration Services: Streamline Your Cloud Migration
      • Build Cloud Foundations: Accelerate Adoption with Strong Cloud Foundations
      • ISV Workloads on Cloud: Defined Solutions and Proven IP
      • High-Performance Computing Services: Exploit Cloud Elasticity and Scalability
      Enforce Security and Compliance
      • Automate Compliance: Meet Corporate and Legal Requirements
      • Web Application Firewall:Protect Vital Data and Functions

      Our landing zones on AWS emphasizes training, documentation, and resources to help teams new to AWS get the skills they need for long-term business agility.

      Gain a Landing Zone That Fits Your Needs Today

  • Industries
      • Industries
       
      • Energy:Cloud Solutions for the Energy Industry
      • Finance: Secure Infrastructure for Improved Customer Service and Analytics
      • Healthcare & Life Sciences: Meet Security & Compliance Requirements
      • Hospitality: Increase Customer Acquisition
       
      • Manufacturing: Innovation with Digital Transformation
      • Retail:Grow Customer Loyalty and Lifetime Value
      • Semiconductor: Legacy Modernization Services
      • Software:Grow Developer Agility and Application Reliability
       

      Read our industry success stories and the benefits our customers saw

      Read the Case Studies

  • Tech
      • Tech
      Flux7 Tech
      • DevOps Toolchain: Reduce DevOps Challenges
      • Amazon Web Services: Reduce Complexities and Risks in AWS Architectures
      • AWS Database Services: Design and Implementation of Infrastructure for Cloud-Based Databases
      Configuration
      • Cloud Configuration: Gain Greater Consistency, Repeatability & Agility
      • HashiCorp Terraform: Defining Infrastructure as Code 
      • AWS CloudFormation: Reduce Maintenance and Improve Security
      Containers
      • Container Infrastructure: Improve Agility with Containers
      • Docker: Build, Ship and Run Applications Anywhere
      • Kubernetes: Container Consulting Services
      • Red Hat OpenShift: Speed Code Delivery

      Rapidly adopt technology to achieve Infrastructure as Code and continuous delivery and support of applications and workloads.

      Create Your DevOps Toolchain

  • Resources
      • Resources
      Research & Reports
      • Analyst Insights & Reports
      • Blog
      • Case Studies
      • White Papers
      News & Events
      • Events Calendar
      • Newsroom
      • Press Releases
      Flux7 Academy
      • Tech Tutorials

      Read about what we do, how we do it and how our customer's benefit from our solutions.

      Read and Download Our Case Studies

  • About
      • About Flux7
      Get To Know Us
      • About Flux7
      • Awards & Recognitions
      • Meet Our Team
      Work With Us
      • Careers
      • Our Culture
      Partner With Us
      • Flux7 partners with technology vendors who deliver solutions to help our customers address scalability, security, reduce the cost of infrastructure delivery and improve speed to market.

      Welcome to Flux7! Get to know us a bit better and discover what makes us different than other DevOps Consultants.

      Discover What Makes Us Different

  • Contact us
An NTT DATA Company

Login Contact us

RabbitMQ Tutorial: Creating Highly Available Message Queues Using RabbitMQ

RabbitMQ Tutorial: Creating Highly Available Message Queues Using RabbitMQ

By Flux7 Labs
November 8, 2019

Every day in the world of modern technology, high availability has become the key requirement of any layer in technology. Message broker software has become a significant component of most stacks. In this article, we will present a RabbitMQ tutorial: how to create highly available message queues using RabbitMQ.

RabbitMQ is an open-source message broker software (also called message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP). RabbitMQ server is written in the Erlang programming language.


The RabbitMQ Cluster

Clustering connects multiple nodes to form a single logical broker. Virtual hosts, exchanges, users and permissions are mirrored across all nodes in a cluster. A client connecting to any node can see all the queues in a cluster.

Clustering enables high availability of queues and increases the throughput.

A node can be a Disc node or RAM node. RAM node keeps the message state in memory with the exception of queue contents which can reside on a disk if the queue is persistent or too big to fit into memory.

RAM nodes perform better than Disc nodes because they don’t have to write to a disk as much as disk nodes. But, it is always recommended to have disk nodes for persistent queues.

We’ll discuss how to create and convert RAM and Disk nodes later in the post.

Prerequisites:

Network connection between nodes must be reliable.

All nodes must run the same version of Erlang and RabbitMQ.

All TCP ports should be open between nodes.

We have used CentOS for the demo. Installation steps may vary for Ubuntu and OpenSuse. In this demo, we have launched two m1.small servers in AWS for master and slave nodes.

1. Install Rabbitmq

Install Rabbitmq in master and slave nodes.

$ yum install rabbitmq-server.noarch

2. Start Rabbitmq

/etc/init.d/rabbitmq-server start

3. Create the Cluster

Stop RabbitMQ in Master and slave nodes. Ensure service is stopped properly.

/etc/init.d/rabbitmq-server stop

Copy the file below to all nodes from the master. This cookie file needs to be the same across all nodes.

$ sudo cat /var/lib/rabbitmq/.erlang.cookie

Make sure you start all nodes after copying the cookie file from the master.

Start RabbitMQ in master and all nodes.

$ /etc/init.d/rabbitmq-server start

Then run the following commands in all the nodes, except the master node:

$ rabbitmqctl stop_app
$ rabbitmqctl reset
$ rabbitmqctl start_app

Now, run the following commands in the master node:

$ rabbitmqctl stop_app
$ rabbitmqctl reset

Do not start the app yet.

The following command is executed to join the slaves to the cluster:

$ rabbitmqctl join_cluster rabbit@slave1 rabbit@slave2

Update slave1 and slave2 with the hostnames/IP address of the slave nodes. You can add as many slave nodes as needed in the cluster.

Check the cluster status from any node in the cluster:

$ rabbitmqctl cluster_status

By default, the cluster stores messages on the disk. You can also choose to store Queues in Memory.

You can have a node as a RAM node while attaching it to the cluster:

$ rabbitmqctl stop_app
$ rabbitmqctl join_cluster --ram rabbit@slave1

It is recommended to have at least one disk node in the cluster so that messages are stored on a persistent disk and can avoid any loss of messages in case of a disaster.

The performance of RAM nodes are a little better than disk nodes, and gives you better throughput.

4. Set the HA Policy

The following command will sync all the queues across all nodes:

$ rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

Ex: Policy where queues whose names begin with “ha.” are mirrored to all nodes in the cluster: $ rabbitmqctl set_policy ha-all “^ha.” ‘{“ha-mode”:”all”}‘

5. Test the Queue mirror

We are going to run a sample python program to create a sample queue. You need the below packages installed from where you want to run the program.

Install python-pip

$ yum install python-pip.noarch

Install Pika

$ sudo pip install pika==0.9.8

Create send.py file and copy the content below. You need to update the “localhost” with name/ip of master/slave node.

#!/usr/bin/env python
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters(
        host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='hello')

channel.basic_publish(exchange='',
                      routing_key='hello',
                      body='Hello World!')
print " [x] Sent 'Hello World!'"
connection.close()

Run the python script using the command:

$ python send.py

This will create a Queue (hello) with a message on the RabbitMQ cluster.

Check if the message is available across all nodes.

$ sudo rabbitmqctl list_queues

Now, create a file named receive.py and copy the content below.

#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
              'localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
def callback(ch, method, properties, body):
   print " [x] Received %r" % (body,)
channel.basic_consume(callback,
                     queue='hello',
                     no_ack=True)
print ' [*] Waiting for messages. To exit press CTRL+C'
channel.start_consuming()

Run the script and check the Queue in either the slave or master:

$ sudo rabbitmqctl list_queues

6. Setup Load Balancer

Now, we have multiple MQs running in a cluster. All are in sync and have the same queues.

How do we point our application to the cluster? We can’t point our application to a single node. If a node fails, we need a mechanism to auto-failover to other nodes in the cluster. There are multiple ways to achieve it. But, we prefer to use the load balancer.

There are two advantages in using the load balancer:

  1. High availability

  2. Better network throughput because the load is evenly distributed across nodes.

Create a load balancer in front of it and map the backend MQ instance. You can choose either HAProxy or Apache or Nginx or any hardware load balancer you use in your organization.

If the servers are running in an AWS VPC, then choose internal load balancer. Update the application to point to the load balancer endpoint.

Get Started with AWS

Receive AWS tips, DevOps best practices, news analysis, commentary and more. Sign up for our blog here and set your topic and frequency preferences. Or, download our guide on getting started with AWS, establishing a secure AWS enterprise architecture with Flux7 Landing Zones.

For more RabbitMQ knowledge, read:  Comparing RabbitMQ for AWS with AWS SQS FIFO queues


Download the AWS Getting Started Guide

Share This Article
Facebook Twitter Pinterest Linkedin
Prev Post
Next Post

Related Articles

IT Modernization and DevOps News Week in Review 1.18.2021
By Flux7 Labs
January 18, 2021

IT Modernization and DevOps News Week in Review 1.18.2021

READ MORE
re:Invent Round-Up of AWS DevOps Announcements
By Flux7 Labs
December 21, 2020

re:Invent Round-Up of AWS DevOps Announcements

READ MORE

Recent Posts

  • IT Modernization and DevOps News Week in Review 1.18.2021

  • re:Invent Round-Up of AWS DevOps Announcements

  • How Will SASE Change Networking in 2021?

  • AWS re:Invent Machine Learning Round-Up

  • How to Publish Managed Images to the Azure Marketplace

  • AWS re:Invent News Round-Up

  • Shave Days off Azure Marketplace Publishing with Automated Testing

  • IT Modernization and DevOps News Week in Review 11.30.2020

  • How To: Multi-Cluster Monitoring in Amazon EKS

  • IT Modernization and DevOps News Week in Review 11.16.2020

Flux7
  • About Flux7
  • Contact Us
  • Careers at Flux7
  • Newsroom
  • Meet our Team
Services
  • Enable Software Innovation
  • Enforce Security and Compliance
  • Adopt Cloud
  • Cloud Migration Services
  • Secure the Cloud
Resources
  • Analysts Reports
  • Case Studies
  • White Papers
About Flux7

Flux7, an NTT DATA Company, helps enterprises reduce the complexities of new and evolving cloud automation strategies. Agile and DevOps-native, Flux7’s robust IT services portfolio prioritizes a fast path to ROI, is transformation focused and creates secure and stable pathways for operational excellence.

Follow Us
Flux7, an NTT DATA Company | All Rights Reserved | Privacy Policy