RabbitMQ Shovel Plugin

Introduction

The RabbitMQ Shovel plugin is a powerful tool in the RabbitMQ ecosystem that enables message transfer between brokers. Think of it as a conveyor belt that reliably moves messages from a source to a destination, even across different RabbitMQ instances or clusters. This capability makes it especially valuable for scenarios like geographic distribution, load balancing, and creating federated architectures.

Unlike federation, which creates a distributed messaging topology with consumers pulling messages, shovels actively push messages from one broker to another, giving you more direct control over the message flow.

Why Use the Shovel Plugin?

The Shovel plugin is particularly useful when you need to:

• Transfer messages between brokers in different geographic locations
• Connect brokers across different security domains
• Implement a data pipeline across multiple systems
• Create backup or disaster recovery solutions
• Balance load across multiple RabbitMQ instances
• Migrate from one RabbitMQ setup to another

Getting Started with Shovel

Prerequisites

Before we begin, ensure you have:

1. RabbitMQ server installed (version 3.8.0 or higher recommended)
2. Administrative access to enable plugins
3. Basic familiarity with RabbitMQ concepts (exchanges, queues, bindings)

Enabling the Plugin

First, let's enable the Shovel plugin and its management interface:

bash

rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_shovel_management

After enabling these plugins, you'll see a new "Shovel" tab in the RabbitMQ management interface.

How Shovel Works

At its core, a shovel:

1. Connects to a source (queue or exchange) as a consumer
2. Receives messages from the source
3. Connects to a destination (queue or exchange) as a publisher
4. Publishes the messages to the destination
5. Acknowledges the messages at the source after successful publication

Configuring a Shovel

Shovels can be configured in two ways:

1. Dynamic Shovels: Created and managed at runtime via the HTTP API or management UI
2. Static Shovels: Defined in the RabbitMQ configuration file

Let's explore both approaches:

Dynamic Shovel Configuration

Dynamic shovels can be created through the management UI or using the HTTP API. Here's how to create one using the RabbitMQ management interface:

1. Navigate to the "Admin" tab
2. Select "Shovel Management"
3. Click "Add a new shovel"
4. Fill in the required parameters:
   • Name
   • Source and destination broker URIs
   • Source and destination queue/exchange names

Alternatively, you can use the HTTP API with curl:

bash

curl -X POST -u guest:guest http://localhost:15672/api/parameters/shovel/%2f/my-first-shovel \
     -H "content-type:application/json" \
     -d '{
         "value": {
           "src-uri": "amqp://guest:[email protected]:5672/%2f",
           "src-queue": "source-queue",
           "dest-uri": "amqp://guest:[email protected]:5672/%2f",
           "dest-queue": "destination-queue"
         }
       }'

Static Shovel Configuration

Static shovels are configured in the RabbitMQ configuration file. Here's an example:

erlang

[
  {rabbitmq_shovel,
    [{shovels,
      [{my_static_shovel,
        [{source,
          [{protocol, amqp091},
           {uris, ["amqp://guest:[email protected]:5672"]},
           {queue, <<"source-queue">>},
           {prefetch_count, 100}]},
         {destination,
          [{protocol, amqp091},
           {uris, ["amqp://guest:[email protected]:5672"]},
           {queue, <<"destination-queue">>}]},
         {ack_mode, on_confirm},
         {reconnect_delay, 5}
        ]}
      ]}
    ]}
].

Practical Example: Creating a Simple Shovel

Let's create a simple shovel that transfers messages between two queues on the same broker (you can extend this to work across different brokers):

Step 1: Set up the source and destination queues

javascript

// Using Node.js amqplib
const amqp = require('amqplib');

async function setupQueues() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();
  
  // Create source queue
  await channel.assertQueue('source-queue', { durable: true });
  
  // Create destination queue
  await channel.assertQueue('destination-queue', { durable: true });
  
  console.log('Queues created successfully');
  await channel.close();
  await connection.close();
}

setupQueues().catch(console.error);

Step 2: Configure the shovel

Using the RabbitMQ Management UI or via the HTTP API:

bash

curl -X POST -u guest:guest http://localhost:15672/api/parameters/shovel/%2f/my-first-shovel \
     -H "content-type:application/json" \
     -d '{
         "value": {
           "src-uri": "amqp://guest:guest@localhost:5672/%2f",
           "src-queue": "source-queue",
           "dest-uri": "amqp://guest:guest@localhost:5672/%2f",
           "dest-queue": "destination-queue"
         }
       }'

Step 3: Publish a message to the source queue

javascript

const amqp = require('amqplib');

async function publishMessage() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();
  
  const message = 'Hello Shovel!';
  channel.sendToQueue('source-queue', Buffer.from(message));
  
  console.log('Message sent to source-queue');
  await channel.close();
  await connection.close();
}

publishMessage().catch(console.error);

Step 4: Consume from the destination queue

javascript

const amqp = require('amqplib');

async function consumeMessages() {
  const connection = await amqp.connect('amqp://localhost');
  const channel = await connection.createChannel();
  
  channel.consume('destination-queue', (msg) => {
    if (msg !== null) {
      console.log('Received from destination-queue:', msg.content.toString());
      channel.ack(msg);
    }
  });
  
  console.log('Consumer started, waiting for messages...');
}

consumeMessages().catch(console.error);

When you run these scripts in order, you'll see your message travel from the source queue to the destination queue via the shovel.

Advanced Shovel Configurations

Shovel Types

Shovels can operate in two modes:

1. One-time Shovel: Transfers messages and terminates when the source is empty
2. Permanent Shovel: Continuously transfers messages as they arrive

To create a one-time shovel, add "delete-after": "queue-length" to your configuration:

json

{
  "value": {
    "src-uri": "amqp://guest:guest@localhost:5672/%2f",
    "src-queue": "source-queue",
    "dest-uri": "amqp://guest:guest@localhost:5672/%2f",
    "dest-queue": "destination-queue",
    "delete-after": "queue-length"
  }
}

Other possible values for delete-after include:

• "never" (default): Shovel runs indefinitely
• "queue-length": Terminates after transferring all messages present when the shovel started
• "10" (or any number): Terminates after transferring that many messages

Working with Exchanges

Shovels can also publish to exchanges instead of queues:

json

{
  "value": {
    "src-uri": "amqp://guest:guest@localhost:5672/%2f",
    "src-queue": "source-queue",
    "dest-uri": "amqp://guest:guest@localhost:5672/%2f",
    "dest-exchange": "destination-exchange",
    "dest-exchange-key": "routing.key"
  }
}

And consume from exchanges:

json

{
  "value": {
    "src-uri": "amqp://guest:guest@localhost:5672/%2f",
    "src-exchange": "source-exchange",
    "src-exchange-key": "routing.key",
    "dest-uri": "amqp://guest:guest@localhost:5672/%2f",
    "dest-queue": "destination-queue"
  }
}

Message Transformation

While the Shovel plugin doesn't provide built-in transformation capabilities, you can achieve this by:

1. Using a dedicated queue and consumer for transformation
2. Publishing the transformed message to another queue
3. Setting up the shovel from this intermediate queue

Real-World Use Case: Geographic Distribution

Let's consider a scenario where you have applications in different regions and need to ensure messages are available locally in each region.

In this setup:

1. Each regional RabbitMQ instance serves local producers and consumers
2. The Shovel plugin ensures messages are available in both regions
3. If one region goes down, the other can still function

Implementation steps:

1. Set up RabbitMQ instances in both regions
2. Configure bidirectional shovels between them
3. Configure applications to connect to their local RabbitMQ instance

This architecture reduces latency for local operations while ensuring global message availability.

Monitoring and Troubleshooting

Checking Shovel Status

You can check the status of your shovels through the management UI or the HTTP API:

bash

curl -u guest:guest http://localhost:15672/api/shovels

The response will include the status of each shovel, which could be:

• running: Operating normally
• starting: In the process of connecting
• terminated: Stopped after completing its work (for one-time shovels)
• stopped: Stopped due to an error

Common Issues and Solutions

1. Connection Failures

   • Check network connectivity between brokers
   • Verify AMQP credentials
   • Ensure the destination broker is running

2. Message Acknowledgement Issues

   • Check that the ack_mode is set appropriately
   • Common options: on_confirm (default), on_publish, no_ack

3. Performance Concerns

   • Adjust prefetch_count to control how many messages the shovel processes at once
   • Consider setting up multiple shovels for high-volume queues

Best Practices

1. Security: Use dedicated users with minimal permissions for shovels
2. Monitoring: Set up alerts for shovel status changes
3. Redundancy: Consider setting up multiple shovels for critical paths
4. Naming: Use descriptive names that identify the source and destination
5. Testing: Always test shovel configurations in a non-production environment first

Summary

The RabbitMQ Shovel plugin provides a flexible and powerful way to move messages between brokers. It's particularly valuable for geographic distribution, connecting different security domains, and building resilient messaging architectures.

Key takeaways from this guide:

• Shovels act as a reliable message transport between brokers
• They can be configured dynamically or statically
• Shovels support various types of sources and destinations (queues and exchanges)
• They can be one-time or persistent
• Proper monitoring and configuration are essential for production use

Further Learning

To deepen your understanding of the RabbitMQ Shovel plugin, try these exercises:

1. Set up a bidirectional shovel between two RabbitMQ instances
2. Create a shovel that filters messages using an exchange and binding keys
3. Implement a disaster recovery scenario using shovels
4. Configure a shovel with TLS for secure message transfer
5. Set up a shovel between RabbitMQ instances in different docker containers

Related Topics

• RabbitMQ Federation Plugin
• RabbitMQ Clustering
• Message Patterns in Distributed Systems
• Event-Driven Architectures