August 19, 2014

###Corrections:

• 4% != .004% : When I was writing the article, my brain translated 99996 into 96000. Big difference. It turns out that I’m unable to dequeue somewhere between .004% and .20% of messages in about half of test runs.

###Note:

I’ve been chatting with some very helpful RabbitMQ-knowledgeable people, and they have some suggestions for the issues I’m seeing that I’m going to check out. I will update this article with my findings.

I want to thank Alvaro Videla and Michael Klishin for reading my first attempt at this post and suggesting different avenues to explore.

Introduction

For the two of you who don’t know, RabbitMQ is a really neat AMQP-compliant queue broker. It exists to facilitate the passing of messages between or within systems. I’ve used it for a couple of different projects, and I’ve found it to be tremendously capable: I’ve seen a RabbitMQ instance running on a single, moderately sized, VM handle almost 3GB/s.

I was doing some load testing with RabbitMQ recently, and I found that, if I started attempting to publish more than around 2500 10KB messages per second, about 4% as much as 0.2% of those messages wouldn’t make it to the queue during some test runs. I am not sure if this is my code’s fault or if I am running into the limits of the RabbitMQ instance I was testing against (probably the former), but with the help of the RabbitMQ community, I was able to come up with some best practices that I’ve described below.

The examples below are all in Go, but I’ve tried my best to explain them in such a way that people who are not familiar with Go can understand them.

Terminology

If you’re unfamiliar with AMQP, here’s some terminology to help understand what’s possible with a queue broker and what the words mean.

• Connection: A connection is a long-lived TCP connection between an AMQP client and a queue broker. Maintaining a connection reduces TCP overhead. A client can re-use a connection, and can share a connection among threads.
• Channel: A channel is a short-lived sub-connection between a client and a broker. The client can create and dispose of channels without incurring a lot of overhead.
• Exchange: A client writes messages to an exchange. The exchange forwards each message on to zero or more queues based on the message’s routing key.
• Queue: A queue is a first-in, first out holder of messages. A client reads messages from a queue. The client can specify a queue name (useful, for example, for a work queue where multiple clients are consuming from the same queue), or allow the queue broker to assign it a queue name (useful if you want to distribute copies of a message to multiple clients).
• Routing Key: A string (optionally) attached to each message. Depending on the exchange type, the exchange may or may not use the Routing Key to determine the queues to which it should publish the message.
• Exchange types:
  • Direct: Delivers all messages with the same routing key to the same queue(s).
  • Fanout: Ignores the routing key, delivers a copy of the message to each queue bound to it.
  • Topic: Each queue subscribes to a topic, which is a regular expression. The exchange delivers the message to a queue if the queue’s subscribed topic matches the message.
  • Header: Ignores the routing key and delivers the message based on the AMQP header. Useful for certain kinds of messages.

Testing methodology

Here is the load tester I wrote: https://github.com/backstop/rabbit-mq-stress-tester. It uses the streadway/amqp library. Per this issue, my stress tester does not share connections or channels between Goroutines – it launches a configurably-sized pool of Goroutines, each of which maintains its own connection to the RabbitMQ server.