Going Down The Rabbit Hole? By Jon Dokulil
Jon Dokulil, VP of Engineering at Hudl, a company that makes software products for athletic teams and coaches, shares his experience of transitioning from MSMQ to RabbitMQ. If you’re interested in more of the nitty-gritty code details, there’s a jump link to a longer form article below.
Fun Fact: 2014 “Cyber Monday” sales were nearly 10% higher
than those from 2013
Going Down The Rabbit Hole? By Jon Dokulil, VP of Engineering at Hudl
Even though we use C# in our application servers, Hudl moved from MSMQ to RabbitMQ several years ago and we’ve been pleased with the result. RabbitMQ is stable, very fast, and provides many options for message routing and durability. Plus the C# driver is solid and easy to work with. At Hudl we offload long-running operations and many of our interactions with third-party systems like Facebook, CRM, credit card processors, etc.
Some lessons learned using RabbitMQ:
- Everything we queue needs to be durable (though RabbitMQ does provide options if you don’t require durability). In addition to durable queues, setting Delivery-Mode=2 means calls to enqueue a message don’t return in our producers until RabbitMQ has written the message to disk. This ensures a crash of our RabbitMQ node will never result in data loss.
- Explicitly acknowledge messages only after processing completes to ensure at-least-once delivery of messages. That way if a consumer crashes during message processing, RabbitMQ will deliver that message to another consumer
- If you are running in the cloud, ensure the RabbitMQ queue files reside on durable storage. We use AWS and store the queue files on PIOPs EBS volumes. PIOPs are critical if you expect high message volume. GP2 volumes could work if your message workloads come in sporadic bursts.
- Tweak your Prefetch count to tune performance. Prefetch tells the consumers how many messages to grab at a time. Too low and you’ll slow down waiting for network requests. A good starting point is 20-50.
- RabbitMQ uses persistent TCP connections. It’s best to have a single connection per server. The RabbitMQ client takes care of multiplexing, so one TCP connection can consume and produce messages across any number of queues.
- We’ve invested a lot of time into our RabbitMQ code base. If we were starting today we’d use EasyNetQ, a wrapper around the RabbitMQ client that greatly simplifies the most common usage patterns.
RabbitMQ and C# work well together. We’ve been using it now for three years in production and are still happy with the choice. It’s easy to get up and running with just the basic functionality but is also flexible if/when you want to use more complex routing scenarios. RabbitMQ is also very fast, we regularly push over 1,500 messages/sec and Rabbit barely breaks a sweat. If you want to move some logic off of your web servers then it’s a great choice to do so in a durable and performant way.
For more details and code samples, check out the full article here
( http://public.hudl.com/bits/archives/2013/11/11/c-rabbitmq-happy-servers/ )