Thursday, February 19, 2015

Introduction to RabbitMQ & Hands on

1.    What is RabbitMQ

RabbitMQ is a popular Open Source message queuing system that implements the Advanced Message Queuing Protocol (AMQP). It has been estimated that there are presently some 30,000 production deployments of RabbitMQ across the globe, and this number is growing rapidly. Most of these deployments are business-critical, underpinning everything from internet-based pizza ordering systems through to providing the central nervous system for OpenStack-based cloud deployments. RabbitMQ currently supports versions 0.8.0 and 0.9.1 of AMQP and will soon also provide support for 1.0. However, a somewhat overlooked capability of RabbitMQ is its ability to also readily provide support via its flexible plugin architecture for a variety other popular Open Source message queuing protocols, including STOMP, MQTT, ZeroMQ, and RESTful messaging via the RabbitHub plugin.  Most good message queuing protocols share many features in common. However some are better suited to a particular set of use cases than others. This ability of RabbitMQ to be able to seamlessly receive and propagate messages simultaneously via multiple protocols is an extremely powerful facility, and one that affords great flexibility. For example, it means that it is possible to use the most appropriate protocol for a particular function or to simultaneously use different protocols to disseminate the same data to different types of users via the most appropriate protocol without having to develop and maintain any separate gateway components. The following text discusses this ability of RabbitMQ to support multiple message queuing protocols. RabbitMQ can be summarized as below-

  • Multi Protocol Messaging Server
  • Open Source (MPL)
  • Polyglot
  • Written in Erlang/OTP

1.1  Multi Protocol

Below is the list of protocols supported by RabbitMQ-
  • AMQP
  • MQTT
  • STOMP

1.2  Polyglot

Below are some programming languages supported by RabbitMQ-
  • Java
  • node.js
  • Erlang
  • PHP
  • Ruby
  • .Net
  • Haskell

2.    Some users of RabbitMQ

Below are some company who are using RabbitMQ-

  • Instagram
  • Indeed.com
  • Telefonica
  • Mercado Libre
  • NHS
  • Mozilla

3.    Some RabbitMQ Features   


  • No message loss
  • Persistent Messages
  • Publisher Confirms
  • Message Acknowledgment
  • Mirrored Queues
  • Message Ordering
  • Dead Letter Exchanges
  • Alternate Exchanges
  • Message and Queues TTLs
  • Consumer Priorities
  • Federation
  • Shovel

4.    RabbitMQ Architecture




Queue: Acts as a message buffer between producer & consumer
Exchange: Determines how messages are routed to queues
Producer: Publishes messages to the exchange
Consumer: Receives message from the queue

5.    Messaging  and AMQP

Messaging or message queuing is a style of communication between applications or components that enables a loosely coupled architecture. The Advanced Message Queuing Protocol (AMQP) is an application layer protocol specification for asynchronous messaging. Being built as a wire-level protocol instead of an API (like e.g. JMS) AMQP clients should be capable of sending and receiving message regardless of their respective vendors. As of now there are already quite a number of server3and client4implementations on multiple platforms available. RabbitMQ is an Erlang-based implementation of AMQP, which supports advanced features such as clustering.

The request-response style of interaction


 One-way interaction with message queuing



 Message enabling a loosely coupled architecture


 The e-mail infrastructure as an analogy for message queuing


6.    The RabbitMQ broker


RabbitMQ is an Erlang implementation of an AMQP broker. Erlang has been chosen to build it because of its intrinsic support for building highly-reliable and distributed applications. Indeed, it is used to run telecommunication switches for which a proverbial total system's availability of 9 nines has been reported (that's 32 milliseconds of downtime per year). Erlang is also able to run on any operating system.


7.    Exchanges and Exchange Types

Exchanges are AMQP entities where messages are sent. Exchanges take a message and route it into zero or more queues. The routing algorithm used depends on the exchange type and rules called bindings. AMQP 0-9-1 brokers provide four exchange types:

Name
Default pre-declared names
Direct exchange
(Empty string) and amq.direct
Fanout exchange
amq.fanout
Topic exchange
amq.topic
Headers exchange
amq.match (and amq.headers in RabbitMQ)

Besides the exchange type, exchanges are declared with a number of attributes, the most important of which are:
  • Name
  • Durability (exchanges survive broker restart)
  • Auto-delete (exchange is deleted when all queues have finished using it)
  • Arguments (these are broker-dependent)

Exchanges can be durable or transient. Durable exchanges survive broker restart whereas transient exchanges do not (they have to be re-declared when broker comes back online). Not all scenarios and use cases require exchanges to be durable.

7.1  Default Exchange

The default exchange is a direct exchange with no name (empty string) pre-declared by the broker. It has one special property that makes it very useful for simple applications: every queue that is created is automatically bound to it with a routing key which is the same as the queue name.
For example, when you declare a queue with the name of "search-indexing-online", the AMQP broker will bind it to the default exchange using "search-indexing-online" as the routing key. Therefore, a message published to the default exchange with the routing key "search-indexing-online" will be routed to the queue "search-indexing-online". In other words, the default exchange makes it seem like it is possible to deliver messages directly to queues, even though that is not technically what is happening.

7.2  Direct Exchange

A direct exchange delivers messages to queues based on the message routing key. A direct exchange is ideal for the unicast routing of messages (although they can be used for multicast routing as well). Here is how it works:

  • A queue binds to the exchange with a routing key K
  • When a new message with routing key R arrives at the direct exchange, the exchange routes it to the queue if K = R
Direct exchanges are often used to distribute tasks between multiple workers (instances of the same application) in a round robin manner. When doing so, it is important to understand that, in AMQP 0-9-1, messages are load balanced between consumers and not between queues.
A direct exchange can be represented graphically as follows:



7.3  Topic exchange

Topic exchanges route messages to one or many queues based on matching between a message routing key and the pattern that was used to bind a queue to an exchange. The topic exchange type is often used to implement various publish/subscribe pattern variations. Topic exchanges are commonly used for the multicast routing of messages.
Topic exchanges have a very broad set of use cases. Whenever a problem involves multiple consumers/applications that selectively choose which type of messages they want to receive, the use of topic exchanges should be considered.
Example uses:

  • Distributing data relevant to specific geographic location, for example, points of sale
  • Background task processing done by multiple workers, each capable of handling specific set of tasks
  • Stocks price updates (and updates on other kinds of financial data)
  • News updates that involve categorization or tagging (for example, only for a particular sport or team)
  • Orchestration of services of different kinds in the cloud
  • Distributed architecture/OS-specific software builds or packaging where each builder can handle only one architecture or OS



7.4  Fanout Exchange

A fanout exchange routes messages to all of the queues that are bound to it and the routing key is ignored. If N queues are bound to a fanout exchange, when a new message is published to that exchange a copy of the message is delivered to all N queues. Fanout exchanges are ideal for the broadcast routing of messages.
Because a fanout exchange delivers a copy of a message to every queue bound to it, its use cases are quite similar:

  • Massively multi-player online (MMO) games can use it for leaderboard updates or other global events
  • Sport news sites can use fanout exchanges for distributing score updates to mobile clients in near real-time
  • Distributed systems can broadcast various state and configuration updates
  • Group chats can distribute messages between participants using a fanout exchange (although AMQP does not have a built-in concept of presence, so XMPP may be a better choice)
A fanout exchange can be represented graphically as follows:



7.5  Headers Exchange

A headers exchange is designed to for routing on multiple attributes that are more easily expressed as message headers than a routing key. Headers exchanges ignore the routing key attribute. Instead, the attributes used for routing are taken from the headers attribute. A message is considered matching if the value of the header equals the value specified upon binding.
It is possible to bind a queue to a headers exchange using more than one header for matching. In this case, the broker needs one more piece of information from the application developer, namely, should it consider messages with any of the headers matching, or all of them? This is what the "x-match" binding argument is for. When the "x-match" argument is set to "any", just one matching header value is sufficient. Alternatively, setting "x-match" to "all" mandates that all the values must match.
Headers exchanges can be looked upon as "direct exchanges on steroids". Because they route based on header values, they can be used as direct exchanges where the routing key does not have to be a string; it could be an integer or a hash (dictionary) for example.


8.    Hands on code in Java 

To be continued ..............

Friday, February 6, 2015

RabbitMQ - Installation & Configuration at AWS EC2

Ø  RabbitMQ will work on 32-bit operating system, but for production usage we recommend 64-bit instances.
Ø  Launched an ami instance Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-864d84ee, 64-bit OS.
RabbitMQ is a message broker based on Erlang. So, before installing RabbitMQ server, we need to install Erlang.
2.1   Adding repository entry 

To add Erlang Solutions repository (including our public key for apt-secure) to your system, call the following commands:

wget http://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
sudo dpkg -i erlang-solutions_1.0_all.deb
Refresh the repository cache and install the erlang package.

sudo apt-get update
sudo apt-get install erlang

For installing latest version of RabbitMQ, use the debian repository and run the below shell script.

#!/bin/sh
cat <<EOF > /etc/apt/sources.list.d/rabbitmq.list
deb http://www.rabbitmq.com/debian/ testing main
EOF
curl http://www.rabbitmq.com/rabbitmq-signing-key-public.asc -o /tmp/rabbitmq-signing-key-public.asc
apt-key add /tmp/rabbitmq-signing-key-public.asc
rm /tmp/rabbitmq-signing-key-public.asc
apt-get -qy update
apt-get -qy install rabbitmq-server
 

The default folders where the package has installed files are
configuration files
/etc/rabbitmq
application files
/usr/lib/rabbitmq
data files
/var/lib/rabbitmq
Log files
/var/log/rabbitmq


sudo apt-get install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer

curl http://www.rabbitmq.com/releases/rabbitmq-java-client/v3.3.4/rabbitmq-java-client-bin-3.3.4.tar.gz \ -o rabbitmq-java-client-bin-3.3.4.tar.gz
tar xzf rabbitmq-java-client-bin-3.3.4.tar.gz
cd rabbitmq-java-client-bin-3.3.4



When the dependencies are available, we can run the test:





Note: If the above url is not accessible, add an entry for TCP (port: 15672) in Inbound rules of Security Groups of your EC2 instance.







8.   Start RabbitMQ Server