Message Queues with Galaxy and Pulsar

Message Queues (MQs) allow securely connecting Galaxy to Pulsar servers. MQs provide a transport method to allow the producer (Galaxy) to publish messages (jobs) which consumers consume (Pulsar).

Message Queue Overview

In brief terms, an MQ is simply a queue of arbitrary messages. The messages themselves don’t conform to any particular specification, and require the service author to do the validation/parsing. This allows MQs to be very efficient, lightweight services. There are a couple of key concepts to understand MQs which will be covered here. If you’re not interested in the background terminology and an overview of functionality in MQs, skip this section.

Galaxy and Pulsar both make use of the Kombu library for accessing MQs. Kombu is an abstraction layer over a number of different queues, allowing the deployer some choice in which service they’d like to use to provide the Galaxy/Pulsar interaction. The Kombu introduction is an excellent resource for those interested in a basic introduction to building services with MQs. For the purpose of this documentation, we’ll be using AMQP urls.

AMQP Urls look like the following:

amqp://guest:password@localhost:5672//

There is user/pass information supplied ahead of the server URL. It is important to note the trailing / at the end, it’s the virtual host. By default, it is /, but more can be configured. Within a virtual host there can be several exchanges. Exchanges manage the routing of messages from producers to consumers. Within an exchange, there are one or more queues. For visual reference:

AMQP Server -> Virtual Host -> Exchange -> Queue

Within a queue, routing keys are used to identify groups of messages. Depending on the exchange type, the routing key helps determine which consumer picks up a message. We are only concerned with direct exchanges which require that the consumer and producer have identical routing keys. Pulsar constructs the routing names with the following snippet:

pulsar%s__kill          # Job kill commands
pulsar%s__setup         # Job submissions
pulsar%s__status_update # Informs of job completions

where %s is the “Manager name”, in Pulsar parlance.

Deploying RabbitMQ

For Ubuntu,

sudo apt-get install rabbitmq-server

This will provide the command rabbitmqctl which can be run as root to handle configuration. If you’re new to MQs in general, you may be interested in enabling the web GUI with:

rabbitmq-plugins enable rabbitmq_management

This will listen on http://fqdn:15672 and provide access to manage and view queue/exchange/vhost information. The default username and password are both “guest”. This user is generally only allowed to login on localhost, though this restriction is disabled on the web interface. (This is a security risk, in case that wasn’t clear. Please do not leave this up, unsecured!)

Locking it Down

As someone new to RabbitMQ, this may not be the best way to do this, but it works for me. Seeing that guest:guest could access everything in /, and not wanting to break other things that might depend on that access, I created a separate virtual host without guest access.

# Add a user specific to galaxy, allows us to restrict/manage logins
sudo rabbitmqctl add_user galaxy some_long_password
# Add the virtualhost
sudo rabbitmqctl add_vhost galaxy
# Grant permissions, allow configure/read/write to ALL exchanges in the galaxy virtualhost
sudo rabbitmqctl set_permissions -p galaxy .* .* .*

Alternatively this can be done in the default virtual host:

# Add a user specific to galaxy, allows us to restrict/manage logins
sudo rabbitmqctl add_user galaxy some_long_password
# Set permissions for the galaxy user, allow conf, read, and write on ONLY the "pulsar" exchange
sudo rabbitmqctl set_permissions -p / galaxy pulsar pulsar pulsar

Configuring Galaxy

The Galaxy configuration portion is covered in Galaxy Configuration.

Configuring Pulsar

The Pulsar configuration portion is covered in Job Managers.