You are reading our blog series: Aegir5 Roadmapping
Previously in this series, we looked at the Aegir5 front-end interface architecture, as well as the lower level entities, Tasks and Operations that provide building blocks.
As mentioned in the first part of the series, our most recent work on Aegir5 itself has been reworking the queue system. In this post, we explore this topic in more detail.
The Aegir5 queue is implemented using
Celery, which is a full-featured
Python-based task queue, built atop RabbitMQ. Initially we built dispatcherd
which operates as the back-end queue worker. For a variety of
reasons that we’ll get into later, we also added relayd to run on the
front end.
dispatcherd is the Aegir5 equivalent of Drush Provision. It is responsible
for writing configuration files and then running the commands that will
instantiate whatever the configurations represent.
dispatcherd runs commands such as ansible playbook. It then gathers the
terminal output and sends it back to the front-end. Initially, this was done
using an HTTP end-point, which would update the log record of the relevant
operation. This worked reasonably well, but it required HTTP access from
dispatcherd to the front end.
This is where relayd comes into play. By hitting the front end with updates
from the terminal output, we had to bootstrap the web UI repeatedly, putting
unnecessary load on the front end. So we built an alternative solution:
relayd running as a queue worker on the front end. This front end worker is
very lightweight, picking up data and passing it on to an aegir:console
command that can be more efficient at injecting the log results from the
back end.
Once we had added relayd, we realized it presented another opportunity. Up to
this point, the front end had been posting directly into the queue for
dispatcherd. However, this required the front end to have credentials to the
Celery queue. Since the intention was for relayd to run in the same context
as the front end, we could instead simply have the front end communicate with
relayd securely without providing credentials to the back-end.
In addition to a more loose coupling of all the components, this represents an improved security posture for Aegir5 overall. By removing queue credentials from the front-end UI, we reduce the risk of a vulnerability in the Drupal site leading to a compromise of the back-end infrasructure.
Having built and tested the queue architecture fairly thoroughly we are poised to tie everything together. In the next post, we explore the integration of our Kubernetes framework with the queue mechanism and existing front-end UI to create a fully functional (albeit MVP) Aegir5 implementation.
The article Aegir5: Queue Architecture first appeared on the Consensus Enterprises blog.
We've disabled blog comments to prevent spam, but if you have questions or comments about this post, get in touch!