This post uses the new .tar.gz binary distribution of ArangoDB to run multiple versions of ArangoDB alongside each other on the same machines. We will do a production-ready deployment on 3 cloud instances with authentication, TLS encryption, (self-signed) certificates and systemd service. In the end, we show how to perform a rolling upgrade for one of the clusters to a new version.

The new .tar.gz binary archive

Starting with Version 3.4.0 we have changed the way how we build our binary distribution. We are now using completely static binaries (see here for another interesting story) which makes it possible to run the same binary on any Linux distribution and version. As a consequence, we can publish a .tar.gz binary archive, which can be installed as any user at any location in the filesystem by extracting the archive. So the first step I did on my three instances was to download and extract the archive:

which produced a directory arangodb3-3.4.1 in my home directory /home/neunhoef. The idea is now that the ArangoDB starter arangodb can simply be run from the bin directory in that directory and will automatically find the rest of the distribution.

Setting up a CA and certificates

Since we want to do things properly, we want to protect all network traffic between instances and from the client to the database by TLS. Therefore we need certificates, more precisely, we want to use a self-signed CA and use this to produce a server certificate and key. Since we sign the server certificate with the CA, we can then give the CA certificate to our browser and thereby silence the nasty security warnings because the browser cannot establish a complete chain of trust from a root certificate it knows.

Fortunately, this is all very simple using the ArangoDB starter executable:

Note that I simply had to use the full path to the starter executable to get it to work.

The first of these commands creates the CA with the two files tls-ca.key (private key) and tls-ca.crt (certificate). The second command produces a server certificate, for convenience I include all three public and private IP addresses as host names as well as the three DNS names I will use in my domain 9hoeffer.de. This makes the server certificates valid for all three hosts. The internal IP addresses are needed for cluster-internal communication, the DNS names are needed for external access. I could have dropped the external IP addresses here, but never mind.

After this, I quickly stuff away tls-ca.key in a secure place, import tls-ca.crt as a trusted certificate agency certificate to verify server certificates into my browser, and copy tls.keyfile (containing both the private key and the certificate) to all three machines as /home/neunhoef/arangodb.keyfile with permissions 400.

Preparations with volumes

If you wanted to use special volumes for the database, now would be the time. You could have locally attached SSDs for particularly fast performance, or create network volumes. All you would have to do here would be to mount the volume in a directory and make it available to the user account under which you want to run the database.

I am lazy, I simply use my home directory and create a directory d there by doing

on all three machines.

Setting up the first cluster using ArangoDB 3.4.1

The only piece that is now missing is to create a shared secret such that the ArangoDB instances on the three machines can authenticate each other. To this end I just create a file /home/neunhoef/arangodb.secret with permissions 400 with these commands on all three machines:

Note that we have not used sudo or the root account in any way so far.

We could now fire up the cluster with a single command, but we want to do things properly, therefore we want to set up a systemd service for our cluster. This needs root privileges, though. Furthermore, this now needs a Linux variant which uses systemd, I used Ubuntu 18.04, for example.

We simply use

sudo vi /etc/systemd/system/arangodb.service

to create the following file in /etc/systemd/system/arangodb.service:

Note that since I am using the same directory layout on all three machines, the file can look exactly the same on all three machines, with a single exception. The option --starter.address of course must be the internal IP address of the particular machine on which we deploy this file. So this has to be adjusted for each file. Note that the --starter.join option allows to use a totally symmetric setup with the very same information for all three machines, namely a comma-separated list of all internal IP addresses and ports that the starters use.

After that we can immediately fire up the cluster with these commands (on all three machines):

The first of these commands makes systemd discover the new service file, the second starts the service with the name arangodb.

You can use these commands to diagnose any problems or see that it works:

Since I have put the public IP addresses of my three machines in the DNS server for my domain and have opened port 8529 in the firewall on all three machines (I used the Google Cloud Console for this), I could now immediately point my browser to https://a.9hoeffer.de:8529 and get the graphical UI of ArangoDB.

If you do this at home, do not forget to immediately change the password of the root user from empty to something sensible!

Setting up the second cluster using ArangoDB 3.4.0

One of the main purposes of this article is to demonstrate how to run multiple versions of ArangoDB on the same machine(s). Therefore I pick another such version, namely 3.4.0 (the only other 3.4 version which is available to me at the time of this writing). I can reuse the certificates and the secret, and therefore I essentially only do this on all machines to deploy the second cluster:

and edit /etc/systemd/system/arangodb2.service to look like this:

All I had to change was the Description, the path to the arangodb executable, the database directory /home/neunhoef/d2 as well as the ports used by the starters, which is now 9528. Therefore I had to add the option --starter.port=9528. Of course, I needed to open the port 9529 in the firewall as well. The port used by the database coordinator is always one greater than the port the starter is using.

After that,

fired up the second cluster.

Performing a rolling upgrade

We will conclude this post by showing how we can upgrade the second cluster smoothly to Version 3.4.1 as well. We are using the rolling upgrade feature which is built into the starter arangodb. We follow the procedure as documented in this manual section.

The first step is to install the new version. With the .tar.gz binary distribution this can simply be done by extracting the archive in some directory. In our particular case here, we have already installed Version 3.4.1, so nothing has to be done.

The second step is to stop the three old starters and restart them with the new executable. Note that only the starter processes arangodb should be restarted, not the arangod processes. This is important to avoid any service interruption.

We achieve this by editing the systemd service file in /etc/systemd/system/arangodb2.service and adjusting the path to the arangodb executable to be run from

to

Then, after doing

to tell systemd about the change, we can simply send

with PID replaced by the process ID of the running arangodb process.
Use

to find this process ID under Main ID. Do not use

since this would kill the arangod subprocesses, too! Furthermore, note that we intentionally added the line

to all systemd service files, otherwise systemd would kill all subprocesses alongside with the arangodb process.

Once the kill has happened, systemd will subsequently automatically restart the new arangodb executable which will pick up the running arangod processes.

Once this has been performed on all three machines, we are in a position to run the rolling upgrade. To this end, issue the following command on exactly one of the machines only:

This will contact the new starter process running on the same machine and all three starter processes on the three machines will work together to perform a smooth rolling upgrade to the new version. On the console you will see progress being logged as follows:

The mission has been accomplished.