Milestone 1 ArangoDB 3.3: Datacenter to Datacenter Replication

02General, ReleasesTags:

Every company needs a disaster recovery plan for all important systems. This is true from small units like single processes running in some container to the largest distributed architectures. For databases in particular this usually involves a mixture of fault-tolerance, redundancy, regular backups and emergency plans. The larger a data store, the more difficult is it to come up with a good strategy.

Therefore, it is desirable to be able to run a distributed database in one datacenter and replicate all transactions to another datacenter in some way. Often, transaction logs are shipped over the network to replicate everything in another, identical system in the other datacenter. Some distributed data stores have built-in support for multiple datacenter awareness and can replicate between datacenters in a fully automatic fashion.

This post gives an overview over the first evolutionary step of ArangoDB towards multi-datacenter support, which is asynchronous datacenter to datacenter replication.

Read more

Sorting number strings numerically

00GeneralTags:

Recently I gave a talk about ArangoDB in front of a community of mathematicians. I advertised that nearly arbitrary data can “easily” be stored in a JSON based document store. The moment I had uttered the word “easily”, one of them asked about long integers. And if a mathematician says “long integer” they do not mean 64bit but “properly long”. He actually wanted to store orders of finite groups. I said one should use a JSON UTF-8 string for this but I should have seen the next question coming because he then wanted that a sorted index would actually sort the documents by the numerical value stored in the string. But most databases – and ArangoDB is no exception here – will compare UTF-8 strings lexicographically (dictionary order). Read more

The new Satellite Collections Feature of ArangoDB

00General

With the new Version 3.2 we have introduced a new feature called “Satellite Collections”. This post explains what this is all about, how it can help you, and explains a concrete use case for which it is essential.

Background and Overview

Join operations are very useful but can be troublesome in a distributed database. This is because quite often, a join operation has to bring together different pieces of your data that reside on different machines. This leads to cluster internal communication and can easily ruin query performance. As in many contexts nowadays, data locality is very important to avoid such headaches. There is no silver bullet, because there will be many cases in which one cannot do much to improve data locality.

One particular case in which one can achieve something, is if you need a join operation between a very large collection (sharded across your cluster) and a small one, because then one can afford to replicate the small collection to every server, and all join operations can be executed without network communications.

Read more

Starting an ArangoDB cluster the easy way

01cluster, GeneralTags: ,

Recently, we have got a lot of feedback about the fact that standing up an ArangoDB cluster “manually” is an awkward and error-prone affair. We have been aware of this for some time, but always expected that most users running ArangoDB clusters would do so on Apache Mesos or DC/OS, where deployment is a breeze due to our ArangoDB framework.

However, for various valid reasons people do not want to use Apache Mesos and thus are back to square one with the problem of deploying an ArangoDB cluster without Apache Mesos.

Manual cluster set-up

So we have listened to this, and have looked what other distributed databases offer, and have put together a tool called arangodb (as opposed to arangod) to help you. It essentially gives you the following experience: Read more

Deploying an ArangoDB 3 Cluster with 2 Clicks

00cluster, DC/OSTags: ,

Hurray! Last week finally saw the release of ArangoDB 3.0 with lots of new features and in particular various improvements for ArangoDB clusters. In this blog post, I want to talk about one aspect of this, which is deployment.

DC/OS

As of last Wednesday, deploying an ArangoDB 3.0 cluster on DC/OS has become even simpler, because the new version of our framework scheduler has been accepted to the DC/OS Universe. Therefore, deployment is literally only two clicks: Read more

Running ArangoDB 3.0.0 on a DC/OS cluster

00Architecture, clusterTags: ,

As you surely recognized we´ve released ArangoDB 3.0 a few days ago. It comes with great cluster improvements like synchronous replication, automatic failover, easy up- and downscaling via the graphical user interface and with lots of other improvements. Furthermore, ArangoDB 3 is even better integrated with Apache Mesos and DC/OS. Read more

Lockfree protection of data structures that are frequently read

04Architecture, Security

Motivation

In multi-threaded applications running on multi-core systems, it occurs often that there are certain data structures, which are frequently read but relatively seldom changed. An example of this would be a database server that has a list of databases that changes rarely, but needs to be consulted for every single query hitting the database. In such situations one needs to guarantee fast read access as well as protection against inconsistencies, use after free and memory leaks.

Therefore we seek a lock-free protection mechanism that scales to lots of threads on modern machines and uses only C++11 standard library methods. The mechanism should be easy to use and easy to understand and prove correct. This article presents a solution to this, which is probably not new, but which we still did not find anywhere else.

The concrete challenge at hand

Assume a global data structure on the heap and a single atomic pointer P to it. If (fast) readers access this completely unprotected, then a (slow) writer can create a completely new data structure and then change the pointer to the new structure with an atomic operation. Since writing is not time critical, one can easily use a mutex to ensure that there is only a single writer at any given time. The only problem is to decide, when it is safe to destruct the old value, because the writer cannot easily know that no reader is still accessing the old values. The challenge is aggravated by the fact that without thread synchronization it is unclear, when a reader actually sees the new pointer value, in particular on a multi-core machine with a complex system of caches.

If you want to see our solution directly, scroll down to “Source code links“. We first present a classical good approach and then try to improve on it. More info

Slides: Is multi-model the future of NoSQL?

00Future-of-nosql, Presentation

Here is a slideshare and recording of my talk about multi-model databases, presented in Santa Clara earlier this month.

Abstract: Recently a new breed of “multi-model” databases has emerged. They are a document store, a graph database and a key/value store combined in one program. Therefore they are able to cover a lot of use cases which otherwise would need multiple different database systems. This approach promises a boost to the idea of “polyglot persistence“, which has become very popular in recent years although it creates some friction in the form of data conversion and synchronisation between different systems. This is, because with a multi-model database one can enjoy the benefits of polyglot persistence without the disadvantages.

In this talk I will explain the motivation behind the multi-model approach, discuss its advantages and limitations, and will then risk to make some predictions about the NoSQL database market in five years time. More info

Do you like ArangoDB?
icon-githubStar this project on GitHub.
Star ArangoDB on GitHub