Open Source DC/OS: The modern way to run a distributed database

The mission of ArangoDB is to simplify the complexity of data work. ArangoDB is a distributed native multi-model NoSQL database that supports JSON documents, graphs and key-value pairs in one database engine with one query language. The cluster management is based on Apache Mesos, a battle-hardened technology. With the launch of DC/OS by a community of more than 50 companies all ArangoDB users can easily scale.

Just a little while ago setup, management, and maintenance of a database cluster was just a world of pain. Everybody who has put effort into getting automatic failover to work or who updated their database cluster know what I am talking about. Many of us may have experienced calls at 4 am in the morning notifying us that something within the cluster just went bad. Say hello to the Fail Whale. (more…)

Posted in Architecture, General | Leave a comment

How to put ArangoDB to Spartan-Mode

Most of us saw the fantastic movie 300 (I did it last night…again) or at least read the comics. 300 spartans barely wearing anything but achieving a lot. This little how-to will show you how to put ArangoDB into Spartan-Mode and thereby reduce memory-footprint and CPU usage.

Big thanks to Conrad from L.A. for his time and for giving us the impulse for this little how-to!

Background

Recently, we had a lot of cool talks with users who are new to ArangoDB or doing their PoC at the moment. Two topics came up several times. First ArangoDB’s memory footprint which is higher compared to some other DBs. We stated this already in our latest performance benchmark. The second issue is the CPU usage in standby mode. Users claimed 4-6% CPU usage when ArangoDB isn’t really doing anything.

First of all we use Google’s V8 engine for our JS framework Foxx. At first glance it might seem that V8 is memory thirsty but most of the memory is virtual memory and therefore does not actually fill up your physical RAM at all. If you don’t need V8 (not using Foxx) then you can run ArangoDB with less V8 contexts which reduces the memory usage significantly.

(more…)

Posted in Foxx, General, how to | Leave a comment

Linenoise NG – a BSD licensed readline replacement with UTF8 support

For projects that are BSD or Apache 2 licensed, Linenoise (by Salvatore Sanfilippo) is a pretty small, portable GNU readline (GPL) replacement. Based on the work of Salvatore and 10gen Inc. this Linenoise NG implementation adds UTF8 and Windows support, uses a BSD license and can be used in any kind of program.

Features:

  • single-line and multi-line editing mode with the usual key bindings implemented
  • history handling
  • completion
  • BSD license source code
  • Only uses a subset of VT100 escapes (ANSI.SYS compatible)
  • UTF8 aware
  • support for Linux, MacOS and Windows

Linenoise NG deviates from Salvatore’s original goal to have a minimal readline replacement for the sake of supporting UTF8 and Windows. It deviates from 10gen Inc.’s goal to create a C++ interface to linenoise. This library uses C++ internally, but to the user it provides a pure C interface that is compatible with the original linenoise API.

Contributions welcome! See the repository on Github for details.

Posted in General | Tagged , | Leave a comment

ArangoDB 2.8.2 with Replication Improvements

ArangoDB 2.8.2 maintenance release comes with several replication improvements and bug fixes. You can download the latest version from our download page.

What’s changed:

  • the continuous replication applier will now prevent the master’s WAL logfiles from being removed if they are still needed by the applier on the slave. This should help slaves that suffered from masters garbage collection WAL logfiles which would have been needed by the slave later.

    The initial synchronization will block removal of still needed WAL logfiles on the master for 10 minutes initially, and will extend this period when further requests are made to the master. Initial synchronization hands over its handle for blocking logfile removal to the continuous replication when started via the setupReplication function. In this case, continuous replication will extend the logfile removal blocking period for the required WAL logfiles when the slave makes additional requests.

    All handles that block logfile removal will time out automatically after at most 5 minutes should a master not be contacted by the slave anymore (e.g. in case the slave’s replication is turned off, the slaves loses the connection to the master or the slave goes down).

  • added all-in-one function setupReplication to synchronize data from master to slave and start the continuous replication:

    The command will return when the initial synchronization is finished and the continuous replication has been started, or in case the initial synchronization has failed. (more…)

Posted in General, Releases, Replication | Leave a comment

Small Things in ArangoDB 2.8: Explain Improvements, POW, Arangoimp

Explain Improvements

Explaining AQL queries becomes even easier in ArangoDB 2.8. While previous versions required writing a hard-to-memorize command like

to explain an AQL query from the ArangoShell, 2.8 reduces this task to a mere

Apart from that, explain in 2.8 is smarter when confronted with very lengthy query strings, and with queries that contain huge hard-coded string, array, or object values. (more…)

Posted in General, Query Language, Releases | Leave a comment

Killing a long-running query

Suppose there is an AQL query that’s executing in the server for a long time already and you want to get rid of it. What can be done to abort that query?

If a connection to the server can still be established, the easiest is to use the ArangoShell to fetch the list of currently executing AQL queries and send a kill command to the server for the correct query.

To start, we can fetch the list of all running queries and print their ids, query strings and runtimes. This is only inspection and does not abort any query:

Here’s an example result for the list of running queries:

To now kill a query from the list, we can pass the query’s id to kill:

If a query was actually killed on the server, that call should return without an error, and the server should have logged a warning in addition. (more…)

Posted in General, Query Language | Leave a comment

ArangoDB 2.8 w/ AQL Graph Traversals, Array Indexes, Aggregation

We welcome 2016 with our first big news yet – the release of ArangoDB 2.8!

Now you can use new AQL keywords to traverse a graph even more convenient – a big deal for those who like to get the maximum out of their connected data. ArangoDB is getting faster with every iteration, in this release we have implemented several AQL functions and arithmetic operations in super-fast C++ code, optimizer rules and indexing improved further to help you getting things done faster. Download ArangoDB 2.8 here.

Array Indexes

The added Array Indexes are a major improvement to ArangoDB that you will love and never want to miss again. Hash indexes and skiplist indexes can now be defined for array values as well, so it’s freaking fast to access documents by individual array values. Let assume you want to retrieve articles that are tagged with “graphdb”, you can now use an index on the tags array:

An added hash-index on tags (ensureHashIndex("tags[*]")) can be used for finding all documents having "graphdb" somewhere in their tags array using the following AQL query:

Have fun with these new indexes!


AQL Graph Traversal

Next, the mentioned AQL graph traversals. The query language AQL adds the keywords GRAPH, OUTBOUND, INBOUND and ANY for use in graph traversals. Using plain AQL in ArangoDB 2.8 you can create a shopping list for your friends birthday gifts, related to products they already own and up to 5 ideas ordered by price.

(more…)

Posted in General, Releases | 4 Comments

ArangoDB 2.7.3 – Maintenance release

ArangoDB 2.7.3 – a maintenance release – is ready for download.

Changes:

  • fixed disappearing of documents for collections transferred via sync or syncCollection if the collection was dropped right before synchronization and drop and (re-)create collection markers were located in the same WAL file

  • fixed an issue where overwriting the system sessions collection would break the web interface when authentication is enabled

Posted in General | Leave a comment

Using Bind Parameters in the AQL Editor

The AQL editor in the web interface is useful for running ad hoc AQL queries and trying things out. It provides a feature to explain the query and inspect its execution plan. This can be used to check if the query uses indexes, and which.

So far the AQL editor only supported using query string literals, but it lacked support for bind parameters. Queries issued by application code however often will use bind parameters for security reasons. Often enough this prevented copying & pasting queries from the application code into the AQL editor and vice versa without making manual adjustments.

This has been fixed in the upcoming ArangoDB version 2.8 (currently in development). Bind parameters can now be used in the AQL editor as well. Bind parameters can be entered as JSON values, the same format that is used for bind parameters in the HTTP REST API and in (JavaScript) application code.

The queries can also be saved in the AQL editor along with their bind parameter values for later reuse.

Screenshot from the feature in 2.8:

bind-parameters

Posted in API, General | Leave a comment

GA of ArangoDB 2.7 – Big + for Indexes, Throughput, AQL and Foxx

Long awaited and now we´ve finished it! New major release of ArangoDB 2.7 is ready for download. First of all a big thanks to our community for your great support! We´ve implemented a lot of your ideas! After your feedback to RC1 and RC2 we are happy to bring a new major release to the world. With ArangoDB 2.7 we increased our performance even further and improved query handling a lot.

What big improvements are in for you?

Index buckets

  • The primary indexes and hash indexes of collections can now split into multiple index buckets.

Throughput Enhancements

  • A lot is not enough. Throughput is another key requirement for a premium database. Again we pushed our throughput a big step forward with 2.7.

AQL Improvements – Ease of Use and Performance

  • Our goal was to further shorten and ease the writing of statements. AQL has always been an efficient and intuitive query language similar to SQL but with ArangoDB 2.7 AQL got even better.

Find a detailed overview in our blogpost about RC1.

Furthermore we fixed some issues and enabled Foxx apps to be installed underneath URL path /_open/, so they can be (intentionally) accessed without authentification. The extensibility for your data-centric microservices got even bigger.

(more…)

Posted in General | Leave a comment