ArangoDB 3.0 alpha5: Step-by-step instruction to setup an ArangoDB test cluster on DC/OS

As we get closer to the final release of ArangoDB 3.0 you can now test our brand new cluster capabilities with 3.0 alpha5.

In this post you will find a step-by-step instruction on how-to get a 3.0 alpha3 cluster up and running on Mesosphere DC/OS for testing. Any feedback is well appreciated. Please use our Slack Channel “feedback30” for your hints, ideas and all other wisdom you´d like to share with us.

Please note that we are well aware of the “clunkyness” of the current cluster setup in 3.0alpha5. The final release of 3.0 will make it much easier.

Launching the DC/OS cluster on AWS

Go to this page and choose a cloud formation template in a region of your choice. It does not matter whether you use “Single Master” or “HA: Three Master”. Once you click you will be taken to the AWS console login and then be presented with various choices. Do not bother with public slaves, just select a number of private slaves.

This will take approximately 15 minutes and in the end you have a Mesos Master instance and some Mesos Agents. (more…)

Posted in General | Leave a comment

Getting closer: ArangoDB 3.0 alpha release

There is this German saying “If it takes long enough, it will be all right in the end.” However, since just “all right” isn’t our quality standard this first alpha of 3.0 took us a bit longer to finish up than planned. We´d like to invite you to give this fully tested alpha a serious spin, test the new functionalities and share your thoughts and feedback with us on Slack in our “feedback30” channel

Within this short release note you´ll find 1) a quick overview of the most important changes; 2) an instruction on how to get the new version and 3) how to get your (test) data from your 2.x version into the 3.0 alpha which has our new binary storage format VelocyPack implemented.

For those who haven´t read about VelocyPack yet: We successfully did a kind of open heart surgery on the ArangoDB core by implementing our very own binary storage format VelocyPack. VelocyPack is even more compact than MessagePack and is very promising to further improve ArangoDBs query response time and memory usage. Before VelocyPack we used two formats (ShapeJSON, tri_json_t) internally which led to a lot of duplicate code. We tested existing formats but didn´t find one that met our needs. Now that we only have one storage format we can simplify and, hopefully, speedup our development cycle significantly.

We think that our alpha version is definitely worth testing. 3.0 alpha consists of 173.000 lines of code: not to mention the 290.000 lines of testcode… all our tests are green and even our performance tests are running smoothly (benchmark results are looking good, we´ll post a new performance benchmark blogpost with the 3.0 release). (more…)

Posted in General, Releases | Tagged , , | Leave a comment

Open Source DC/OS: The modern way to run a distributed database

The mission of ArangoDB is to simplify the complexity of data work. ArangoDB is a distributed native multi-model NoSQL database that supports JSON documents, graphs and key-value pairs in one database engine with one query language. The cluster management is based on Apache Mesos, a battle-hardened technology. With the launch of DC/OS by a community of more than 50 companies all ArangoDB users can easily scale.

Just a little while ago setup, management, and maintenance of a database cluster was just a world of pain. Everybody who has put effort into getting automatic failover to work or who updated their database cluster know what I am talking about. Many of us may have experienced calls at 4 am in the morning notifying us that something within the cluster just went bad. Say hello to the Fail Whale. (more…)

Posted in Architecture, General | Leave a comment

How to put ArangoDB to Spartan-Mode

Most of us saw the fantastic movie 300 (I did it last night…again) or at least read the comics. 300 spartans barely wearing anything but achieving a lot. This little how-to will show you how to put ArangoDB into Spartan-Mode and thereby reduce memory-footprint and CPU usage.

Big thanks to Conrad from L.A. for his time and for giving us the impulse for this little how-to!

Background

Recently, we had a lot of cool talks with users who are new to ArangoDB or doing their PoC at the moment. Two topics came up several times. First ArangoDB’s memory footprint which is higher compared to some other DBs. We stated this already in our latest performance benchmark. The second issue is the CPU usage in standby mode. Users claimed 4-6% CPU usage when ArangoDB isn’t really doing anything.

First of all we use Google’s V8 engine for our JS framework Foxx. At first glance it might seem that V8 is memory thirsty but most of the memory is virtual memory and therefore does not actually fill up your physical RAM at all. If you don’t need V8 (not using Foxx) then you can run ArangoDB with less V8 contexts which reduces the memory usage significantly.

(more…)

Posted in Foxx, General, how to | Leave a comment

Linenoise NG – a BSD licensed readline replacement with UTF8 support

For projects that are BSD or Apache 2 licensed, Linenoise (by Salvatore Sanfilippo) is a pretty small, portable GNU readline (GPL) replacement. Based on the work of Salvatore and 10gen Inc. this Linenoise NG implementation adds UTF8 and Windows support, uses a BSD license and can be used in any kind of program.

Features:

  • single-line and multi-line editing mode with the usual key bindings implemented
  • history handling
  • completion
  • BSD license source code
  • Only uses a subset of VT100 escapes (ANSI.SYS compatible)
  • UTF8 aware
  • support for Linux, MacOS and Windows

Linenoise NG deviates from Salvatore’s original goal to have a minimal readline replacement for the sake of supporting UTF8 and Windows. It deviates from 10gen Inc.’s goal to create a C++ interface to linenoise. This library uses C++ internally, but to the user it provides a pure C interface that is compatible with the original linenoise API.

Contributions welcome! See the repository on Github for details.

Posted in General | Tagged , | Leave a comment

ArangoDB 2.8.2 with Replication Improvements

ArangoDB 2.8.2 maintenance release comes with several replication improvements and bug fixes. You can download the latest version from our download page.

What’s changed:

  • the continuous replication applier will now prevent the master’s WAL logfiles from being removed if they are still needed by the applier on the slave. This should help slaves that suffered from masters garbage collection WAL logfiles which would have been needed by the slave later.

    The initial synchronization will block removal of still needed WAL logfiles on the master for 10 minutes initially, and will extend this period when further requests are made to the master. Initial synchronization hands over its handle for blocking logfile removal to the continuous replication when started via the setupReplication function. In this case, continuous replication will extend the logfile removal blocking period for the required WAL logfiles when the slave makes additional requests.

    All handles that block logfile removal will time out automatically after at most 5 minutes should a master not be contacted by the slave anymore (e.g. in case the slave’s replication is turned off, the slaves loses the connection to the master or the slave goes down).

  • added all-in-one function setupReplication to synchronize data from master to slave and start the continuous replication:

    The command will return when the initial synchronization is finished and the continuous replication has been started, or in case the initial synchronization has failed. (more…)

Posted in General, Releases, Replication | Leave a comment

Small Things in ArangoDB 2.8: Explain Improvements, POW, Arangoimp

Explain Improvements

Explaining AQL queries becomes even easier in ArangoDB 2.8. While previous versions required writing a hard-to-memorize command like

to explain an AQL query from the ArangoShell, 2.8 reduces this task to a mere

Apart from that, explain in 2.8 is smarter when confronted with very lengthy query strings, and with queries that contain huge hard-coded string, array, or object values. (more…)

Posted in General, Query Language, Releases | Leave a comment

Killing a long-running query

Suppose there is an AQL query that’s executing in the server for a long time already and you want to get rid of it. What can be done to abort that query?

If a connection to the server can still be established, the easiest is to use the ArangoShell to fetch the list of currently executing AQL queries and send a kill command to the server for the correct query.

To start, we can fetch the list of all running queries and print their ids, query strings and runtimes. This is only inspection and does not abort any query:

Here’s an example result for the list of running queries:

To now kill a query from the list, we can pass the query’s id to kill:

If a query was actually killed on the server, that call should return without an error, and the server should have logged a warning in addition. (more…)

Posted in General, Query Language | Leave a comment

ArangoDB 2.8 w/ AQL Graph Traversals, Array Indexes, Aggregation

We welcome 2016 with our first big news yet – the release of ArangoDB 2.8!

Now you can use new AQL keywords to traverse a graph even more convenient – a big deal for those who like to get the maximum out of their connected data. ArangoDB is getting faster with every iteration, in this release we have implemented several AQL functions and arithmetic operations in super-fast C++ code, optimizer rules and indexing improved further to help you getting things done faster. Download ArangoDB 2.8 here.

Array Indexes

The added Array Indexes are a major improvement to ArangoDB that you will love and never want to miss again. Hash indexes and skiplist indexes can now be defined for array values as well, so it’s freaking fast to access documents by individual array values. Let assume you want to retrieve articles that are tagged with “graphdb”, you can now use an index on the tags array:

An added hash-index on tags (ensureHashIndex("tags[*]")) can be used for finding all documents having "graphdb" somewhere in their tags array using the following AQL query:

Have fun with these new indexes!


AQL Graph Traversal

Next, the mentioned AQL graph traversals. The query language AQL adds the keywords GRAPH, OUTBOUND, INBOUND and ANY for use in graph traversals. Using plain AQL in ArangoDB 2.8 you can create a shopping list for your friends birthday gifts, related to products they already own and up to 5 ideas ordered by price.

(more…)

Posted in General, Releases | 4 Comments

ArangoDB 2.7.3 – Maintenance release

ArangoDB 2.7.3 – a maintenance release – is ready for download.

Changes:

  • fixed disappearing of documents for collections transferred via sync or syncCollection if the collection was dropped right before synchronization and drop and (re-)create collection markers were located in the same WAL file

  • fixed an issue where overwriting the system sessions collection would break the web interface when authentication is enabled

Posted in General | Leave a comment