Lockfree protection of data structures that are frequently read

Motivation

In multi-threaded applications running on multi-core systems, it occurs often that there are certain data structures, which are frequently read but relatively seldom changed. An example of this would be a database server that has a list of databases that changes rarely, but needs to be consulted for every single query hitting the database. In such situations one needs to guarantee fast read access as well as protection against inconsistencies, use after free and memory leaks.

Therefore we seek a lock-free protection mechanism that scales to lots of threads on modern machines and uses only C++11 standard library methods. The mechanism should be easy to use and easy to understand and prove correct. This article presents a solution to this, which is probably not new, but which we still did not find anywhere else.

The concrete challenge at hand

Assume a global data structure on the heap and a single atomic pointer P to it. If (fast) readers access this completely unprotected, then a (slow) writer can create a completely new data structure and then change the pointer to the new structure with an atomic operation. Since writing is not time critical, one can easily use a mutex to ensure that there is only a single writer at any given time. The only problem is to decide, when it is safe to destruct the old value, because the writer cannot easily know that no reader is still accessing the old values. The challenge is aggravated by the fact that without thread synchronization it is unclear, when a reader actually sees the new pointer value, in particular on a multi-core machine with a complex system of caches.

If you want to see our solution directly, scroll down to “Source code links“. We first present a classical good approach and then try to improve on it. (more…)

Posted in Architecture, Security | 4 Comments

Running V8 isolates in a multi-threaded ArangoDB database

ArangoDB allows running user-defined JavaScript code in the database. This can be used for more complex, stored procedures-like database operations. Additionally, ArangoDB’s Foxx framework can be used to make any database functionality available via an HTTP REST API. It’s easy to build data-centric microservices with it, using the scripting functionality for tasks like access control, data validation, sanitation etc.

We often get asked how the scripting functionality is implemented under the hood. Additionally, several people have asked how ArangoDB’s JavaScript functionality relates to node.js.

This post tries to explain that in detail.

(more…)

Posted in API, Architecture, C++, Documentation, nodejs | Leave a comment

Fulltext Index Enhancements

This post is about improvements for the fulltext index in ArangoDB 2.6. The improvements address the problem that non-string attributes were ignored when fulltext-indexing.

Effectively this prevented string values inside arrays or objects from being indexed. Though this behavior was documented, it was limited the usefulness of the fulltext index much. Several users requested the fulltext index to be able to index arrays and object attributes, too.

Finally this has been accomplished, so the fulltext index in 2.6 supports indexing arrays and objects!

Read on in Jan’s blog post about Fulltext Index Enhancements.

Posted in API, Architecture | Leave a comment

Agile development vs. schema enforcement – a paradox resolved

The fans of modern and agile software development usually propose to use schemaless database engines to allow for greater flexibility, in particular during the early rapid prototyping phase of IT projects. The more traditionally minded insist that having a strict schema that is enforced by the persistence layer throughout the lifetime of a project is necessary to ensure quality and security.
schema_enforcement
In this post I would like to explain briefly, why I believe that both groups are completely right and why this is not so paradoxical as it sounds at first glance. (more…)

Posted in Architecture | Leave a comment

Building Hypermedia APIs – a Design Approach using Statecharts

This is the second blog post on building hypermedia APIs with the focus on API design. In part 1 Lucas describes the concept of links in JSON.

Imagine we have an API where people can like books and other people can then see, who likes a certain book. We want this API to be highly connected: We don’t want to look up URLs in a documentation, we want to follow links as we know it from the world wide web. All we want to do as the author of the API is give our users a single URL from which they can then follow links to all other resources. This is similar to the way we would do this with a website. Leonard Richardson and Mike Amundsen refer to this as the billboard URL for this reason: If you put this URL on some billboard, people know everything to get started with your API. (more…)

Posted in API, Architecture, Foxx | Tagged , , , | Leave a comment

Modeling Data in MongoDB vs ArangoDB

MongoDB is a document DB whereas ArangoDB is a multi-model DB supporting documents, graphs and key/values within a single database. When it comes to data modeling and data querying, they pursue somewhat different approaches.


In a Nutshell: In MongoDB, data modeling is “aggregate-oriented”, avoiding relations and joins. On the other side, everybody has probably used relational databases which organize the data in tables with relations and try to avoid as much redundancy as possible. Both approaches have their pros and cons. ArangoDB is somewhat in-between: You can both model and query your data in a “relational way” but also in an “aggregate-oriented way”, depending on your use case. ArangoDB offers joins, nesting of sub-documents and multi-collection graphs. (more…)

Posted in Architecture, Documentation, Graphs, Query Language | Tagged , , , | 5 Comments

FullStack London

I recently had the chance to visit FullStack London, a well organized conference. Thanks a lot to Skills Matter. FullStack was opened by Douglas Crockford about “The Better Parts” of ES6. I cannot wait to start using them. Douglas was followed by Isaac Schlueter talking about open source in companies. Although this talk was not technical I learned a lot and it was very inspiring.

The remainder of the conference was all about using JavaScript mostly on server-side using Node.js or in robotics. As robotics is not my kind of topic I visited the talks about server-side JS. They confirmed my impression where JS development is heading to: Microservices. (more…)

Posted in Architecture, Conferences, Foxx, General, nodejs | Tagged , , | Leave a comment

Disk space usage with different journal sizes

A while ago we wrote some blog article that explained how ArangoDB uses disk space. That article compared the disk usage of ArangoDB, CouchDB, and MongoDB for loading some particular datasets. In this post, we’ll show in more detail the disk usage of ArangoDB for insert, update, and delete operations. We’ll also compare it to CouchDB for reference.
(more…)

Posted in Architecture, Performance | Leave a comment

Disk space usage in ArangoDB

In this post we’ll explain how ArangoDB stores collection data on disk and look at its storage space requirements, compared to other popular NoSQL databases such as CouchDB and MongoDB.

How ArangoDB allocates disk space

ArangoDB stores documents in collections. The collection data is persisted on disk so it does not get lost in case of a server restart.
(more…)

Posted in Architecture, General, Performance | 2 Comments

Is UNQL Dead?

Note: We changed the name of the database in May 2012. AvocadoDB is now called ArangoDB.

UNQL started with quite some hype last year. However, after some burst of activity the project came to a hold. So it seems, that – at least as a project – UNQL has been a failure. IMHO one of the major issues with the current UNQL is, that it tries to cover everything in NoSQL, from key-value stores to document-stores to graph-database. Basically you end up with greatest common divisor – namely key-value access. But with graph structures and also document-structures you really want to supports joins, paths or some sort of sub-structures.

Apart from all the technical and theoretical benefits of SQL and what advantages the underlying theory has to offer, the major plus from an users point of view is that it is readable. You simple can see an SQL statement – be it in C, Java, Ruby – and understand what is going on. It is declarative, not imperative. With other imperative solution, like a fluent interface or a map-reduce, you need to understand the underlying syntax or language. With SQL you only need to understand English – at least most of the time.

And here I think is where UNQL is totally right. We need something similar for the NoSQL world. But it should not try to be a “fits all situation”. It should be a fit for 80% of the problems. For simple key-values stores a fluent-interface is indeed enough. For very complex graph traversals a traversal program must be written. For very complex map-reduces you might need to write a program – but check out Google’s talk (www.nosql-matters.org/program) about NoNoSQL. There they describe why they are developing a SQL-like interface for Map/Reduce.

In my experience most of the time you have a set of collections holding different “types” of documents with some relations between them. One of the biggest advantages of document stores or graph databases is that you can have lists and sub-objects. The problem with SQL is, that it has no good way to deal with these structures. So I believe UNQL would be quite successful if it would concentrate on these strong advantages of NoSQL, instead of trying to unify everything – especially after hear Jan’s talk about a document query language at the NoSQL Cologne UG (an English version is also available).

Cheers

Frank

 

Posted in Architecture, Query Language | Tagged | 4 Comments