The new Javascript driver no longer imposes any promises implementation. It follows the standard callback pattern with a callback using err and res.

I wanted to give the new driver a try. A github crawler seemed like a good side-project, especially because the node-github driver follows the same conventions as the Javascript driver.

There are a lot of promise libraries out there. The most popular one – according to NPM – was promises. It should be possible to use any implementation. Therefore I used this one.

The following source code can be found on github.

Pagination with Promises made easy

The github driver has a function to get all followers. However, the result is paginated. With two helper functions and promises it is straight forward to implement a function to retrieve all followers of an user.

The followPages function simply extends the result with the next page until the last page is reached.

The promote error helper is a convenience function to bridge callbacks and promises.

I’ve decided to stick to the sequence reject (aka err) followed by resolve (aka res) – like the callbacks. The promoteError can be used for the github callback as well as the ArangoDB driver.

Queues, Queues, Queues

I’ve only needed a very simple job queue, so queue-it is a good choice. It provides a very simple API for handling job queues:

The new Javascript driver allows to access arbitrary endpoint. First install a Foxx implementing the queue microservice in an ArangoDB instance.

Adding a new job from node.js is now easy


I wanted to crawl users and their repos. The relations (“follows”, “owns”, “is_member”, “stars”) is stored in an edge collection. I only add an edge if it is not already there. Therefore I check inside a transaction, if the edge exists and add it, if it does not.

Please note that the action function is executed on the server and not in the nodejs client. Therefore we need to pass the relevant data as parameters. It is not possible to use the closure variables.

Riding the Beast

Start an ArangoDB instance (i.e. inside a docker container) and install the simple queue.

Start the arangosh and create collections users, repos and relations.

Now everything is initialized. Fire up nodejs and start crawling.

Please keep in mind that this is just an experiment. There is no good error handling and convenience functions for setup and start. It is also not optimized for performance. For instance, it would easily be possible to avoid nodejs / ArangoDB roundtrips using more transactions.

Sources used in this example:

The source code of this example is available from Github:

If you want to continue with other JavaScript related resources, you should start with ArangoDB NoSQL and JavaScript.