The need for replicated logs and state machines, resilient to local failures and network partitions, has led to the development of the Paxos and more recently of the RAFT consensus protocols. While many very good implementations of RAFT have been developed by the community over the last years, we demonstrate that significant added value can be derived from immersing an application framework into the consensus realm.
The real world implementation of such a consensus appliance, regardless of the seeming simplicity of RAFT, proves to be very tricky in detail. One should preferably rely on tested and battle-hardened implementations. At ArangoDB however, we realised that there is more to be gained from RAFT, if we allowed for the consensus cluster to be extended in functionality through custom user apps.
Arango speak for consensus is “Agency”. The agency is formed through a limited number of specialised database nodes called agents. In addition to serving the replicated log and state machine, the agency performs the supervision and failover of servers, shards and of the foxx application framework.
The agency can be deployed independently of ArangoDB clusters as a consensus state machine and allow users to expand upon the RAFT functionality with their own logic in runtime compiled Javascript. Such expansions can rely upon the strong guarantees of consensus for their respective internal state structures. Deployment of such programs and REST API expansions is straightforward. Some intriguingly simple examples are demonstrated live to highlight the key advantages over other RAFT implementations.
