arangobench is ArangoDB’s benchmark and test tool. It can be used to issue test requests to the database system for performance and server function testing. It supports parallel querying and batch requests.
arangobench is a client tool which makes network connections to an ArangoDB server in about the same way as a client application would do via an ArangoDB client driver. It thus often provides good enough throughput and performance estimates. It provides different test cases that can be executed, that reflect a broader set of use cases. It is useful to pick and run the test cases that most closely resemble typical or expected workloads.
arangobench can be run on the same host as the ArangoDB server, or on a different host. When using it against a cluster, it must be connected to one of the cluster’s Coordinators. It can communicate over normal (unencrypted) TCP connections and encrypted SSL/TLS connections.
The most important general arangobench options are:
--server.endpoint: the server endpoint to connect to. This can be a remote server or a server running on the same host. The endpoint also specifies whether encryption at transit (TLS) should be used.
--server.password: these can be used to authenticate with an existing ArangoDB installation.
--test-case: selects the test case to be executed by arangobench. A list of the available test cases can be retrieved by running arangobench with the
--helpoption. For detailed descriptions see Test Cases.
--requests: total number of requests to be executed by arangobench in the selected test case. If batching is used, multiple operations will still be counted individually, even though they may be sent together in a single request.
--runs: number of test case runs to perform. This option defaults to
1, but it can be increased so that result outliers have less influence on the test results.
General options that can affect test case performance and throughput:
--concurrency: number of parallel threads to use by arangobench. Increasing this value should normally increase throughput, unless some saturation or congestion is reached in either arangobench itself, the network layer or the ArangoDB instance. If increasing
--concurrencydoes not improve throughput or even decrease it, it should be determined which part of the setup is the bottleneck.
--wait-for-sync: whether or not all write operations performed by test cases should be executed with the
waitForSyncflag. If this is true, write operations are blocking and only return once they have been fully acknowledged by the server’s disk subsystem(s). Setting
truewill have a large negative impact on write performance and is thus not recommended for most use cases.
--async: if set to
true, it will make arangobench send fire-and-forget requests. These requests will be responded directly after having been added to the server’s request processing queue. arangobench will not wait for the operation to be fully executed on the server side. All it checks for is whether the server was able to queue the operation(s). The arangobench test case may complete before the queued operations have been fully processed on the server. In addition, sending more requests than the server can handle for a prolonged time may lead to the server’s scheduler queue filling up. The server-side scheduler queue has a limited capacity, and once it is full, any further incoming requests will be rejected by the server with HTTP 503 “Service unavailable” until there is again some capacity in the queue.
--batch-size: by default, arangobench will send one HTTP request per test case operation. This is often okay for test cases that execute a certain AQL query or such, when there is naturally no other request to batch the query with. However, in some use cases multiple operations can actually be sent together in a single HTTP request. The prime example for this is bulk-inserting documents, which are normally sent to the server in batches by client programs anyway. Any value greater than
1will make arangobench send batch requests. Using batching should normally increase the throughput.
--complexity: some test cases can be adjusted via the
--complexityparameter, which often controls the number of document attributes that are inserted in document-centric test cases.
--keep-alive: whether or not arangobench should use HTTP keep-alive connections. This should always be turned on.
Important cluster-specific options are:
--number-of-shards: number of shards for collections created by arangobench. This option is only meaningful for test cases that create collections.
--replication-factor: number of replicas for each shard in collections created by arangobench. This option is only meaningful for test cases that write into collections. The larger the replication factor is, the more expensive write operations will become.
arangobench provides the following predefined test cases. The test case to be
executed can be selected via the
--test-case startup option.
Note that these test cases have been added over time, and not all of them may be fully appropriate for a given workload test.
In order to benchmark custom AQL queries, the appropriate test case is
||performs AQL queries that insert one document per query. The
||creates 3 empty collections and then performs different AQL read queries on these collections, partially using joins. This test was once used to test shard locking, but is now largely obsolete. In a cluster, it still provides a little value because it effectively measures query setup and shutdown times for concurrent AQL queries.|
||performs AQL queries that insert one document per query. The
||creates as many separate (empty) collections as provided in the value of
||executes a custom AQL query, that can be specified either via the
||will perform a mix of insert, update, get and remove operations for documents. 20% of the operations will be single-document inserts, 20% of the operations will be single-document updates, 40% of the operations are single-document read requests, and 20% of the operations will be single-document removals. There will be a total of
||will perform a mix of insert, update and get operations for documents. 25% of the operations will be single-document inserts, 25% of the operations will be single-document updates, and 50% of the operations are single-document read requests. There will be a total of
||will perform a 50-50 mix of insert and retrieval operations for documents. 50% of the operations will be single-document inserts, 50% of the operations will be single-document read requests. There will be a total of
||performs single-document insert operations via the specialized insert API (in contrast to performing inserts via generic AQL). The
||will perform a mix of insert, update and get operations for edges. 25% of the operations will be single-edge inserts, 25% of the operations will be single-edge updates, and 50% of the operations are single-edge read requests. There will be a total of
||will perform a mix of insert, update and get operations for documents. The collection created by this test does have an extra, non-unique, non-sparse
||performs multi-document imports using the specialized import API (in contrast to performing inserts via generic AQL). Each inserted document will have two attributes. The
||will perform a mix of insert, get and remove operations for documents with randomized attribute names. 33% of the operations will be single-document inserts, 33% of the operations will be single-document reads, and 33% of the operations are single-document removals. There will be a total of
||will perform a mix of insert, get and remove operations for documents with different, but predictable attribute names. 33% of the operations will be single-document inserts, 33% of the operations will be single-document reads, and 33% of the operations are single-document removals. There will be a total of
||will perform a mix of insert and get operations for documents with randomized attribute names. 50% of the operations will be single-document inserts, and 50% of the operations will be single-document reads. There will be a total of
||identical to the
||creates 500 documents in a collection, and then performs a mix of AQL update queries (all on the same document) and a streaming AQL query that returns all documents from the collection. The
||queries the server version and then instantly returns. In a cluster, this means that Coordinators instantly respond to the requests without ever accessing DB-Servers. This test can be used to establish a baseline for single server or Coordinator throughput. The
The test cases provided by arangobench vary significantly in how they perform operations. For example, inserting documents into ArangoDB can be achieved by either:
- single-document inserts
(one document per request to
- multi-document inserts
(multiple documents per request to
- single-document AQL insert queries
(one document per AQL query to
- multi-document AQL insert queries
(multiple documents per AQL query to
(one or multiple documents per transaction to
- other ways omitted here, such as Stream Transactions, because none of the current test cases makes use of them
Many test cases can benefit from using request batching, that can be turned on
in arangobench via the
--batch-size option. Batching makes sense in cases
where a client application would also send multiple operations in a single
request, e.g. when inserting documents in bulk. Batching is often the easiest
way to improve the throughput.
--concurrency for a given benchmark does not increase
throughput or even decrease it, it is likely that some saturation or congestion
is occurring somewhere in the stack.
On Linux systems, running
top during the tests on the participating hosts
should reveal details about CPU usage (user, system, iowait) and memory usage.
This can be used as a quick first probe to see if any of the hosts is maxed out
on a particular resource (CPU power, available RAM, I/O throughput).
The ArangoDB servers also provide detailed metrics about CPU usage, I/O wait,
memory usage etc., which can be monitored during the benchmarks. The most
useful way of analyzing metrics is to have them scraped automatically by a
Prometheus instance and to make them available via Grafana. This allows metrics
to be collected over time and to compare them for multiple test runs with
different configurations. If Prometheus/Grafana cannot be used, the ArangoDB
web interface also provides a way to access the current values of all metrics
for privileged accounts.
A few useful metrics on the ArangoDB server side include:
arangodb_process_statistics_resident_set_size: resident memory usage for an arangod process, in bytes. Can be used to determine how much memory was used during the tests. If the memory usage is close to the capacity limit, this indicates that adding more memory could speed things up.
arangodb_process_statistics_resident_set_size_percent: resident memory usage for an arangod process, in percent of available RAM. Can be used to determine how much memory was used during the tests. If the memory usage is close to the capacity limit, this indicates that adding more memory could speed things up.
arangodb_scheduler_queue_full_failures_total: contains the number of rejected requests because the scheduler’s queue capacity was exceeded. If this contains values other than
0, it indicates overload (i.e. more requests coming in than can be handled by the server).
arangodb_scheduler_queue_length: number of requests in scheduler queue at a given point. This is expected to deviate from
0if requests are queued, but the closer it is to the configured maximum queue length (default value:
4096), the more close the server is to its processing capacity limit.
arangodb_server_statistics_system_percent: on Linux, provides total %sys CPU time for a host (most likely used by arangod or arangobench).
arangodb_server_statistics_user_percent: on Linux, provides total %user CPU time for a host (most likely used by arangod or arangobench).
arangodb_server_statistics_iowait_percent: on Linux, provides total %iowait CPU time for a host (most likely used by arangod or arangobench).
Another thing to check is the arangobench location: if arangobench runs on the same host as the ArangoDB server, arangobench and arangod may compete for the same resources. This may be appropriate if the production use case will also use a localhost setup. If this is not the case, arangobench should be executed on a separate host and send its request over the network, in the same way as the client applications are expected to do later.