ArangoDB Server Metrics
arangod exports metrics which can be used to monitor the healthiness and performance of the system. Out of all exposed metrics the most relevant ones are highlighted below. In addition, the thresholds for alerts are described.
The list of exposed metrics is subject to change in every minor version. While they should stay backwards compatible for the most part, some metrics are coupled to specific internals that may be replaced by other mechanisms in the future.
Below monitoring recommendations are limited to those metrics that are considered future-proof. If you setup your monitoring to use the recommendations described here, you can safely upgrade to new versions.
Cluster Health
This group of metrics are used to measure how healthy the cluster processes are and if they can communicate properly to one another.
Heartbeats
Heartbeats are a core mechanism in ArangoDB Clusters to define the liveliness of Servers. Every server will send heartbeats to the agency, if too many heartbeats are skipped or cannot be delivered in time, the server is declared dead and a failover of the data will be triggered. By default we expect at least 1 heartbeat per second. If a server does not deliver 5 heartbeats in a row (5 seconds without a single heartbeat) it is considered dead.
Metric
arangodb_heartbeat_send_time_msec: The time a single heartbeat took to be delivered.arangodb_heartbeat_failures: Amount of heartbeats which this server failed to deliver.
Exposed by Coordinator, DB-Server
Threshold
- For
arangodb_heartbeat_send_time_msec:- Depending on your network latency, we typically expect this to be somewhere below 100ms.
- below 1000ms is still acceptable but not great.
- 1000ms - 3000ms is considered critical, but cluster should still operate, consider contacting our support.
- above 3000ms expect outages! If any of these fails to deliver, the server will be flagged as dead and we will trigger failover. With this timing the failovers will most likely stack up and cause more trouble.
- For
arangodb_heartbeat_failures:- Typically this should be 0.
- If you see any other value here this is typically a network hiccup.
- If this is constantly growing this means the server is somehow undergoing a network split.
Troubleshoot
Heartbeats are precious and on the fastest possible path internally. If they slow down or cannot be delivered this in almost all cases can be appointed to network issues. If you constantly have this on a high value please make sure the latency between your cluster machines and all agents is low, this will be a lower bound for the value we achieve here. If this is not the case, the network might be overloaded.
Agency Plan Sync on DB-Servers
In order to update the data definition on databases servers from the definition stored in the agency, DB-Servers have a repeated job called Agency Plan Sync. Timings for collection and database creations are strongly correlated to the overall runtime of this job.
Metric
arangodb_maintenance_agency_sync_runtime_msec: Histogram containing the runtimes of individual runs.arangodb_maintenance_agency_sync_accum_runtime_msec: The accumulated runtime of all runs.
Exposed by DB-Server
Threshold
- For
arangodb_maintenance_agency_sync_runtime_msec:- This should not exceed 1000ms.
Troubleshoot
If the Agency Plan Sync becomes the bottleneck of database and collection distribution you should consider reducing the amount of those.
Shard Distribution
Allows insight in the shard distribution in the cluster and the state of replication of the data.
Metric
arangodb_shards_out_of_sync: Number of shards not replicated with their required replication factor (for which this server is the leader)arangodb_shards_total_count: Number of shards located on this server (leader and follower shards)arangodb_shards_leader_count: Number of shards for which this server is the leader.arangodb_shards_not_replicated: Number of shards that are not replicated, i.e. this data is at risk as there is no other copy available.
Exposed by DB-Server
Threshold
- For
arangodb_shards_out_of_sync:- Eventually all shards should be in sync and this value equal to zero.
- It can increase when new collections are created or servers are rotated.
- For
arangodb_shards_total_countandarangodb_shards_leader_count:- This value should be roughly equal for all servers.
- For
arangodb_shards_not_replicated:- This value should be zero at all times. If not, you currently have a single point of failure and data is at risk. Please contact our support team.
- This can happen if you lose 1 DB-Server and have
replicationFactor2, if you lose 2 DB-Servers onreplicationFactor3 and so on. In this cases the system will try to heal itself, if enough healthy servers remain.
Troubleshoot
The distribution of shards should be roughly equal. If not please consider rebalancing shards.
Scheduler
The Scheduler is responsible for managing growing workloads and distributing tasks across the available threads. Whenever more work is available than the system can handle, it adjusts the number of threads. The scheduler maintains an internal queue for tasks ready for execution. A constantly growing queue is a clear sign for the system reaching its limits.
Metric
arangodb_scheduler_queue_length: Length of the internal task queue.arangodb_scheduler_awake_threads: Number of actively working threads.arangodb_scheduler_num_worker_threads: Total number of currently available threads.
Exposed by Coordinator, DB-Server, Agents
Threshold
- For
arangodb_scheduler_queue_length:- Typically this should be 0.
- Having an non-zero queue length is not a problem as long as it eventually becomes smaller again. This can happen for example during load spikes.
- Having a longer queue results in bigger latencies as the requests need to wait longer before they are executed.
- If the queue runs full you will eventually get a
queue fullerror.
- For
arangodb_scheduler_num_worker_threadsandarangodb_scheduler_awake_threads:- They should increase as load increases.
- If the queue length is non-zero for more than a minute you should see
arangodb_scheduler_awake_threads == arangodb_scheduler_num_worker_threads. If not, consider contacting our support.
Troubleshoot
Queuing requests will result in bigger latency. If your queue is constantly growing, you should consider scaling your system according to your needs. Remember to rebalance shards if you scale up DB-Servers.
Metric
arangodb_scheduler_queue_full_failures: Number of times a request/task could not be added to the scheduler queue because the queue was full. If this happens, the corresponding request will be responded to with an HTTP 503 (“Service Unavailable”) response.
Exposed by Coordinator, DB-Server, Agents
Threshold
- For
arangodb_scheduler_queue_full_failures:- This should be 0, as dropping requests is an extremely undesirable event.
Troubleshoot
If the number of queue full failures is greater than zero and even growing over time, it indicates that the server (or one of the server in a cluster) is overloaded and cannot keep up with the workload. There are many potential reasons for this, e.g. servers with too little capacity, spiky workloads, or even network connectivity issues. Whenever this problem happens, it needs further detailed analysis of what could be the root cause.
Supervision
The supervision is an integral part of the cluster and runs on the leading agent. It is responsible for handling MoveShard jobs and server failures. It is intended to run every second, thus its runime should be below one second.
Metric
arangodb_agency_supervision_runtime_msec: Time in ms of a single supervision run.arangodb_agency_supervision_runtime_wait_for_replication_msec: Time the supervision has to wait for its decisions to be committed.
Exposed by Agents
Threshold
- For
agency_supervision_runtime_msec:-
This value should stay below 1000ms. However, when a DB-Server is rotated there can be single runs that have much higher runtime. However, this should not be true in general.
This value will only increase for the leading agent.
-
Troubleshoot
If the supervision is not able to run approximately once per second, cluster resilience is affected. Please consider contacting our support.
Metrics API
Read the metrics
return the current instance metrics
GET /_admin/metrics
Returns the instance’s current metrics in Prometheus format. The returned document collects all instance metrics, which are measured at any given time and exposes them for collection by Prometheus.
The document contains different metrics and metrics groups dependent
on the role of the queried instance. All exported metrics are
published with the arangodb_ or rocksdb_ string to distinguish
them from other collected data.
The API then needs to be added to the Prometheus configuration file for collection.
Return codes
-
200: Metrics were returned successfully.
-
404: The metrics API may be disabled using
--server.export-metrics-api falsesetting in the server. In this case, the result of the call indicates the API to be not found.
Examples
shell> curl --header 'accept: application/json' --dump - http://localhost:8529/_admin/metrics
HTTP/1.1 OK
content-type: text/plain
x-content-type-options: nosniff
#TYPE arangodb_heartbeat_failures counter
#HELP arangodb_heartbeat_failures Counting failed heartbeat transmissions
arangodb_heartbeat_failures{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_heartbeat_send_time_msec histogram
#HELP arangodb_heartbeat_send_time_msec Time required to send heartbeat [ms]
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="19"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="35"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="66"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="128"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="253"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="503"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="1003"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="2003"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="4002"} 0
arangodb_heartbeat_send_time_msec_bucket{role="SINGLE",shortname="Single0001",le="+Inf"} 0
arangodb_heartbeat_send_time_msec_count{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_load_current_runtime histogram
#HELP arangodb_load_current_runtime Current loading runtimes [ms]
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="0.308525"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="0.838657"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="2.279705"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="6.196882"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="16.844870"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="45.789104"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="124.467682"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="338.338226"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="919.698608"} 0
arangodb_load_current_runtime_bucket{role="SINGLE",shortname="Single0001",le="+Inf"} 0
arangodb_load_current_runtime_count{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_scheduler_queue_full_failures counter
#HELP arangodb_scheduler_queue_full_failures Tasks dropped and not added to internal queue
arangodb_scheduler_queue_full_failures{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_scheduler_num_worker_threads gauge
#HELP arangodb_scheduler_num_worker_threads Number of worker threads
arangodb_scheduler_num_worker_threads{role="SINGLE",shortname="Single0001"} 3
#TYPE arangodb_scheduler_awake_threads gauge
#HELP arangodb_scheduler_awake_threads Number of awake worker threads
arangodb_scheduler_awake_threads{role="SINGLE",shortname="Single0001"} 1
#TYPE arangodb_v8_context_entered counter
#HELP arangodb_v8_context_entered V8 context enter events
arangodb_v8_context_entered{} 34
#TYPE arangodb_v8_context_enter_failures counter
#HELP arangodb_v8_context_enter_failures V8 context enter failures
arangodb_v8_context_enter_failures{} 0
#TYPE arangodb_load_plan_runtime histogram
#HELP arangodb_load_plan_runtime Plan loading runtimes [ms]
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="0.308525"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="0.838657"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="2.279705"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="6.196882"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="16.844870"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="45.789104"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="124.467682"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="338.338226"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="919.698608"} 0
arangodb_load_plan_runtime_bucket{role="SINGLE",shortname="Single0001",le="+Inf"} 0
arangodb_load_plan_runtime_count{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_v8_context_exited counter
#HELP arangodb_v8_context_exited V8 context exit events
arangodb_v8_context_exited{} 34
#TYPE arangodb_transactions_started counter
#HELP arangodb_transactions_started Transactions started
arangodb_transactions_started{} 35069
#TYPE arangodb_aql_total_query_time_msec counter
#HELP arangodb_aql_total_query_time_msec Total execution time of all queries
arangodb_aql_total_query_time_msec{role="SINGLE",shortname="Single0001"} 1849
#TYPE arangodb_agencycomm_request_time_msec histogram
#HELP arangodb_agencycomm_request_time_msec Request time for Agency requests
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="292"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="526"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="995"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="1932"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="3806"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="7554"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="15050"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="30043"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="60029"} 0
arangodb_agencycomm_request_time_msec_bucket{role="SINGLE",shortname="Single0001",le="+Inf"} 0
arangodb_agencycomm_request_time_msec_count{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_v8_context_destroyed counter
#HELP arangodb_v8_context_destroyed V8 contexts destroyed
arangodb_v8_context_destroyed{} 0
#TYPE arangodb_load_current_accum_runtime_msec counter
#HELP arangodb_load_current_accum_runtime_msec Accumulated runtime of Current loading [ms]
arangodb_load_current_accum_runtime_msec{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_aql_slow_query counter
#HELP arangodb_aql_slow_query Number of slow AQL queries
arangodb_aql_slow_query{} 0
#TYPE arangodb_transactions_committed counter
#HELP arangodb_transactions_committed Transactions committed
arangodb_transactions_committed{} 34778
#TYPE arangodb_v8_context_created counter
#HELP arangodb_v8_context_created V8 contexts created
arangodb_v8_context_created{} 2
#TYPE arangodb_load_plan_accum_runtime_msec counter
#HELP arangodb_load_plan_accum_runtime_msec Accumulated runtime of Plan loading [ms]
arangodb_load_plan_accum_runtime_msec{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_transactions_aborted counter
#HELP arangodb_transactions_aborted Transactions aborted
arangodb_transactions_aborted{} 291
#TYPE arangodb_intermediate_commits counter
#HELP arangodb_intermediate_commits Intermediate commits
arangodb_intermediate_commits{} 0
#TYPE arangodb_scheduler_queue_length gauge
#HELP arangodb_scheduler_queue_length Servers internal queue length
arangodb_scheduler_queue_length{role="SINGLE",shortname="Single0001"} 0
#TYPE arangodb_process_statistics_minor_page_faults gauge
#HELP arangodb_process_statistics_minor_page_faults The number of minor faults the process has made which have not required loading a memory page from disk. This figure is not reported on Windows
arangodb_process_statistics_minor_page_faults 170128
#TYPE arangodb_process_statistics_major_page_faults gauge
#HELP arangodb_process_statistics_major_page_faults On Windows, this figure contains the total number of page faults. On other system, this figure contains the number of major faults the process has made which have required loading a memory page from disk
arangodb_process_statistics_major_page_faults 0
#TYPE arangodb_process_statistics_user_time gauge
#HELP arangodb_process_statistics_user_time On Windows, this figure contains the total amount of memory that the memory manager has committed for the arangod process. On other systems, this figure contains The size of the virtual memory the process is using
arangodb_process_statistics_user_time 10.360000
#TYPE arangodb_process_statistics_system_time gauge
#HELP arangodb_process_statistics_system_time Amount of time that this process has been scheduled in kernel mode, measured in seconds
arangodb_process_statistics_system_time 3.510000
#TYPE arangodb_process_statistics_number_of_threads gauge
#HELP arangodb_process_statistics_number_of_threads Number of threads in the arangod process
arangodb_process_statistics_number_of_threads 59
#TYPE arangodb_process_statistics_resident_set_size gauge
#HELP arangodb_process_statistics_resident_set_size The total size of the number of pages the process has in real memory. This is just the pages which count toward text, data, or stack space. This does not include pages which have not been demand-loaded in, or which are swapped out. The resident set size is reported in bytes
arangodb_process_statistics_resident_set_size 356835328
#TYPE arangodb_process_statistics_resident_set_size_percent gauge
#HELP arangodb_process_statistics_resident_set_size_percent The relative size of the number of pages the process has in real memory compared to system memory. This is just the pages which count toward text, data, or stack space. This does not include pages which have not been demand-loaded in, or which are swapped out. The value is a ratio between 0.00 and 1.00
arangodb_process_statistics_resident_set_size_percent 0.005285
#TYPE arangodb_process_statistics_virtual_memory_size gauge
#HELP arangodb_process_statistics_virtual_memory_size On Windows, this figure contains the total amount of memory that the memory manager has committed for the arangod process. On other systems, this figure contains The size of the virtual memory the process is using
arangodb_process_statistics_virtual_memory_size 1494765568
#TYPE arangodb_server_statistics_physical_memory gauge
#HELP arangodb_server_statistics_physical_memory Physical memory in bytes
arangodb_server_statistics_physical_memory 67513573376
#TYPE arangodb_server_statistics_server_uptime gauge
#HELP arangodb_server_statistics_server_uptime Number of seconds elapsed since server start
arangodb_server_statistics_server_uptime 20.585458
#TYPE arangodb_client_connection_statistics_client_connections gauge
#HELP arangodb_client_connection_statistics_client_connections The number of client connections that are currently open
arangodb_client_connection_statistics_client_connections 2
#TYPE arangodb_client_connection_statistics_connection_time_bucket gauge
#HELP arangodb_client_connection_statistics_connection_time_bucket Total connection time of a client
arangodb_client_connection_statistics_connection_time_bucket{le="0.01"} 0
arangodb_client_connection_statistics_connection_time_bucket{le="1.0"} 0
arangodb_client_connection_statistics_connection_time_bucket{le="60.0"} 0
arangodb_client_connection_statistics_connection_time_bucket{le="+Inf"} 0
#TYPE arangodb_client_connection_statistics_total_time_bucket gauge
#HELP arangodb_client_connection_statistics_total_time_bucket Total time needed to answer a request
arangodb_client_connection_statistics_total_time_bucket{le="0.01"} 34724
arangodb_client_connection_statistics_total_time_bucket{le="0.05"} 21
arangodb_client_connection_statistics_total_time_bucket{le="0.1"} 2
arangodb_client_connection_statistics_total_time_bucket{le="0.2"} 1
arangodb_client_connection_statistics_total_time_bucket{le="0.5"} 0
arangodb_client_connection_statistics_total_time_bucket{le="1.0"} 1
arangodb_client_connection_statistics_total_time_bucket{le="+Inf"} 0
#TYPE arangodb_client_connection_statistics_request_time_bucket gauge
#HELP arangodb_client_connection_statistics_request_time_bucket Request time needed to answer a request
arangodb_client_connection_statistics_request_time_bucket{le="0.01"} 34724
arangodb_client_connection_statistics_request_time_bucket{le="0.05"} 21
arangodb_client_connection_statistics_request_time_bucket{le="0.1"} 2
arangodb_client_connection_statistics_request_time_bucket{le="0.2"} 1
arangodb_client_connection_statistics_request_time_bucket{le="0.5"} 0
arangodb_client_connection_statistics_request_time_bucket{le="1.0"} 1
arangodb_client_connection_statistics_request_time_bucket{le="+Inf"} 0
#TYPE arangodb_client_connection_statistics_queue_time_bucket gauge
#HELP arangodb_client_connection_statistics_queue_time_bucket Request time needed to answer a request
arangodb_client_connection_statistics_queue_time_bucket{le="0.01"} 34748
arangodb_client_connection_statistics_queue_time_bucket{le="0.05"} 0
arangodb_client_connection_statistics_queue_time_bucket{le="0.1"} 0
arangodb_client_connection_statistics_queue_time_bucket{le="0.2"} 0
arangodb_client_connection_statistics_queue_time_bucket{le="0.5"} 0
arangodb_client_connection_statistics_queue_time_bucket{le="1.0"} 0
arangodb_client_connection_statistics_queue_time_bucket{le="+Inf"} 0
#TYPE arangodb_client_connection_statistics_io_time_bucket gauge
#HELP arangodb_client_connection_statistics_io_time_bucket Request time needed to answer a request
arangodb_client_connection_statistics_io_time_bucket{le="0.01"} 34749
arangodb_client_connection_statistics_io_time_bucket{le="0.05"} 0
arangodb_client_connection_statistics_io_time_bucket{le="0.1"} 0
arangodb_client_connection_statistics_io_time_bucket{le="0.2"} 0
arangodb_client_connection_statistics_io_time_bucket{le="0.5"} 0
arangodb_client_connection_statistics_io_time_bucket{le="1.0"} 0
arangodb_client_connection_statistics_io_time_bucket{le="+Inf"} 0
#TYPE arangodb_client_connection_statistics_bytes_sent_bucket gauge
#HELP arangodb_client_connection_statistics_bytes_sent_bucket Bytes sent for a request
arangodb_client_connection_statistics_bytes_sent_bucket{le="250"} 396
arangodb_client_connection_statistics_bytes_sent_bucket{le="1000"} 33951
arangodb_client_connection_statistics_bytes_sent_bucket{le="2000"} 362
arangodb_client_connection_statistics_bytes_sent_bucket{le="5000"} 32
arangodb_client_connection_statistics_bytes_sent_bucket{le="10000"} 4
arangodb_client_connection_statistics_bytes_sent_bucket{le="+Inf"} 4
#TYPE arangodb_client_connection_statistics_bytes_received_bucket gauge
#HELP arangodb_client_connection_statistics_bytes_received_bucket Bytes received for a request
arangodb_client_connection_statistics_bytes_received_bucket{le="250"} 33763
arangodb_client_connection_statistics_bytes_received_bucket{le="1000"} 986
arangodb_client_connection_statistics_bytes_received_bucket{le="2000"} 0
arangodb_client_connection_statistics_bytes_received_bucket{le="5000"} 0
arangodb_client_connection_statistics_bytes_received_bucket{le="10000"} 0
arangodb_client_connection_statistics_bytes_received_bucket{le="+Inf"} 0
#TYPE arangodb_http_request_statistics_async_requests gauge
#HELP arangodb_http_request_statistics_async_requests Number of asynchronously executed HTTP requests
arangodb_http_request_statistics_async_requests 2
#TYPE arangodb_http_request_statistics_http_delete_requests gauge
#HELP arangodb_http_request_statistics_http_delete_requests Number of HTTP DELETE requests
arangodb_http_request_statistics_http_delete_requests 105
#TYPE arangodb_http_request_statistics_http_get_requests gauge
#HELP arangodb_http_request_statistics_http_get_requests Number of HTTP GET requests
arangodb_http_request_statistics_http_get_requests 882
#TYPE arangodb_http_request_statistics_http_head_requests gauge
#HELP arangodb_http_request_statistics_http_head_requests Number of HTTP HEAD requests
arangodb_http_request_statistics_http_head_requests 0
#TYPE arangodb_http_request_statistics_http_options_requests gauge
#HELP arangodb_http_request_statistics_http_options_requests Number of HTTP OPTIONS requests
arangodb_http_request_statistics_http_options_requests 0
#TYPE arangodb_http_request_statistics_http_patch_requests gauge
#HELP arangodb_http_request_statistics_http_patch_requests Number of HTTP PATH requests
arangodb_http_request_statistics_http_patch_requests 2
#TYPE arangodb_http_request_statistics_http_post_requests gauge
#HELP arangodb_http_request_statistics_http_post_requests Number of HTTP POST requests
arangodb_http_request_statistics_http_post_requests 33727
#TYPE arangodb_http_request_statistics_http_put_requests gauge
#HELP arangodb_http_request_statistics_http_put_requests Number of HTTP PUT requests
arangodb_http_request_statistics_http_put_requests 33
#TYPE arangodb_http_request_statistics_other_http_requests gauge
#HELP arangodb_http_request_statistics_other_http_requests Number of other HTTP requests
arangodb_http_request_statistics_other_http_requests 0
#TYPE arangodb_http_request_statistics_total_requests gauge
#HELP arangodb_http_request_statistics_total_requests Total number of HTTP requests
arangodb_http_request_statistics_total_requests 34749
#TYPE rocksdb_cache_allocated counter
#HELP rocksdb_cache_allocated rocksdb_cache_allocated
rocksdb_cache_allocated 1357112
#TYPE rocksdb_cache_hit_rate_lifetime counter
#HELP rocksdb_cache_hit_rate_lifetime rocksdb_cache_hit_rate_lifetime
rocksdb_cache_hit_rate_lifetime 2
#TYPE rocksdb_cache_hit_rate_recent counter
#HELP rocksdb_cache_hit_rate_recent rocksdb_cache_hit_rate_recent
rocksdb_cache_hit_rate_recent 2
#TYPE rocksdb_cache_limit counter
#HELP rocksdb_cache_limit rocksdb_cache_limit
rocksdb_cache_limit 16341522432
#TYPE rocksdb_actual_delayed_write_rate counter
#HELP rocksdb_actual_delayed_write_rate rocksdb_actual_delayed_write_rate
rocksdb_actual_delayed_write_rate 0
#TYPE rocksdb_background_errors counter
#HELP rocksdb_background_errors rocksdb_background_errors
rocksdb_background_errors 0
#TYPE rocksdb_base_level counter
#HELP rocksdb_base_level rocksdb_base_level
rocksdb_base_level 6
#TYPE rocksdb_block_cache_capacity counter
#HELP rocksdb_block_cache_capacity rocksdb_block_cache_capacity
rocksdb_block_cache_capacity 19609826918
#TYPE rocksdb_block_cache_pinned_usage counter
#HELP rocksdb_block_cache_pinned_usage rocksdb_block_cache_pinned_usage
rocksdb_block_cache_pinned_usage 0
#TYPE rocksdb_block_cache_usage counter
#HELP rocksdb_block_cache_usage rocksdb_block_cache_usage
rocksdb_block_cache_usage 0
#TYPE rocksdb_compaction_pending counter
#HELP rocksdb_compaction_pending rocksdb_compaction_pending
rocksdb_compaction_pending 0
#TYPE rocksdb_compression_ratio_at_level0 counter
#HELP rocksdb_compression_ratio_at_level0 rocksdb_compression_ratio_at_level0
rocksdb_compression_ratio_at_level0 0
#TYPE rocksdb_compression_ratio_at_level1 counter
#HELP rocksdb_compression_ratio_at_level1 rocksdb_compression_ratio_at_level1
rocksdb_compression_ratio_at_level1 0
#TYPE rocksdb_compression_ratio_at_level2 counter
#HELP rocksdb_compression_ratio_at_level2 rocksdb_compression_ratio_at_level2
rocksdb_compression_ratio_at_level2 0
#TYPE rocksdb_compression_ratio_at_level3 counter
#HELP rocksdb_compression_ratio_at_level3 rocksdb_compression_ratio_at_level3
rocksdb_compression_ratio_at_level3 0
#TYPE rocksdb_compression_ratio_at_level4 counter
#HELP rocksdb_compression_ratio_at_level4 rocksdb_compression_ratio_at_level4
rocksdb_compression_ratio_at_level4 0
#TYPE rocksdb_compression_ratio_at_level5 counter
#HELP rocksdb_compression_ratio_at_level5 rocksdb_compression_ratio_at_level5
rocksdb_compression_ratio_at_level5 0
#TYPE rocksdb_compression_ratio_at_level6 counter
#HELP rocksdb_compression_ratio_at_level6 rocksdb_compression_ratio_at_level6
rocksdb_compression_ratio_at_level6 0
#TYPE rocksdb_cur_size_active_mem_table counter
#HELP rocksdb_cur_size_active_mem_table rocksdb_cur_size_active_mem_table
rocksdb_cur_size_active_mem_table 105151688
#TYPE rocksdb_cur_size_all_mem_tables counter
#HELP rocksdb_cur_size_all_mem_tables rocksdb_cur_size_all_mem_tables
rocksdb_cur_size_all_mem_tables 105151688
#TYPE rocksdb_estimate_live_data_size counter
#HELP rocksdb_estimate_live_data_size rocksdb_estimate_live_data_size
rocksdb_estimate_live_data_size 0
#TYPE rocksdb_estimate_num_keys counter
#HELP rocksdb_estimate_num_keys rocksdb_estimate_num_keys
rocksdb_estimate_num_keys 1636
#TYPE rocksdb_estimate_pending_compaction_bytes counter
#HELP rocksdb_estimate_pending_compaction_bytes rocksdb_estimate_pending_compaction_bytes
rocksdb_estimate_pending_compaction_bytes 0
#TYPE rocksdb_estimate_table_readers_mem counter
#HELP rocksdb_estimate_table_readers_mem rocksdb_estimate_table_readers_mem
rocksdb_estimate_table_readers_mem 0
#TYPE rocksdb_is_file_deletions_enabled counter
#HELP rocksdb_is_file_deletions_enabled rocksdb_is_file_deletions_enabled
rocksdb_is_file_deletions_enabled 1
#TYPE rocksdb_is_write_stopped counter
#HELP rocksdb_is_write_stopped rocksdb_is_write_stopped
rocksdb_is_write_stopped 0
#TYPE rocksdb_live_sst_files_size counter
#HELP rocksdb_live_sst_files_size rocksdb_live_sst_files_size
rocksdb_live_sst_files_size 0
#TYPE rocksdb_mem_table_flush_pending counter
#HELP rocksdb_mem_table_flush_pending rocksdb_mem_table_flush_pending
rocksdb_mem_table_flush_pending 0
#TYPE rocksdb_min_log_number_to_keep counter
#HELP rocksdb_min_log_number_to_keep rocksdb_min_log_number_to_keep
rocksdb_min_log_number_to_keep 0
#TYPE rocksdb_num_deletes_active_mem_table counter
#HELP rocksdb_num_deletes_active_mem_table rocksdb_num_deletes_active_mem_table
rocksdb_num_deletes_active_mem_table 92379
#TYPE rocksdb_num_deletes_imm_mem_tables counter
#HELP rocksdb_num_deletes_imm_mem_tables rocksdb_num_deletes_imm_mem_tables
rocksdb_num_deletes_imm_mem_tables 0
#TYPE rocksdb_num_entries_active_mem_table counter
#HELP rocksdb_num_entries_active_mem_table rocksdb_num_entries_active_mem_table
rocksdb_num_entries_active_mem_table 185715
#TYPE rocksdb_num_entries_imm_mem_tables counter
#HELP rocksdb_num_entries_imm_mem_tables rocksdb_num_entries_imm_mem_tables
rocksdb_num_entries_imm_mem_tables 0
#TYPE rocksdb_num_files_at_level0 counter
#HELP rocksdb_num_files_at_level0 rocksdb_num_files_at_level0
rocksdb_num_files_at_level0 0
#TYPE rocksdb_num_files_at_level1 counter
#HELP rocksdb_num_files_at_level1 rocksdb_num_files_at_level1
rocksdb_num_files_at_level1 0
#TYPE rocksdb_num_files_at_level2 counter
#HELP rocksdb_num_files_at_level2 rocksdb_num_files_at_level2
rocksdb_num_files_at_level2 0
#TYPE rocksdb_num_files_at_level3 counter
#HELP rocksdb_num_files_at_level3 rocksdb_num_files_at_level3
rocksdb_num_files_at_level3 0
#TYPE rocksdb_num_files_at_level4 counter
#HELP rocksdb_num_files_at_level4 rocksdb_num_files_at_level4
rocksdb_num_files_at_level4 0
#TYPE rocksdb_num_files_at_level5 counter
#HELP rocksdb_num_files_at_level5 rocksdb_num_files_at_level5
rocksdb_num_files_at_level5 0
#TYPE rocksdb_num_files_at_level6 counter
#HELP rocksdb_num_files_at_level6 rocksdb_num_files_at_level6
rocksdb_num_files_at_level6 0
#TYPE rocksdb_num_immutable_mem_table counter
#HELP rocksdb_num_immutable_mem_table rocksdb_num_immutable_mem_table
rocksdb_num_immutable_mem_table 0
#TYPE rocksdb_num_immutable_mem_table_flushed counter
#HELP rocksdb_num_immutable_mem_table_flushed rocksdb_num_immutable_mem_table_flushed
rocksdb_num_immutable_mem_table_flushed 0
#TYPE rocksdb_num_live_versions counter
#HELP rocksdb_num_live_versions rocksdb_num_live_versions
rocksdb_num_live_versions 7
#TYPE rocksdb_num_running_compactions counter
#HELP rocksdb_num_running_compactions rocksdb_num_running_compactions
rocksdb_num_running_compactions 0
#TYPE rocksdb_num_running_flushes counter
#HELP rocksdb_num_running_flushes rocksdb_num_running_flushes
rocksdb_num_running_flushes 0
#TYPE rocksdb_num_snapshots counter
#HELP rocksdb_num_snapshots rocksdb_num_snapshots
rocksdb_num_snapshots 0
#TYPE rocksdb_oldest_snapshot_time counter
#HELP rocksdb_oldest_snapshot_time rocksdb_oldest_snapshot_time
rocksdb_oldest_snapshot_time 0
#TYPE rocksdb_size_all_mem_tables counter
#HELP rocksdb_size_all_mem_tables rocksdb_size_all_mem_tables
rocksdb_size_all_mem_tables 105151688
#TYPE rocksdb_total_sst_files_size counter
#HELP rocksdb_total_sst_files_size rocksdb_total_sst_files_size
rocksdb_total_sst_files_size 0
#TYPE rocksdbengine_throttle_bps counter
#HELP rocksdbengine_throttle_bps rocksdbengine_throttle_bps
rocksdbengine_throttle_bps 0
List of exposed metrics
| Label | Description |
|---|---|
arangodb_agency_append_hist |
Agency RAFT follower append histogram (ms) |
arangodb_agency_commit_hist |
Agency RAFT commit histogram (ms) |
arangodb_agency_compaction_hist |
Agency compaction histogram (ms) |
arangodb_agency_local_commit_index |
This agent’s commit index |
arangodb_agency_log_size_bytes |
Agency replicated log size (bytes) |
arangodb_agency_read_no_leader |
Agency read no leader |
arangodb_agency_read_ok |
Agency read ok |
arangodb_agency_supervision_accum_runtime_msec |
Accumulated Supervision Runtime (ms) |
arangodb_agency_supervision_accum_runtime_wait_for_replication_msec |
Accumulated Supervision wait for replication time (ms) |
arangodb_agency_supervision_failed_server_count |
Counter for FailedServer jobs |
arangodb_agency_supervision_runtime_msec |
Agency Supervision runtime histogram (ms) |
arangodb_agency_supervision_runtime_wait_for_replication_msec |
Agency Supervision wait for replication time (ms) |
arangodb_agency_term |
Agency’s term |
arangodb_agency_write_hist |
Agency write histogram (ms) |
arangodb_agency_write_no_leader |
Agency write no leader |
arangodb_agency_write_ok |
Agency write ok |
arangodb_agencycomm_request_time_msec |
Request time for Agency requests |
arangodb_aql_slow_query |
Number of AQL slow queries |
arangodb_aql_total_query_time_msec |
Total execution time of all AQL queries |
arangodb_client_connection_statistics_bytes_received_bucket |
Bytes received for a request |
arangodb_client_connection_statistics_bytes_received_count |
Bytes received for a request |
arangodb_client_connection_statistics_bytes_received_sum |
Bytes received for a request |
arangodb_client_connection_statistics_bytes_sent_bucket |
Bytes sent for a request |
arangodb_client_connection_statistics_bytes_sent_count |
Bytes sent for a request |
arangodb_client_connection_statistics_bytes_sent_sum |
Bytes sent for a request |
arangodb_client_connection_statistics_client_connections |
The number of client connections that are currently open |
arangodb_client_connection_statistics_connection_time_bucket |
Total connection time of a client |
arangodb_client_connection_statistics_connection_time_count |
Total connection time of a client |
arangodb_client_connection_statistics_connection_time_sum |
Total connection time of a client |
arangodb_client_connection_statistics_io_time_bucket |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_io_time_count |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_io_time_sum |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_queue_time_bucket |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_queue_time_count |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_queue_time_sum |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_request_time_bucket |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_request_time_count |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_request_time_sum |
Request time needed to answer a request (ms) |
arangodb_client_connection_statistics_total_time_bucket |
Total time needed to answer a request (ms) |
arangodb_client_connection_statistics_total_time_count |
Total time needed to answer a request (ms) |
arangodb_client_connection_statistics_total_time_sum |
Total time needed to answer a request (ms) |
arangodb_dropped_followers_count |
Number of drop-follower events |
arangodb_heartbeat_failures |
Counting failed heartbeat transmissions |
arangodb_heartbeat_send_time_msec |
Time required to send heartbeat (ms) |
arangodb_http_request_statistics_async_requests |
Number of asynchronously executed HTTP requests |
arangodb_http_request_statistics_http_delete_requests |
Number of HTTP DELETE requests |
arangodb_http_request_statistics_http_get_requests |
Number of HTTP GET requests |
arangodb_http_request_statistics_http_head_requests |
Number of HTTP HEAD requests |
arangodb_http_request_statistics_http_options_requests |
Number of HTTP OPTIONS requests |
arangodb_http_request_statistics_http_patch_requests |
Number of HTTP PATH requests |
arangodb_http_request_statistics_http_post_requests |
Number of HTTP POST requests |
arangodb_http_request_statistics_http_put_requests |
Number of HTTP PUT requests |
arangodb_http_request_statistics_other_http_requests |
Number of other HTTP requests |
arangodb_http_request_statistics_total_requests |
Total number of HTTP requests |
arangodb_intermediate_commits |
Intermediate commits |
arangodb_load_current_accum_runtime_msec |
Accumulated Current loading time (ms) |
arangodb_load_current_runtime |
Current loading runtimes (ms) |
arangodb_load_plan_accum_runtime_msec |
Accumulated runtime of Plan loading (ms) |
arangodb_load_plan_runtime |
Plan loading runtimes (ms) |
arangodb_maintenance_action_accum_queue_time_msec |
Accumulated action queue time (ms) |
arangodb_maintenance_action_accum_runtime_msec |
Accumulated action runtime (ms) |
arangodb_maintenance_action_done_counter |
Counter of action that are done and have been removed from the registry |
arangodb_maintenance_action_duplicate_counter |
Counter of action that have been discarded because of a duplicate |
arangodb_maintenance_action_failure_counter |
Failure counter for the action |
arangodb_maintenance_action_queue_time_msec |
Time spend in the queue before execution (ms) |
arangodb_maintenance_action_registered_counter |
Counter of action that have been registered in the action registry |
arangodb_maintenance_action_runtime_msec |
Time spend execution the action (ms) |
arangodb_maintenance_agency_sync_accum_runtime_msec |
Accumulated runtime of agency sync phase (ms) |
arangodb_maintenance_agency_sync_runtime_msec |
Total time spend on agency sync (ms) |
arangodb_maintenance_phase1_accum_runtime_msec |
Accumulated runtime of phase one (ms) |
arangodb_maintenance_phase1_runtime_msec |
Maintenance Phase 1 runtime histogram (ms) |
arangodb_maintenance_phase2_accum_runtime_msec |
Accumulated runtime of phase two (ms) |
arangodb_maintenance_phase2_runtime_msec |
Maintenance Phase 2 runtime histogram (ms) |
arangodb_process_statistics_major_page_faults |
On Windows, this figure contains the total number of page faults. On other system, this figure contains the number of major faults the process has made which have required loading a memory page from disk |
arangodb_process_statistics_minor_page_faults |
The number of minor faults the process has made which have not required loading a memory page from disk. This figure is not reported on Windows |
arangodb_process_statistics_number_of_threads |
Number of threads in the arangod process |
arangodb_process_statistics_resident_set_size_percent |
The relative size of the number of pages the process has in real memory compared to system memory. This is just the pages which count toward text, data, or stack space. This does not include pages which have not been demand-loaded in, or which are swapped out. The value is a ratio between 0.00 and 1.00 |
arangodb_process_statistics_resident_set_size |
The total size of the number of pages the process has in real memory. This is just the pages which count toward text, data, or stack space. This does not include pages which have not been demand-loaded in, or which are swapped out. The resident set size is reported in bytes |
arangodb_process_statistics_system_time |
Amount of time that this process has been scheduled in kernel mode, measured in seconds |
arangodb_process_statistics_user_time |
On Windows, this figure contains the total amount of memory that the memory manager has committed for the arangod process. On other systems, this figure contains The size of the virtual memory the process is using |
arangodb_process_statistics_virtual_memory_size |
On Windows, this figure contains the total amount of memory that the memory manager has committed for the arangod process. On other systems, this figure contains The size of the virtual memory the process is using |
arangodb_scheduler_awake_threads |
Number of awake worker threads |
arangodb_scheduler_num_worker_threads |
Number of worker threads |
arangodb_scheduler_queue_full_failures |
Number of times the scheduler queue was full and a task/request was rejected |
arangodb_scheduler_queue_length |
Server’s internal queue length |
arangodb_server_statistics_physical_memory |
Physical memory in bytes |
arangodb_server_statistics_server_uptime |
Number of seconds elapsed since server start |
arangodb_shards_leader_count |
Number of leader shards on this machine |
arangodb_shards_not_replicated |
Number of shards not replicated at all |
arangodb_shards_out_of_sync |
Number of leader shards not fully replicated |
arangodb_shards_total_count |
Number of shards on this machine |
arangodb_transactions_aborted |
Transactions aborted |
arangodb_transactions_committed |
Transactions committed |
arangodb_transactions_started |
Transactions started |
arangodb_v8_context_created |
Number of V8 contexts created |
arangodb_v8_context_destroyed |
Number of V8 contexts destroyed |
arangodb_v8_context_enter_failures |
Number of times a V8 context could not be entered/acquired |
arangodb_v8_context_entered |
Number of times a V8 context was successfully entered |
arangodb_v8_context_exited |
Number of times a V8 context was successfully exited |