API Changes in ArangoDB 3.12

HTTP RESTful API

Behavior changes

VelocyStream protocol removed

ArangoDB’s own bi-directional asynchronous binary protocol VelocyStream is no longer supported.

The server immediately closes the connection if you attempt to use the VelocyStream protocol. If you specify any scheme starting with vst in the --server.endpoint startup option of a client tool, the HTTP protocol is used instead.

The following metrics related to VelocyStream have been removed:

• arangodb_request_body_size_vst
• arangodb_vst_connections_total

VelocyPack remains as ArangoDB’s binary storage format and you can continue to use it in transport over the HTTP protocol, as well as use JSON over the HTTP protocol.

HTTP headers

The following long-deprecated features have been removed from ArangoDB’s HTTP server:

• Overriding the HTTP method by setting one of the HTTP headers:

  • x-http-method
  • x-http-method-override
  • x-method-override

  This functionality posed a potential security risk and was thus removed. Previously, it was only enabled when explicitly starting the server with the --http.allow-method-override startup option. The functionality has now been removed and setting the startup option does nothing.

• Optionally hiding ArangoDB’s server response header. This functionality could optionally be enabled by starting the server with the startup option --http.hide-product-header. The functionality has now been removed and setting the startup option does nothing.

--database.extended-names enabled by default

The --database.extended-names startup option is now enabled by default. The names of databases, collections, Views, and indexes may contain Unicode characters using the default settings.

Collection API

When creating a collection using the POST /_api/collection endpoint, the server log now displays a deprecation message if illegal combinations and unknown attributes and values are detected in the request body.

Note that all invalid elements and combinations will be rejected in future versions.

Index API

Stored values can contain the _id attribute

The usage of the _id system attribute was previously disallowed for persistent indexes inside of storedValues. This is now allowed in v3.12.

Note that it is still forbidden to use _id as a top-level attribute or sub-attribute in fields of persistent indexes. On the other hand, inverted indexes have been allowing to index and store the _id system attribute.

Optimizer rule changes

Due to the improved joins in AQL, there is a new join-index-nodes optimizer rule and a JoinNode that may appear in execution plans.

The remove-unnecessary-projections AQL optimizer rule has been renamed to optimize-projections and now includes an additional optimization.

Moreover, a remove-unnecessary-calculations-4 and batch-materialize-documents rule have been added.

The affected endpoints are POST /_api/cursor, POST /_api/explain, and GET /_api/query/rules.

Gharial API

The PATCH /_api/gharial/{graph}/edge/{collection}/{edge} endpoint to update edges in named graphs now validates the referenced vertex when modifying either the _from or _to edge attribute. Previously, the validation only occurred if both were set in the request.

Validation of smartGraphAttribute in SmartGraphs

Introduced in: v3.10.13, v3.11.7

The attribute defined by the smartGraphAttribute graph property is not allowed to be changed in the documents of SmartGraph vertex collections. This is now strictly enforced. You must set the attribute when creating a document. Any attempt to modify or remove the attribute afterward by update or replace operations now throws an error. Previously, the smartGraphAttribute value was checked only when inserting documents into a SmartGraph vertex collection, but not for update or replace operations.

The missing checks on update and replace operations allowed to retroactively modify the value of the smartGraphAttribute for existing documents, which could have led to problems when the data of such a SmartGraph vertex collection was replicated to a new follower shard. On the new follower shard, the documents went through the full validation and led to documents with modified smartGraphAttribute values being rejected on the follower. This could have led to follower shards not getting in sync.

Now, the value of the smartGraphAttribute is fully validated with every insert, update, or replace operation, and every attempt to modify the value of the smartGraphAttribute retroactively fails with the 4003 error, ERROR_KEY_MUST_BE_PREFIXED_WITH_SMART_GRAPH_ATTRIBUTE. Additionally, if upon insertion the smartGraphAttribute is missing for a SmartGraph vertex, the error code is error 4001, ERROR_NO_SMART_GRAPH_ATTRIBUTE.

To retroactively repair the data in any of the affected collections, it is possible to update every (affected) document with the correct value of the smartGraphAttribute via an AQL query as follows:

FOR doc IN @@collection
  LET expected = SUBSTRING(doc._key, 0, FIND_FIRST(doc._key, ':'))
  LET actual = doc.@attr
  FILTER expected != actual
  UPDATE doc WITH {@attr: expected} IN @@collection
  COLLECT WITH COUNT INTO updated
  RETURN updated

This updates all documents with the correct (expected) value of the smartGraphAttribute if it deviates from the expected value. The query returns the number of updated documents as well.

The bind parameters necessary to run this query are:

• @@collection: name of a SmartGraph vertex collection to be updated
• @attr: attribute name of the smartGraphAttribute of the collection

Limit to the number of databases in a deployment

Introduced in: v3.10.10, v3.11.2

The new --database.max-databases startup option can cap the number of databases and creating databases using the POST /_api/database endpoint can thus now fail for this reason if your deployment is at or above the configured maximum. Example:

{
  "code": 400,
  "error": true,
  "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)",
  "errorNum": 32
}

Adjustable Stream Transaction size

The Stream Transactions HTTP API may now allow larger transactions or be limited to smaller transactions because the maximum transaction size can now be configured with the --transaction.streaming-max-transaction-size startup option. The default value remains 128 MiB.

Analyzer API

The /_api/analyzer endpoints supports a new multi_delimiter Analyzer that accepts an array of strings in a delimiters attribute of the properties object.

Adjustable writeConcern for collections with distributeShardsLike

Collections that are sharded like another collection via the distributeShardsLike property use the replicationFactor, numberOfShards, and shardingStrategy properties of the prototype collection. In previous versions, the writeConcern property of the prototype collection was used as well. Now, you can independently set a writeConcern when creating a collection with distributeShardsLike. The property defaults to the writeConcern of the prototype collection if you don’t specify it explicitly. You can adjust the writeConcern later on in either case.

Log API

The /_admin/log/* endpoints no longer use the ldap log topic. Changing the log level of the ldap topic or any other unknown topic is not an error, however. Also see Incompatible changes in ArangoDB 3.12.

Error code 12 removed

The unused error ERROR_OUT_OF_MEMORY_MMAP with the number 12 has been removed.

Endpoint return value changes

Storage engine API

• The storage engine API at GET /_api/engine does not return the attribute dfdb anymore.

• On single servers and DB-Servers, the GET /_api/engine endpoint now returns an endianness attribute. Currently, only Little Endian is supported as an architecture by ArangoDB. The value is therefore "little".

Endpoints added

Effective and available startup options

The new GET /_admin/options and GET /_admin/options-description HTTP API endpoints allow you to return the effective configuration and the available startup options of the queried arangod instance.

Previously, it was only possible to fetch the current configuration on single servers and Coordinators using a JavaScript transaction, and to list the available startup options with --dump-options.

See the HTTP interface for administration for details.

Available key generators

You can now retrieve the available key generators for collections using the new GET /_api/key-generators endpoint.

See the HTTP API description

Shard usage metrics

With GET /_admin/usage-metrics you can retrieve detailed shard usage metrics on DB-Servers.

These metrics can be enabled by setting the --server.export-shard-usage-metrics startup option to enabled-per-shard to make DB-Servers collect per-shard usage metrics, or to enabled-per-shard-per-user to make DB-Servers collect usage metrics per shard and per user whenever a shard is accessed.

For more information, see the HTTP API description and Monitoring per collection/database/user.

Endpoints augmented

View API

Views of type arangosearch accept a new optimizeTopK View property for the ArangoSearch WAND optimization. It is an immutable array of strings, optional, and defaults to [].

See the optimizeTopK View property for details.

Document API

The following endpoints accept a new versionAttribute query parameter that adds external versioning support:

• PATCH /_api/document/{collection}/{key} (single document update)
• PATCH /_api/document/{collection} (multiple document update)
• PUT /_api/document/{collection} (single document replace)
• PUT /_api/document/{collection}/{key} (multiple document replace)
• POST /_api/document/{collection} (single document insert, when used to update/replace a document)
• POST /_api/document/{collection}/{key} (multiple document insert, when used to update/replace a document)

If set, the attribute with the name specified by the option is looked up in the stored document and the attribute value is compared numerically to the value of the versioning attribute in the supplied document that is supposed to update/replace it. The document is only changed if the new number is higher. See the Document API for details.

Index API

optimizeTopK for inverted indexes

Indexes of type inverted accept a new optimizeTopK property for the ArangoSearch WAND optimization. It is an array of strings, optional, and defaults to [].

See the inverted index optimizeTopK property for details.

Multi-dimensional indexes

The previously experimental zkd index type is now stable and has been renamed to mdi. Existing indexes keep the zkd type. The HTTP API still allows the old name to create new indexes that behave exactly like mdi indexes but this is discouraged. The zkd alias may get removed in a future version.

An additional mdi-prefixed index variant has been added. This is a new index type in the API with the same settings as the mdi index but with one additional prefixFields attribute. It is a required setting for the mdi-prefixed index type and accepts an array of strings similar to the fields attribute. You can use it to narrow down the search space using equality checks.

Both multi-dimensional index variants now support a sparse setting (boolean) and storedValues setting (array of strings) that were not supported by the zkd index type in previous versions.

See Working with multi-dimensional indexes for details.

Progress indication on the index generation

Introduced in: v3.10.13, v3.11.7

The GET /_api/index endpoint may now include a progress attribute for the elements in the indexes array. For every index that is currently being created, it indicates the progress of the index generation (in percent).

To return indexes that are not yet fully built but are in the building phase, add the withHidden=true query parameter to the call of the endpoint.

curl "http://localhost:8529/_api/index?collection=myCollection&withHidden=true"

Optimizer rule descriptions

Introduced in: v3.10.9, v3.11.2

The GET /_api/query/rules endpoint now includes a description attribute for every optimizer rule that briefly explains what it does.

Query parsing API

The POST /_api/query endpoint for parsing AQL queries now unconditionally returns the warnings attribute, even if no warnings were produced while parsing the query. In that case, warnings contains an empty array. In previous versions, no warnings attribute was returned when parsing a query produced no warnings.

Metrics API

The metrics endpoint includes the following new metrics about AQL queries, ongoing dumps, ArangoSearch parallelism and used file descriptors:

• arangodb_aql_cursors_active
• arangodb_dump_memory_usage
• arangodb_dump_ongoing
• arangodb_dump_threads_blocked_total
• arangodb_search_execution_threads_demand
• arangodb_search_file_descriptors

The following new metrics for improved memory observability have been added:

• arangodb_agency_node_memory_usage
• arangodb_aql_cursors_memory_usage
• arangodb_index_estimates_memory_usage
• arangodb_internal_cluster_info_memory_usage
• arangodb_requests_memory_usage
• arangodb_revision_tree_buffered_memory_usage
• arangodb_scheduler_queue_memory_usage
• arangodb_scheduler_stack_memory_usage
• arangodb_search_consolidations_memory_usage
• arangodb_search_mapped_memory
• arangodb_search_readers_memory_usage
• arangodb_search_writers_memory_usage
• arangodb_transactions_internal_memory_usage
• arangodb_transactions_rest_memory_usage

Introduced in: v3.11.2

The following metrics have been added about the LZ4 compression for values in the in-memory edge cache:

• rocksdb_cache_edge_inserts_effective_entries_size_total
• rocksdb_cache_edge_inserts_uncompressed_entries_size_total
• rocksdb_cache_edge_compression_ratio

Introduced in: v3.10.11, v3.11.4

The following metrics have been added to improve the observability of in-memory cache subsystem:

• rocksdb_cache_free_memory_tasks_total
• rocksdb_cache_free_memory_tasks_duration_total
• rocksdb_cache_migrate_tasks_total
• rocksdb_cache_migrate_tasks_duration_total

Introduced in: v3.11.4

The following metrics have been added to improve the observability of in-memory edge cache:

• rocksdb_cache_edge_compressed_inserts_total
• rocksdb_cache_edge_empty_inserts_total
• rocksdb_cache_edge_inserts_total

Introduced in: v3.11.5

The following metrics have been added to monitor and detect temporary or permanent connectivity issues as well as how many scheduler threads are in the detached state:

• arangodb_network_connectivity_failures_coordinators
• arangodb_network_connectivity_failures_dbservers_total
• arangodb_scheduler_num_detached_threads

Endpoints deprecated

JavaScript Transactions API

JavaScript Transactions and thus the POST /_api/transaction endpoint is deprecated from v3.12.0 onward and will be removed in a future version. The endpoints for Stream Transactions (POST /_api/transaction/begin etc.) are unaffected.

Endpoints removed

Database target version API

The GET /_admin/database/target-version endpoint has been removed in favor of the more general version API with the endpoint GET /_api/version. The endpoint was deprecated since v3.11.3.

JavaScript-based traversal using /_api/traversal

The long-deprecated JavaScript-based traversal functionality has been removed in v3.12.0, including the REST API endpoint /_api/traversal.

The functionality provided by this API was deprecated and unmaintained since v3.4.0. JavaScript-based traversals have been replaced with AQL traversals in v2.8.0. Additionally, the JavaScript-based traversal REST API could not handle larger amounts of data and was thus very limited.

Users of the /_api/traversal REST API should use AQL traversal queries instead.

Pregel API

The /_api/control_pregel/* endpoints have been removed in v3.12.0 as Pregel graph processing is no longer supported. The arangodb_pregel_* metrics and the pregel log topic have been removed as well from the respective endpoints.

JavaScript API

Collection creation

When creating a collection using the db._create(), db._createDocumentCollection(), or db._createEdgeCollection() method, the server log now displays a deprecation message if illegal combinations and unknown properties are detected in the properties object.

Note that all invalid elements and combinations will be rejected in future versions.

@arangodb/graph/traversal module

The long-deprecated JavaScript-based traversal functionality has been removed in v3.12.0, including the bundled @arangodb/graph/traversal JavaScript module.

The functionality provided by this traversal module was deprecated and unmaintained since v3.4.0. JavaScript-based traversals have been replaced with AQL traversals in v2.8.0. Additionally, the JavaScript-based traversals could not handle larger amounts of data and were thus very limited.

Users of the JavaScript-based traversal API should use AQL traversal queries instead.

collection object

The following methods now accept a versionAttribute option that adds external versioning support:

• collection.update(object, data, options)
• collection.replace(object, data, options)
• collection.insert(data, options) when used to update/replace a document

If set, the attribute with the name specified by the option is looked up in the stored document and the attribute value is compared numerically to the value of the versioning attribute in the supplied document that is supposed to update/replace it. The document is only changed if the new number is higher. See the JavaScript API for details.

@arangodb/pregel package

The @arangodb/pregel module of the JavaScript API has been removed in v3.12.0 as Pregel is no longer supported.

db._executeTransaction() deprecated

JavaScript Transactions and thus the db._executeTransaction() method is deprecated from v3.12.0 onward and will be removed in a future version. The db._createTransaction() method for starting Stream Transactions is unaffected.