Oasis Cloud Migration Guide
This guide will walk you through using Oasisctl to migrate your existing ArangoDB Deployment to your Oasis deployment.
In this guide we:
- Discuss the benefits of using Oasisctl
- Review common Oasisctl options for migrating data
- Perform example migration with Oasisctl
- Explore how Oasisctl handles challenges with migrations
There are a few tools ArangoDB provides for backing up, migrating, or downloading data for ArangoDB deployments.
- Arangodump is a command-line client tool to create backups of the data and structures stored in ArangoDB.
- Arangobackup is a command-line client tool for instantaneous and consistent hot backups of the data and structures stored in ArangoDB.(Enterprise only)
- Arangoexport is a command-line client tool to export data from ArangoDB servers to formats like JSON, CSV or XML for consumption by third-party tools.
- Arangoimport is a command-line client tool to import data in JSON, CSV, and TSV format to ArangoDB servers.
- Arangorestore is a command-line client tool to restore backups created by Arangodump to ArangoDB servers.
While these are all still valuable tools, when migrating from a personally managed or a non-Oasis cloud provider, Oasisctl is the best tool for the job.
The benefits of using Oasisctl for this process include:
- Little to no downtime for your running deployment
- Faster migration compared to other methods
- The entire process is scriptable
- Error handling and built-in recovery
Before You Begin
Some prerequisites for migrating are:
- Oasis API Key
- Log in using Oasisctl
- The destination Oasis Deployment ID
If you are not familiar with how to obtain an API key or how to authenticate once you have a key please see the guide for Getting Started with Oasisctl.
As a reminder, when executing commands with Oasisctl you can either supply the
--token flag along with your authentication token or you can set the
OASIS_TOKEN variable in your environment. If you use the
setx command with Windows, as described in the Getting Started guide, you must restart your terminal before it will be recognized.
Once you have successfully authenticated in your terminal of choice, obtaining the deployment ID is easy. You can list deployment details using the the
list command, for example:
oasisctl list deployments --organization-id "Oasis Organization" --project-id "Migrating to Oasis"
Notice that you will need to supply the organization and project name. If you don’t have these you can continue exploring with the
list command until you have all of the information you need. My example also includes the
--format flag and I supply JSON as it is my preferred format, this is optional.
Let’s Get Movin’
Let’s have a look at the common flags used when importing and then we will have a look at an example import. The most common flags you will use are the destination deployment id and the address, username, and password for the deployment you intend to migrate from. These flags and their definitions are:
--destination-deployment-id: Destination deployment id to import data into. It can be provided instead of address, username and password.
--source-address: Source database address to copy data from.
--source-password: Source database password if required.
--source-username: Source database username if required.
In the example below I have included one additional flag,
--included-database, when you supply this flag
import only migrates the supplied database from your local instance. Here is its definition:
--included-database: A list of database names which should be included. If provided, only these databases will be copied.
import will attempt to migrate every database you have in your local deployment to your Oasis deployment.
Now that we have gone over the basics and obtained all of the necessary information here is an example import command using the flags mentioned above:
oasisctl import --destination-deployment-id dbuiDeploymentID --included-database 'imdb' --source-address 'http://127.0.0.1:8529' --source-username 'root'` --source-password 'password'
Once you execute the command you will be shown the versions of the source database and your Oasis instance and asked to confirm your request to begin the migration. Once you hit enter the process begins with a progress bar indicating the status.
Once complete your data is immediately available in your Oasis deployment. This same process can be altered for all of your different databases you intend to bring to the cloud.
Time to Kick Back
Well done! Hopefully that is the end of the migration process and you can kick back and relax knowing your data is securely stored in Oasis!
However, to make sure you don’t end up thirsty at a mirage we have prepared some tips for handling surprises while migrating.
Trouble in Paradise?
As with all things in the world, something could go wrong while importing data to your deployment. The rest of this guide explores some of these situations.
One of the most common problems during a long import sequence can be that the network falters and has a stutter. This could be a broken pipe, or a dropped packet or a full split.
Oasisctl import in that case will pause for a second at first and then gradually back off and retry with an increasing wait time.
This can be configured via:
-r, --max-retries: The number of maximum retries attempts. Increasing this number will also increase the exponential fallback timer. (default 9)
In some cases, the database might fail to receive the data or will fail to respond or the cursor can break. In this case, import will try the same data again until it either succeeds or fails completely. In which case it will abort the import process.
Oasis import, by default, will overwrite everything in the destination database. This can be prevented with the use of these options:
--excluded-collection: A list of collections names which should be excluded. Exclusion takes priority over inclusion.
--excluded-graph: A list of graph names which should be excluded. Exclusion takes priority over inclusion.
--excluded-view: A list of view names which should be excluded. Exclusion takes priority over inclusion.
--excluded-database: A list of database names which should be excluded. Exclusion takes priority over inclusion.
Something even worse than a split can be a slow network. Import uses parallel processing of several collections at a time. This can be configured using
-maximum-parallel-collections. Another great way of speeding up things is increasing
--batch-size. This corresponds to the ArangoDB batch size in other tools.
If all things break, or you have to abort a long-running process,
import can continue stopped work. During the migration,
import stores what it has already completed in terms of database and collections. If the process is aborted with ctrl+c or is unable to recover after an unknown failure, it will display something along these lines:
It will tell the user what databases and collections have already been completed and give a list of items in the form of exclude filters to continue work where it left off. It will ignore the items in the exclude filter and just continue with the next database. This will ensure that it’s possible to import several gigabytes of data without having to worry that the process halts in the middle and one has to start from the beginning.