Migrating to ArvanCloud Object Storage Using Rclone

Rclone is an effective alternative to rsync. Designed to communicate with cloud storages that are compatible with S3, this tool helps you migrate and transfer data from one bucket to another, even if the buckets are hosted on different service providers.
Using the open-source Rclone tool, you will be able to back up data from one server or a cloud storage and store it on another server, object storage or cloud. This software offers several options, including bucket management, mounting virtual drives via a bucket, managing files, and locating duplicate files in addition to backups.
By adding ArvanCloud Object Storage to the list of Rclone providers, using it for ArvanCloud users will be extremely easy and optimal.
This guide will introduce and examine the method of transferring data using Rclone.

Installing Rclone

Depending on your operating system, you will need to perform several steps.

Installing Rclone on Windows

To install this software on a Windows operating system, you simply must download it from the Rclone website. After that, you should execute the downloaded file via the appropriate path and through CLI.

Installing Rclone on Linux

You will need to follow the steps below to install Rclone on Unix-based operating systems:

• Connect to your server or system as the root user.
• Update APT:

  apt update && apt upgrade -y

• By running the following commands respectively, download and install Rclone:

  git clone https://github.com/rclone/rclone.git
  cd rclone
  go build

  Before running these commands, you must install git and go1.17 on your operating system. Do this by running the following commands:

  apt-get install git
  tar -C /usr/local -xzf go1.17.5.linux-amd64.tar.gz
  wget https://dl.google.com/go/go1.17.7.linux-amd64.tar.gz
  sudo tar -xvf go1.17.7.linux-amd64.tar.gz
  sudo mv go /usr/local
  export GOROOT=/usr/local/go
  export GOPATH=$HOME/Projects/Proj1
  export PATH=$GOPATH/bin:$GOROOT/bin:$PATH

• Once Rclone is installed, type the following command to verify that the version and the installation are correct:

  ./rclone version

To make it easier to run the ./rclone commands, you can define an alias to rclone in ~/.bashrc.

Rclone Configuration and Settings

Use the following command to start configuring Rclone:

 rclone config

You will see the following output in case you have not already set up Rclone on your system:

When you have already configured Rclone, a different output will be displayed, but the new configuration will not affect the next steps.

Type n to create a new remote control or configuration. Then you will be prompted to enter the name of your choice. In this case, we have chosen ArvanCloud.

The following output will be shown:

Type S3 and press Enter. Here’s what the output looks like:

Type ArvanCloud and press Enter to confirm Arvan Cloud as the S3 provider. The output below will be displayed:

Type False and press Enter as to enter the credentials in the next step. You will be shown an output similar to the image below at this point:

Copy your access key from the panel and enter it.

Copy your secret key from the panel and enter it.

Type 1 for Asiatech and 2 for Shahriar, and then press Enter.

Enter the endpoint value from Tabriz (2) and Tehran (1). The output will be like the one in the image:

Choose the ACL you wish to use and type it in. For example, we will choose private here. Then enter n to see a summary of the configuration:

Type y to confirm the configuration. This will give you the following output:

In the end, type q to exit the configuration.

Note: When you want to transfer data from another provider’s bucket to the provider you have just configured, simply repeat the steps to set up a new remote. You can easily select the desired provider in steps 2 and 3.

Transferring Data

There are two commands to transfer data between two buckets:

1. Use the copy command to copy the data from the source to the destination:

   rclone copy --progress <SOURCE_BACKEND>:<SOURCE_PATH> <DEST_BACKEND>:<DEST_PATH>

   For example, the following command will copy the data from the my-first-bucket bucket in the ArvanCloud remote, which we set earlier, to the my-second-bucket bucket in the same remote. Using the –progress flag, we can track the progress of this process.

   rclone copy --progress ArvanCloud:my-first-bucket ArvanCloud:my-second-bucket

2. The sync command will copy the data from the source to the destination, except that it will delete the objects that are in the destination bucket but not in the source bucket:

   rclone sync --progress <SOURCE_BACKEND>:<SOURCE_PATH> <DEST_BACKEND>:<DEST_PATH>

   The following command, for example, will copy the data from the my-first-bucket bucket in the ArvanCloud remote we set previously to the my-third-bucket in the AmazonAws remote. Any data that is in my-third-bucket and not in my-first-bucket will be removed.

You can find more information and an overview of the available commands in the official Rclone documentation.