Sync Cloud Storage Drives using Rclone
Created: Wed 29 Mar 2023 Updated: 10 months ago
Rclone is an Open-source, multi-threaded, command line tool. It makes data synchronization much more easier. Whether you are doing Data migration or Shifting from on-premise to cloud, Rclone makes the process so much easier.
By taking advantage of Rclone, you can finish the days work in hours. With the help of Rclone, you can do Data Synchronization between two or more different Storage providers.
With the help of Rclone, you have the ability to clone or sync your data from Storage_A to Storage_B. You can even sync data between multiple storages or from cloud storage to your local system.
Getting started with Rclone
Installing Rclone is a pretty straightforward process. To download the Rclone package, You can go to Rclone official website or you can simply copy and paste the below command: (you can copy the command simply by clicking on it)
sudo -v ; curl https://rclone.org/install.sh | sudo bash
This command will install the latest version (stable) of Rclone in your system. Upon
successful installation, run rclone config
command to start the configuration process of Rclone. Follow the steps shown below:
If you are not sure about any option, you can simply press ENTER to use the default value for that option. After following through the process as described above in the video, a new browser window will open. Here you need to log in to your Google drive credentials and give permission to Rclone. You will get 'Success' message after authentication. This will generate a token for Rclone to use Google Drive via an API to perform actions on drive as instructed by you using the Rclone command line.
You can attach and generate tokens for all your required storages. Then by using Rclone command line, you can sync data between those storages.
Data safety measure:
- Encryption is essential nowdays. Rclone provides an option to encrypt data transferring between different storage providers.
- Secure Connection such as HTTPS is also recommended while transferring data over the internet. It ensures that data is secure and cannot be intercepted.
- Make sure that the destination directory does not hold any data because the data in destination directory prior to Synchronization will be moved to trash.
Sync Google drive data with Local system:
rclone sync gdrive:Files /home/user/gdrive_sync
This command will sync Files named directory from my Google drive with my local directory name gdrive_sync. After the process, all the data contain in my Files directory in google drive will be locally available in my system inside /home/user/gdrive_sync directory.
Sync data from Local system to Google Drive:
rclone sync /home/user/gdrive_sync gdrive:Files
With this command, data in the local system directory will be synced with Google drive directory. These are some commands you can use to list files and directories from your storage:
rclone ls gdrive:
$
rclone lsd gdrive:
For more commands and usage of Rclone, you can run rclone -h command to see various commands that
can be used with Rclone.
Sync data between Amazon S3 and Google drive
You can again run rclone config
command and add new remote for Amazon S3. However, the configuration will be different as
compared to google drive. You can refer to Rclone S3 guide to create new remote of S3. After creating both Google drive (or any other storage) and Amazon S3, you can run the below command:
rclone sync s3:bucket-name/ gdrive:google-drive-folder-name/
This command will sync your data from S3 bucket to Google drive. You can use different storage providers as per requirement but the underlying process is the same, which is configuring rclone remotely specifically to work with a specific storage drive.
