Rclone is an extremely powerful command-line tool, mainly used for syncing, backing up, and migrating files between different cloud storage services. It supports a wide range of cloud storage providers, such as Google Drive, OneDrive, Dropbox, and more.

This article covers Rclone’s installation and configuration in detail, walks through its common commands (such as copy, move, sync, and so on) and various flags (such as include and exclude), and explains how to write a filter. Everything here can be found on Rclone’s official site — this is mainly an organized version, meant to serve as a “quick” reference.

1. What Is Object Storage

Before using Rclone, we need to understand the concept of object storage. Object storage is a way of storing large amounts of unstructured data. Rclone supports most object storage services — internationally, providers like Google, Microsoft, and Amazon; domestically, providers like Alibaba Cloud, Baidu Cloud, Tencent Cloud, and Volcano Engine. See here for the detailed list.

Cyber bodhisattva Cloudflare and Oracle Cloud are especially recommended here.

Besides public object storage, Rclone also supports self-hosted object storage such as MinIO. See this guide for setting up a local MinIO object storage instance.

2. Installing Rclone

There are several ways to install Rclone — pick whichever one works for you.

2.1. Installing directly via a package manager

sudo dnf install rclone # RedHat
sudo apt install rclone # Ubuntu
brew install rclone # MacOS

2.2. Installing via the unified install script

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Installing the beta version:

sudo -v ; curl https://rclone.org/install.sh | sudo bash -s beta

2.3. Manually downloading and installing the package

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64

3. Configuring Rclone

There are two ways to configure it — using rclone config, or directly editing the ~/.config/rclone/rclone.conf file. In fact, the first approach ultimately edits the config file too. A typical config file looks like:

[aoi]
type = s3
access_key_id = xxxx
secret_access_key = yyyy
endpoint = http://oss.aoi.ai

Here, access_key_id and secret_access_key come from the cloud provider, and endpoint is the address. The part you get to choose yourself is the name. In later commands, the remote address is aoi:xxxx, where xxxx is the bucket name. Rclone’s operands can also be entirely local filesystem paths.

3.1. Configuring Cloudflare R2 storage

Cyber bodhisattva Cloudflare provides free object storage. A dedicated article on the specific steps will be written later.

4. Rclone Commands

Below are Rclone’s commonly used commands.

4.1. Listing: ls, lsl, lsd, lsf, lsjson

  • ls: lists only the size and path of objects.
  • lsl: lists an object’s modification time, size, and path.
  • lsd: lists directories only.
  • lsf: lists objects and directories in an easy-to-parse format.
  • lsjson: lists objects and directories in JSON format.

The output of ls, lsl, and lsd is human-readable. lsf’s output is readable by both humans and machines, while lsjson is designed to be machine-readable — its name already tells you the output is JSON. ls and lsf are recursive by default; adding --max-depth 1 controls the recursion depth. lsd, lsf, and lsjson are not recursive by default; add -R to make them recursive.

Rclone is very fast — using it via subprocess in Python is much faster than using an S3 library directly in Python.

4.2. Syncing: sync

This is Rclone’s most important feature. It syncs files from the source directory to the destination directory. If a file with the same name exists in both the source and destination, Rclone decides whether to overwrite the destination file based on modification times. If the source file is newer than the destination file, or the destination file doesn’t exist, Rclone copies the source file to the destination. If the destination file is newer than the source file, Rclone decides whether to overwrite the source file based on configuration.

rclone sync source: destination: [options]

For example:

rclone sync local_folder:remote_drive:backup_folder --exclude "*.tmp"

This syncs files from the local folder local_folder to the backup_folder folder on the remote drive remote_drive, while excluding every file ending in .tmp.

For a command with this much potential for destruction, you can add the “regret pill” --dry-run, which just lists the operations that would be performed and the files that would be affected, without actually carrying them out. The following two flags are also quite useful in practice:

  • -P — progress.
  • --verbose: shows detailed information.

4.3. Copying: copy

Copies files from the source directory to the destination directory. Unlike sync, copy doesn’t delete files in the destination that don’t exist in the source. It simply copies files from the source to the destination; if a file with the same name already exists at the destination, Rclone decides whether to overwrite it based on configuration.

rclone copy source: destination: [options]

For example:

rclone copy remote_drive:source_folder local_folder --include "*.pdf"

This command copies every file ending in .pdf under the source_folder folder on the remote drive to the local folder local_folder.

It’s worth explaining the difference between sync and copy here: sync keeps the contents of two folders identical, while copy is simply a copy operation.

4.4. Moving: move

rclone move source: destination: [options]

Moves files from the source directory to the destination directory. This means files in the source directory will be deleted, and the same files will appear in the destination directory. If a file with the same name already exists at the destination, Rclone decides whether to overwrite it based on configuration.

rclone move local_folder:remote_drive:archive_folder --exclude "*.log"

This moves files from the local folder local_folder to the archive_folder folder on the remote drive remote_drive, while excluding every file ending in .log.

4.5. Deleting: delete

Used to delete a specific file, if given an address:

rclone delete remote:path [options]

Deletes files or directories in remote storage. This command needs to be used carefully, since once deleted, files may not be recoverable.

For example:

rclone delete remote_drive:old_folder --dry-run

This command simulates deleting the old_folder folder on the remote drive without actually performing the deletion — it just shows what would happen if the delete command were actually run.

4.6. Purging: purge

rclone purge remote:path [options]

Deletes a specified path in remote storage along with all of its contents, including subdirectories and files. This command is more thorough than delete, since it removes the entire directory structure.

For example:

rclone purge remote_drive:old_data_folder

This command deletes the old_data_folder folder on the remote drive along with all of its contents.

4.7. The check command: check

rclone check source: destination: [options]

Compares files in the source and destination directories to determine whether they’re identical. If files differ, Rclone reports the differences.

For example:

rclone check local_folder:remote_drive:backup_folder

This checks whether the files in the local folder local_folder and the backup_folder folder on the remote drive are identical.

4.8. The mount command: mount

rclone mount source: mountpoint [options]

Mounts remote storage into the local filesystem, letting you access remote storage as if it were a local disk.

For example:

rclone mount remote_drive: /mnt/remote_drive --vfs-cache-mode writes

This mounts the remote drive at /mnt/remote_drive in the local filesystem, and sets the virtual filesystem cache mode to writes to improve write performance.

4.9 Viewing: cat

Use the cat command just like you would to view a local file.

4.10. Summary

Naturally, there are too many commands to remember all at once. That’s fine — come back here when you forget one.

5. Filtering: include and exclude

This mainly applies to commands that change files, such as sync, move, ls, lsl, md5sum, sha1sum, size, delete, and check. The purge command doesn’t support filters.

Sometimes when running Rclone operations, you need to apply a series of filters to files — that’s when filtering comes in.

When working with Rclone, filtering files is often necessary, and that’s what filter is for. filter is a fairly comprehensive feature that, in practice, usually reduces to include or exclude. In fact, shell syntax itself already has some notion of wildcards. Avoid using include, exclude, and filter at the same time, since Rclone may run into unexpected issues when parsing them together.

Rclone’s matching rules follow glob-style syntax:

*         matches any sequence of non-separator (/) characters
**        matches any sequence of characters, including the / separator
?         matches any single non-separator (/) character
[ [! ] { character-range } ]
          a character class (must be non-empty)
{ pattern-list }
          pattern alternatives
{{ regexp }}
          regular expression matching
c         matches the character c (c != *, **, ?, \, [, {, })
\c        matches the reserved character c (c = *, **, ?, \, [, {, }) or a character class

character-range:

c         matches the character c (c != \, -, ])
\c        matches the reserved character c (c = \, -, ])
lo - hi   matches character c such that lo <= c <= hi

pattern-list:

pattern {, pattern }
          a comma-separated (no spaces) list of patterns

6. Summary

This article gave a comprehensive overview of the various ways to use Rclone.