How to parallelize rsync

GNU parallel can run multiple rsync processes concurrently to transfer files in batches.

This command:

find path/to/src/. -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avP path/to/src/ dest/

is approximately equivalent to:

rsync -avP path/to/src/ dest/

Options:

• -print0 and -0: Safely handle filenames containing whitespace, including newlines.
• -j 4: Run up to four concurrent jobs.
• -N 100: Transfer up to 100 files per job.
• –eta: Display progress and estimated completion time.
• rsync -aR: Preserve file attributes and relative directory paths.

Similarly, this command:

find path/to/./src -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avP path/to/src dest/

is approximately equivalent to:

rsync -avP path/to/src dest/

This command:

find path/to/src -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avRP path/to/src dest/

is approximately equivalent to:

rsync -avRP path/to/src dest/

The position of /./ determines which path components are preserved by rsync -R.

• path/to/src/. copies the contents of src/ directly into dest/.
• path/to/./src preserves src/ under dest/ but removes path/to/.
• /path/to/src/. copies the contents of src/ directly into dest/.
• /path/to/./src preserves src/ under dest/ but removes /path/to/.

For absolute paths, explicitly position /./ to preserve only the intended directory structure. Without /./, rsync -R preserves additional leading path components.

• Only regular files are transferred; symbolic links and empty directories are excluded.
• Directory metadata may not be fully preserved.
• Progress is measured by completed batches, not bytes.
• Parallel transfers may be slower on I/O-constrained systems.
• –del is not supported directly by this approach.

The final rsync pass handles symbolic links, empty directories, directory metadata, and any remaining file differences. The && operator ensures that it runs only if GNU parallel succeeds. Add –del to the final rsync command to remove extraneous destination files.

• GNU Parallel
• rsync