This is an old revision of the document!
Table of Contents
How to parallelize rsync
GNU parallel can run multiple rsync processes concurrently to transfer files in batches.
Usage
find path/to/src/. -type f -print0 | parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ && rsync -avP path/to/src/ dest/
This command approximately corresponds to:
rsync -avP path/to/src/ dest/
Options:
-print0and-0: Safely handle filenames containing whitespace, including newlines.-j 4: Run up to four concurrent jobs.-N 100: Transfer up to 100 files per job.–eta: Display progress and estimated completion time.rsync -aR: Preserve file attributes and relative directory paths.
Similarly,
find path/to/./src -type f -print0 | parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ && rsync -avP path/to/src dest/
This command approximately corresponds to:
rsync -avP path/to/src dest/
Directory Structure
The position of /./ determines which path components are preserved by rsync -R.
path/to/src/.copies the contents ofsrc/directly intodest/.path/to/./srcpreservessrc/underdest/but removespath/to/./path/to/src/.copies the contents ofsrc/directly intodest/./path/to/./srcpreservessrc/underdest/but removes/path/to/.
For absolute paths, explicitly position /./ to preserve only the intended directory structure. Without /./, rsync -R preserves additional leading path components.
Limitations
- Only regular files are transferred; symbolic links and empty directories are excluded.
- Directory metadata may not be fully preserved.
- Progress is measured by completed batches, not bytes.
- Parallel transfers may be slower on I/O-constrained systems.
–delis not supported directly by this approach.
The final rsync pass handles symbolic links, empty directories, directory metadata, and any remaining file differences. The && operator ensures that it runs only if GNU parallel succeeds. Add –del to the final rsync command to remove extraneous destination files.