HydroCS Wiki

Pioneering Hydrologic Computational Science

User Tools

Site Tools


howtos:how_to_parallelize_rsync

This is an old revision of the document!


How to parallelize rsync

GNU parallel can run multiple rsync processes concurrently to transfer files in batches.

Usage

find path/to/src/. -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avP path/to/src/ dest/

This command approximately corresponds to:

rsync -avP path/to/src/ dest/

Options:

  • -print0 and -0: Safely handle filenames containing whitespace, including newlines.
  • -j 4: Run up to four concurrent jobs.
  • -N 100: Transfer up to 100 files per job.
  • –eta: Display progress and estimated completion time.
  • rsync -aR: Preserve file attributes and relative directory paths.

Similarly,

find path/to/./src -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avP path/to/src dest/

This command approximately corresponds to:

rsync -avP path/to/src dest/

Directory Structure

The position of /./ determines which path components are preserved by rsync -R.

  • path/to/src/. copies the contents of src/ directly into dest/.
  • path/to/./src preserves src/ under dest/ but removes path/to/.
  • /path/to/src/. copies the contents of src/ directly into dest/.
  • /path/to/./src preserves src/ under dest/ but removes /path/to/.

For absolute paths, explicitly position /./ to preserve only the intended directory structure. Without /./, rsync -R preserves additional leading path components.

Limitations

  • Only regular files are transferred; symbolic links and empty directories are excluded.
  • Directory metadata may not be fully preserved.
  • Progress is measured by completed batches, not bytes.
  • Parallel transfers may be slower on I/O-constrained systems.
  • –del is not supported directly by this approach.

The final rsync pass handles symbolic links, empty directories, directory metadata, and any remaining file differences. The && operator ensures that it runs only if GNU parallel succeeds. Add –del to the final rsync command to remove extraneous destination files.

References

howtos/how_to_parallelize_rsync.1792066651.txt.gz · Last modified: by hcho

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki