====== How to parallelize rsync ======

GNU parallel can run multiple ''rsync'' processes concurrently to transfer files in batches.

===== Usage =====

This command:

<code bash>
find path/to/src/. -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avP path/to/src/ dest/
</code>

is approximately equivalent to:

<code bash>
rsync -avP path/to/src/ dest/
</code>

Options:

  * ''-print0'' and ''-0'': Safely handle filenames containing whitespace, including newlines.
  * ''-j 4'': Run up to four concurrent jobs.
  * ''-N 100'': Transfer up to 100 files per job.
  * ''--eta'': Display progress and estimated completion time.
  * ''rsync -aR'': Preserve file attributes and relative directory paths.

Similarly, this command:

<code bash>
find path/to/./src -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avP path/to/src dest/
</code>

is approximately equivalent to:

<code bash>
rsync -avP path/to/src dest/
</code>

This command:

<code bash>
find path/to/src -type f -print0 |
    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
    rsync -avRP path/to/src dest/
</code>

is approximately equivalent to:

<code bash>
rsync -avRP path/to/src dest/
</code>

===== Directory Structure =====

The position of ''/./'' determines which path components are preserved by ''rsync -R''.

  * ''path/to/src/.'' copies the contents of ''src/'' directly into ''dest/''.
  * ''path/to/./src'' preserves ''src/'' under ''dest/'' but removes ''path/to/''.
  * ''/path/to/src/.'' copies the contents of ''src/'' directly into ''dest/''.
  * ''/path/to/./src'' preserves ''src/'' under ''dest/'' but removes ''/path/to/''.

For absolute paths, explicitly position ''/./'' to preserve only the intended directory structure. Without ''/./'', ''rsync -R'' preserves additional leading path components.

===== Limitations =====

  * Only regular files are transferred; symbolic links and empty directories are excluded.
  * Directory metadata may not be fully preserved.
  * Progress is measured by completed batches, not bytes.
  * Parallel transfers may be slower on I/O-constrained systems.
  * ''--del'' is not supported directly by this approach.

The final ''rsync'' pass handles symbolic links, empty directories, directory metadata, and any remaining file differences. The ''&&'' operator ensures that it runs only if GNU parallel succeeds. Add ''--del'' to the final ''rsync'' command to remove extraneous destination files.

===== References =====

  * [[https://www.gnu.org/software/parallel/|GNU Parallel]]
  * [[https://rsync.samba.org/|rsync]]