HydroCS Wiki

Pioneering Hydrologic Computational Science

User Tools

Site Tools

You are here: start » howtos » how_to_parallelize_rsync
Trace:
howtos:how_to_parallelize_rsync

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
howtos:how_to_parallelize_rsync [2026-10-15 12:09 pm] – created hchohowtos:how_to_parallelize_rsync [2026-10-15 12:34 pm] (current) – hcho
Line 1: Line 1:
 ====== How to parallelize rsync ====== ====== How to parallelize rsync ======
  
-GNU Parallel can run multiple ''rsync'' processes concurrently to transfer files in batches.+GNU parallel can run multiple ''rsync'' processes concurrently to transfer files in batches.
  
 ===== Usage ===== ===== Usage =====
 +
 +This command:
  
 <code bash> <code bash>
Line 11: Line 13:
 </code> </code>
  
-This command approximately corresponds to:+is approximately equivalent to:
  
 <code bash> <code bash>
Line 25: Line 27:
   * ''rsync -aR'': Preserve file attributes and relative directory paths.   * ''rsync -aR'': Preserve file attributes and relative directory paths.
  
-Similarly,+Similarly, this command:
  
 <code bash> <code bash>
Line 33: Line 35:
 </code> </code>
  
-This command approximately corresponds to:+is approximately equivalent to:
  
 <code bash> <code bash>
 rsync -avP path/to/src dest/ rsync -avP path/to/src dest/
 +</code>
 +
 +This command:
 +
 +<code bash>
 +find path/to/src -type f -print0 |
 +    parallel -0 -j 4 -N 100 --eta rsync -aR {} dest/ &&
 +    rsync -avRP path/to/src dest/
 +</code>
 +
 +is approximately equivalent to:
 +
 +<code bash>
 +rsync -avRP path/to/src dest/
 </code> </code>
  
Line 58: Line 74:
   * ''--del'' is not supported directly by this approach.   * ''--del'' is not supported directly by this approach.
  
-The final ''rsync'' pass handles symbolic links, empty directories, directory metadata, and any remaining file differences. The ''&&'' operator ensures that it runs only if GNU Parallel succeeds. Add ''--del'' to the final ''rsync'' command to remove extraneous destination files.+The final ''rsync'' pass handles symbolic links, empty directories, directory metadata, and any remaining file differences. The ''&&'' operator ensures that it runs only if GNU parallel succeeds. Add ''--del'' to the final ''rsync'' command to remove extraneous destination files.
  
 ===== References ===== ===== References =====
  
-  * [[https://www.gnu.org/software/parallel/man.html|GNU Parallel]] +  * [[https://www.gnu.org/software/parallel/|GNU Parallel]] 
-  * [[https://download.samba.org/pub/rsync/rsync.1|rsync]]+  * [[https://rsync.samba.org/|rsync]]
howtos/how_to_parallelize_rsync.1792066185.txt.gz · Last modified: by hcho
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki