CentOS 7.x Owncloud upload files parallelly via weddav

<yambe:breadcrumb>CentOS_7.x_owncloud|Owncloud</yambe:breadcrumb>

CentOS 7.x Owncloud upload files parallelly via weddav

Uploading files to owncloud sequentially using rsync or cp can be slow. To upload multiple files in parallel use:

1. yum -y install parallel
2. Create list of files to be copied by comparing only size. This is required as owncloud creates its own timestamps and davfs2 timestamps shown on command-line are not copied to backend.

   rsync -nvr --size-only /mnt/source/ /mnt/owncloud-dest/ > /root/copy-list.txt 2>/root/error-list.txt &

   and wait for file-list to be created

3. Remove first line similar to:

   sending incremental file list

   and last 3 lines similar to:

   sent 65 bytes  received 19 bytes  168.00 bytes/sec
   total size is 0  speedup is 0.00 (DRY RUN)

   from the created files

4. Use parallel to copy files in parallel using above list to owncloud

   cd /mnt/source ##Very important
   cat /root/copy-list.txt | parallel --will-cite -j 5  cp -v --parents {} /mnt/owncloud-dest/ > /root/cp-output.txt 2>&1 &

   where -j 5 indicates 5 parallel copies at any time.

5. At any time see 5 copy process running using:

   ps aux | grep "cp -v"

   Also

   ps aux | grep "cp -v" | wc -l

   will show 7 (2 more than -j value) due to grep, parallel commands also getting grepped

6. To continuously monitor uploads use:

   watch "ifconfig br0; echo -n "No of copy processes:"; ps aux | grep 'cp -v' | wc -l; echo -n "No of files copied: "; grep -v 'cannot\|omitting' /root/cp-output.txt | wc -l; echo; df -h /; echo; du -sh /var/cache/davfs2; echo; tail /root/cp-output.txt"

   where br0 should be replaced with name of interface. This is monitoring:

   1. Interface statistics to get idea on uploads
   2. No of parallel cp processes running.
   3. No of files copied based on no. of lines in /root/cp-output.txt file
   4. Space in "/". Necessary to monitor this to ensure that cache space is not too large to accommodate in "/" filesystem.
   5. Disk space usage of davfs2 cache folder
   6. Last 10 copied files
7. If davfs2 size increases automatically, to pause and continue above processes automatically use:

   ps aux | grep parallel
   
   while true; do sleep 7200; kill -19 <parallel-pid>; sleep 3600; kill -18 <parallel-pid>; done

   where <parallel-pid> is the PID of parallel process as seen in output of ps command.

   This will allow parallel to spawn processes for 2 hours and then pause it for 1 hour and then again continue it for another 2 hours and so on.

Refer:

• http://www.yourownlinux.com/2015/04/speed-up-file-transfers-using-rsync-with-gnu-parallel.html
• https://bash.cyberciti.biz/guide/Sending_signal_to_Processes

<yambe:breadcrumb>CentOS_7.x_owncloud|Owncloud</yambe:breadcrumb>