From Notes Wiki
Jump to: navigation, search
Home > CentOS > CentOS 7.x > Web Based Tools > Owncloud > CentOS 7.x Owncloud upload files parallelly via weddav

CentOS 7.x Owncloud upload files parallelly via weddav

Uploading files to owncloud sequentially using rsync or cp can be slow. To upload multiple files in parallel use:

  1. yum -y install parallel
  2. Create list of files to be copied by comparing only size. This is required as owncloud creates its own timestamps and davfs2 timestamps shown on command-line are not copied to backend.
    rsync -nvr --size-only /mnt/source/ /mnt/owncloud-dest/ > /root/copy-list.txt 2>/root/error-list.txt &
    and wait for file-list to be created
  3. Remove first line similar to:
    sending incremental file list
    and last 3 lines similar to:
    sent 65 bytes received 19 bytes 168.00 bytes/sec
    total size is 0 speedup is 0.00 (DRY RUN)
    from the created files
  4. Use parallel to copy files in parallel using above list to owncloud
    cd /mnt/source ##Very important
    cat /root/copy-list.txt | parallel --will-cite -j 5 cp -v --parents {} /mnt/owncloud-dest/ > /root/cp-output.txt 2>&1 &
    where -j 5 indicates 5 parallel copies at any time.
  5. At any time see 5 copy process running using:
    ps aux | grep "cp -v"
    Also
    ps aux | grep "cp -v" | wc -l
    will show 7 (2 more than -j value) due to grep, parallel commands also getting grepped
  6. To continuously monitor uploads use:
    watch "ifconfig br0; echo -n "No of copy processes:"; ps aux | grep 'cp -v' | wc -l; echo -n "No of files copied: "; grep -v 'cannot\|omitting' /root/cp-output.txt | wc -l; echo; df -h /; echo; du -sh /var/cache/davfs2; echo; tail /root/cp-output.txt"
    where br0 should be replaced with name of interface. This is monitoring:
    1. Interface statistics to get idea on uploads
    2. No of parallel cp processes running.
    3. No of files copied based on no. of lines in /root/cp-output.txt file
    4. Space in "/". Necessary to monitor this to ensure that cache space is not too large to accommodate in "/" filesystem.
    5. Disk space usage of davfs2 cache folder
    6. Last 10 copied files
  7. If davfs2 size increases automatically, to pause and continue above processes automatically use:
    ps aux | grep parallel
    while true; do sleep 7200; kill -19 <parallel-pid>; sleep 3600; kill -18 <parallel-pid>; done
    where <parallel-pid> is the PID of parallel process as seen in output of ps command.
    This will allow parallel to spawn processes for 2 hours and then pause it for 1 hour and then again continue it for another 2 hours and so on.


Refer:


Home > CentOS > CentOS 7.x > Web Based Tools > Owncloud > CentOS 7.x Owncloud upload files parallelly via weddav