AWS CLI 是否将大于 4GB 的文件上传到 Amazon Glacier？

Question

AWS 命令​​行界面 (CLI) 允许 to upload a file to AWS Glacier。但在 AWS Rest API 中也有一个 limit of 4GB for file uploads。如果我需要通过 Rest API 上传大于 4GB 的文件，我需要使用分段上传。

我的问题是：AWS CLI 是否在内部处理大于 4GB 的文件上传，或者在处理大于 4GB 的文件时我是否需要自己处理分段上传？我可以将一个 20Gb 的文件传递给 AWS CLI 的 upload-archive 选项，它就可以工作吗？如果 CLI 不能直接处理大文件上传，有什么命令行工具可以帮我完成（让我免于在部分上传失败时实现所有校验和计算、错误处理和重试逻辑）？

我了解 AWS Rest API 有 4GB 限制，但我找不到任何关于如何在 CLI 中处理此限制的信息。我可以做测试，但我的上传速度不是那么快，我担心会浪费几个小时才发现它不起作用。

Answer 1

我正在使用 glacier-cmd (https://github.com/uskudnik/amazon-glacier-cmd-interface)，效果很好，但最近似乎不受支持。有时它会因大文件（~50GB）而超时。

Answer 2

下面的脚本可以正常工作。我已经为树哈希计算和文件部分单独上传创建了块。它工作得很好。

#!/bin/bash

date1=$(date +"%s")

byteSize=1073741824

CHUNK_SIZE=1073741824

hashsize=1048576

if [[ -z "${1}" ]]; then
    echo "No file provided."
    exit 1
fi

ARCHIVE="/mnt/dbfiles/mahipal/splitfiles/${1}"
ARCHIVE_SIZE=`cat "${ARCHIVE}" | wc --bytes`

cd /mnt/dbfiles/mahipal/splitfiles

rm -rf TEMP

rm -rf HASH

mkdir TEMP

mkdir HASH

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

date3=$(date +"%s")

split -d --bytes=${CHUNK_SIZE} "${ARCHIVE}" chunk  -a 4

date4=$(date +"%s")

diff2=$(($date4-$date3))

cd /mnt/dbfiles/mahipal/splitfiles/HASH 

date5=$(date +"%s")

split -d --bytes=${hashsize} "${ARCHIVE}" chunk  -a 5

date6=$(date +"%s")

diff3=$(($date6-$date5))

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

lastpartsize=`expr $(ls -l | tail -1 | awk '{print$5}') + 0`

lastfile=$(ls -l | tail -1 | awk '{print$9}')

cont=$(ls -l | wc -l)
cnt=`expr $cont - 2`

fileCount=$(ls -1 | grep "^chunk" | wc -l)
echo "Total parts to upload: " $fileCount

files=$(ls | grep "^chunk")

init=$(/bin/aws glacier initiate-multipart-upload --account-id - --part-size $byteSize --vault-name final_vault --archive-description "${1}_${ARCHIVE_SIZE}_${byteSize}")

echo "---------------------------------------"


uploadId=$(echo $init | jq '.uploadId' | xargs)

touch commands.txt

i=0
for f in $files 
  do
     byteStart=$((i*byteSize))
     byteEnd=$((i*byteSize+byteSize-1))
     echo /bin/aws glacier upload-multipart-part --body $f --range "'"'bytes '"$byteStart"'-'"$byteEnd"'/*'"'" --account-id - --vault-name final_vault --upload-id $uploadId >> commands.txt
     i=$(($i+1))

    if [ "$i" == "$cnt" ]
        then
    byteEnd=`expr $byteEnd + 1`
    byteEnd2=$((i*byteSize+lastpartsize-1))
    byteSize=$lastpartsize 
     echo /bin/aws glacier upload-multipart-part --body $lastfile --range "'"'bytes '"$byteEnd"'-'"$byteEnd2"'/*'"'" --account-id - --vault-name final_vault --upload-id $uploadId >> commands.txt
    break   
    fi      

  done

parallel --load 100% -a commands.txt --no-notice --bar

cd /mnt/dbfiles/mahipal/splitfiles/HASH

files=$(ls | grep "^chunk")

for f in $files
  do
     openssl dgst -sha256 -binary ${f} > "hash${f:5}"

  done


echo "List Active Multipart Uploads:"
echo "Verify that a connection is open:"

/bin/aws glacier list-multipart-uploads --account-id - --vault-name final_vault >> /mnt/dbfiles/mahipal/splitfiles/TEMP/commands.txt

echo "-------------"
echo "Contents of commands.txt"

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

cat commands.txt

# Calculate tree hash.

cd /mnt/dbfiles/mahipal/splitfiles/HASH

echo "Calculating tree hash..."
while true; do
    COUNT=`ls hash* | wc -l`
    if [[ ${COUNT} -le 2 ]]; then
        TREE_HASH=$(cat hash* | openssl dgst -sha256 | awk '{print $2}')
        break
    fi
    ls hash* | xargs -n 2 | while read PAIR; do
        PAIRARRAY=(${PAIR})
        if [[ ${#PAIRARRAY[@]} -eq 1 ]]; then
            break
        fi
        cat ${PAIR} | openssl dgst -sha256 -binary > temphash
        rm ${PAIR}
        mv temphash "${PAIRARRAY[0]}"
    done
done

cd /mnt/dbfiles/mahipal/splitfiles/TEMP

echo "Finalizing..."
/bin/aws glacier complete-multipart-upload  --account-id=- --vault-name="final_vault" --upload-id="$uploadId" --checksum="${TREE_HASH}" --archive-size=${ARCHIVE_SIZE} >>commands.txt
RETVAL=$?
if [[ ${RETVAL} -ne 0 ]]; then
    echo "complete-multipart-upload failed with status code: ${RETVAL}" >>commands.txt
    echo "Aborting upload ${uploadId}"  >>commands.txt
    /bin/aws glacier abort-multipart-upload --account-id=- --vault-name="final_vault" --upload-id="${uploadId}" >>commands.txt
    exit 1
fi

echo "--------------"
echo "Deleting temporary commands.txt file"

#rm commands.txt

date2=$(date +"%s")

diff=$(($date2-$date1))

echo "Total Split Duration for Chunk Part Size: $(($diff2/ 3600 )) hours $((($diff2 % 3600) / 60)) minutes $(($diff2 % 60)) seconds" >>commands.txt 

echo "Total Split Duration for hash Part Size: $(($diff3/ 3600 )) hours $((($diff3 % 3600) / 60)) minutes $(($diff3 % 60)) seconds"  >>commands.txt

echo "Total upload Duration: $(($diff/ 3600 )) hours $((($diff % 3600) / 60)) minutes $(($diff % 60)) seconds" >>commands.txt

echo "Done."
exit 0