从 Amazon S3 存储桶下载文件的脚本

Question

尝试编写脚本以从 Amazon S3 存储桶下载文件。

在 cURL 网站上的示例遇到问题。下面的脚本产生：

我们计算的请求签名与您的签名不匹配 假如。检查您的密钥和签名方法。

感谢任何帮助。

#!/bin/sh 
file="filename.php"
bucket="my-bucket"
resource="/${bucket}/${file}"
contentType="text/html"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key='ABCABCABCABCABCABCAB'
s3Secret='xyzxyzyxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzxyzx'
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -      binary | base64`

curl -v -H "Host:lssngen-updates-east.s3.amazonaws.com" \
        -H "Date:${dateValue}" \
        -H "Content-Type:${contentType}" \
        -H "Authorization: AWS ${s3Key}:${signature}" \
        https://${bucket}.s3.amazonaws.com/${file}

Answer 1

一个真正有效的解决方案（工作于 2021 年 12 月）是使用此脚本。使用前只需要导出密钥（或将其值复制到 .sh 文件）。

export AWS_ACCESS_KEY_ID=AKxxx
export AWS_SECRET_ACCESS_KEY=zzzz

只需运行即可下载

./s3download.sh get s3://mybucket/myfile.txt myfile.txt

您只需要传递get、s3 存储桶以及文件名和输出文件。

需要脚本

创建一个s3download.sh 文件和chmod +x s3download.sh，用于上面的命令。

#!/bin/bash
set -eu
s3simple() {
  local command="$1"
  local url="$2"
  local file="${3:--}"

  # todo: nice error message if unsupported command?

  if [ "${url:0:5}" != "s3://" ]; then
    echo "Need an s3 url"
    return 1
  fi
  local path="${url:4}"

  if [ -z "${AWS_ACCESS_KEY_ID-}"  ]; then
    echo "Need AWS_ACCESS_KEY_ID to be set"
    return 1
  fi

  if [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
    echo "Need AWS_SECRET_ACCESS_KEY to be set"
    return 1
  fi

  local method md5 args
  case "$command" in
  get)
    method="GET"
    md5=""
    args="-o $file"
    ;;
  put)
    method="PUT"
    if [ ! -f "$file" ]; then
      echo "file not found"
      exit 1
    fi
    md5="$(openssl md5 -binary $file | openssl base64)"
    args="-T $file -H Content-MD5:$md5"
    ;;
  *)
    echo "Unsupported command"
    return 1
  esac

  local date="$(date -u '+%a, %e %b %Y %H:%M:%S +0000')"
  local string_to_sign
  printf -v string_to_sign "%s\n%s\n\n%s\n%s" "$method" "$md5" "$date" "$path"
  local signature=$(echo -n "$string_to_sign" | openssl sha1 -binary -hmac "${AWS_SECRET_ACCESS_KEY}" | openssl base64)
  local authorization="AWS ${AWS_ACCESS_KEY_ID}:${signature}"

  curl $args -s -f -H Date:"${date}" -H Authorization:"${authorization}" https://s3.amazonaws.com"${path}"
}

s3simple "$@"

您可以找到有关 s3simple 脚本here 的更多信息。

Answer 2

我创建了一个完整的 AWS 签名创建器。你可以找到原始文件here（请注意：它有多个 gist cmets，所以它绝对有效！）。

readonly AWS_ACCESS_KEY_ID='<your_access_key_id>'
readonly AWS_SECRET_ACCESS_KEY='<your_secret_access_key>'
readonly AWS_SERVICE='s3'
readonly AWS_REGION='us-east-1'
readonly AWS_S3_BUCKET_NAME='<your_bucket_name>'
readonly AWS_SERVICE_ENDPOINT_URL="\
${AWS_S3_BUCKET_NAME}.${AWS_SERVICE}.amazonaws.com"

# Create an SHA-256 hash in hexadecimal.
# Usage:
#   hash_sha256 <string>
function hash_sha256 {
  printf "${1}" | openssl dgst -sha256 | sed 's/^.* //'
}

# Create an SHA-256 hmac in hexadecimal format.
# Usage:
#   hmac_sha256 <key> <data>
function hmac_sha256 {
  key="$1"
  data="$2"
  printf "${data}" | openssl dgst -sha256 -mac HMAC -macopt "${key}" | \
      sed 's/^.* //'
}

readonly CURRENT_DATE_DAY="$(date -u '+%Y%m%d')"
readonly CURRENT_DATE_TIME="$(date -u '+%H%M%S')"
readonly CURRENT_DATE_ISO8601="${CURRENT_DATE_DAY}T${CURRENT_DATE_TIME}Z"

readonly HTTP_REQUEST_METHOD='GET'
readonly HTTP_REQUEST_PAYLOAD=''
readonly HTTP_REQUEST_PAYLOAD_HASH="$(printf "${HTTP_REQUEST_PAYLOAD}" | \
    openssl dgst -sha256 | sed 's/^.* //')"
readonly HTTP_CANONICAL_REQUEST_URI='/video_clips/0940.m3u8'
readonly HTTP_CANONICAL_REQUEST_QUERY_STRING=''
readonly HTTP_REQUEST_CONTENT_TYPE='application/x-www-form-urlencoded'

readonly HTTP_CANONICAL_REQUEST_HEADERS="\
content-type:${HTTP_REQUEST_CONTENT_TYPE}
host:${AWS_SERVICE_ENDPOINT_URL}
x-amz-content-sha256:${HTTP_REQUEST_PAYLOAD_HASH}
x-amz-date:${CURRENT_DATE_ISO8601}"
# Note: The signed headers must match the canonical request headers.
readonly HTTP_REQUEST_SIGNED_HEADERS="\
content-type;host;x-amz-content-sha256;x-amz-date"

readonly HTTP_CANONICAL_REQUEST="\
${HTTP_REQUEST_METHOD}
${HTTP_CANONICAL_REQUEST_URI}
${HTTP_CANONICAL_REQUEST_QUERY_STRING}
${HTTP_CANONICAL_REQUEST_HEADERS}\n
${HTTP_REQUEST_SIGNED_HEADERS}
${HTTP_REQUEST_PAYLOAD_HASH}"

# Create the signature.
# Usage:
#   create_signature
function create_signature {
  stringToSign="AWS4-HMAC-SHA256
${CURRENT_DATE_ISO8601}
${CURRENT_DATE_DAY}/${AWS_REGION}/${AWS_SERVICE}/aws4_request
$(hash_sha256 "${HTTP_CANONICAL_REQUEST}")"

  dateKey=$(hmac_sha256 key:"AWS4${AWS_SECRET_ACCESS_KEY}" \
      "${CURRENT_DATE_DAY}")
  regionKey=$(hmac_sha256 hexkey:"${dateKey}" "${AWS_REGION}")
  serviceKey=$(hmac_sha256 hexkey:"${regionKey}" "${AWS_SERVICE}")
  signingKey=$(hmac_sha256 hexkey:"${serviceKey}" "aws4_request")

  printf "${stringToSign}" | openssl dgst -sha256 -mac HMAC -macopt \
      hexkey:"${signingKey}" | awk '{print $2}'
}

readonly SIGNATURE="$(create_signature)"

readonly HTTP_REQUEST_AUTHORIZATION_HEADER="\
AWS4-HMAC-SHA256 Credential=${AWS_ACCESS_KEY_ID}/${CURRENT_DATE_DAY}/\
${AWS_REGION}/${AWS_SERVICE}/aws4_request, \
SignedHeaders=${HTTP_REQUEST_SIGNED_HEADERS};x-amz-date, Signature=${SIGNATURE}"

curl -X "${HTTP_REQUEST_METHOD}" -v \
    "https://${AWS_SERVICE_ENDPOINT_URL}${HTTP_CANONICAL_REQUEST_URI}" \
    -H "Authorization: ${HTTP_REQUEST_AUTHORIZATION_HEADER}" \
    -H "content-type: ${HTTP_REQUEST_CONTENT_TYPE}" \
    -H "x-amz-content-sha256: ${HTTP_REQUEST_PAYLOAD_HASH}" \
    -H "x-amz-date: ${CURRENT_DATE_ISO8601}"

但请注意，如果您没有理由创建签名，最好使用 AWS API。

Answer 3

截至 2019 年 8 月，我发现这可行。已添加区域，URL 格式已更改。

#!/bin/sh 
outputFile="/PATH/TO/LOCALLY/SAVED/FILE"
amzFile="BUCKETPATH/TO/FILE"
region="YOUR-REGION"
bucket="SOME-BUCKET"
resource="/${bucket}/${amzFile}"
contentType="binary/octet-stream"
dateValue=`TZ=GMT date -R`
# You can leave our "TZ=GMT" if your system is already GMT (but don't have to)
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: s3-${region}.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://s3-${region}.amazonaws.com/${bucket}/${amzFile} -o $outputFile

Answer 4

避免自己签署请求，很多事情都可能出错或很难做到。例如，您应该检查日期是否设置为 GMT 或使用 x-amz-date 标头。

另一种方法是使用AWS Command Line Interface，因此使用$ aws s3 cp 或$ aws s3 sync。

Answer 5

#!/bin/sh
# This works for cross region
outputFile="/PATH/TO/FILE"
awsFile="BUCKETPATH/TO/FILE"
bucket="SOME-BUCKET"
resource="/${bucket}/${awsFile}"
contentType="application/x-compressed-tar"
# Change the content type as desired
dateValue=`TZ=GMT date -R`
#Use dateValue=`date -R` if your TZ is already GMT
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="ACCESS_KEY_ID"
s3Secret="SECRET_ACCESS_KEY"
signature=`echo -n ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://${bucket}.s3.amazonaws.com/${awsFile} -o $outputFile

Answer 6

我写这个bash脚本从s3下载文件（我下载压缩文件，你可以改变contentType来下载其他类型的文件）

#!/bin/sh 
outputFile="Your_PATH"
amzFile="AMAZON_FILE_PATH"
bucket="YOUR_BUCKET"
resource="/${bucket}/${amzFile}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="YOUR_S3_KEY"
s3Secret="YOUR_S3SECRET"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`

curl  -H "Host: ${bucket}.s3.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3Key}:${signature}" \
     https://${bucket}.s3.amazonaws.com/${amzFile} -o $outputFile