Azure ML：将文件上传到 Step Run 的输出 - 身份验证错误

【问题标题】：Azure ML: Upload File to Step Run's Output - Authentication ErrorAzure ML：将文件上传到 Step Run 的输出 - 身份验证错误
【发布时间】：2021-10-27 11:58:33
【问题描述】：

在 Azure ML 管道中的 PythonScriptStep 期间，我将模型作为 joblib pickle 转储保存到 Azure Blob 存储中的 Blob 容器中的目录，该目录是我在设置 Azure ML 工作区期间创建的。之后我尝试使用

将此模型文件上传到步骤运行的输出目录 
Run.upload_file (name, path_or_stream)

（有关函数的文档，请参阅https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.run(class)?view=azure-ml-py#upload-file-name--path-or-stream--datastore-name-none-）

前段时间，当我使用 azureml-sdk 版本 1.18.0 创建脚本时，一切正常。现在，我已经更新了脚本的功能，并在此过程中将 azureml-sdk 升级到了 1.33.0 版本，上传功能现在遇到以下错误：

Traceback (most recent call last):
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_file_utils/upload.py", line 64, in upload_blob_from_stream
    validate_content=True)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 93, in execute_func_with_reset
    return ClientBase._execute_func_internal(backoff, retries, module_logger, func, reset_func, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 367, in _execute_func_internal
    left_retry = cls._handle_retry(back_off, left_retry, total_retry, error, logger, func)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 399, in _handle_retry
    raise error
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/clientbase.py", line 358, in _execute_func_internal
    response = func(*args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/blockblobservice.py", line 614, in create_blob_from_stream
    initialization_vector=iv
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 98, in _upload_blob_chunks
    range_ids = [f.result() for f in futures]
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 98, in <listcomp>
    range_ids = [f.result() for f in futures]
  File "/opt/miniconda/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/opt/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/opt/miniconda/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 210, in process_chunk
    return self._upload_chunk_with_progress(chunk_offset, chunk_bytes)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 224, in _upload_chunk_with_progress
    range_id = self._upload_chunk(chunk_offset, chunk_data)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/_upload_chunking.py", line 269, in _upload_chunk
    timeout=self.timeout,
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/blob/blockblobservice.py", line 1013, in _put_block
    self._perform_request(request)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/storageclient.py", line 432, in _perform_request
    raise ex
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/storageclient.py", line 357, in _perform_request
    raise ex
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/storageclient.py", line 343, in _perform_request
    HTTPError(response.status, response.message, response.headers, response.body))
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_vendor/azure_storage/common/_error.py", line 115, in _http_error_handler
    raise ex
azure.common.AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5d4e1b7e-c01e-0070-0d47-9bf8a0000000
Time:2021-08-27T13:30:02.2685991Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcw
2021-08-27T13:19:56Z
2021-08-28T13:29:56Z
/blob/mystorage/azureml/ExperimentRun/dcid.98d11a7b-2aac-4bc0-bd64-bb4d72e0e0be/outputs/models/Model.pkl

2019-07-07
b

</AuthenticationErrorDetail></Error>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/batch/tasks/shared/LS_root/jobs/.../azureml-setup/context_manager_injector.py", line 243, in execute_with_context
    runpy.run_path(sys.argv[0], globals(), run_name="__main__")
  File "/opt/miniconda/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/opt/miniconda/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/opt/miniconda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 318, in <module>
    main()
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 286, in main
    path_or_stream=model_path)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 53, in wrapped
    return func(self, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 1989, in upload_file
    datastore_name=datastore_name)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 114, in upload_artifact
    return self.upload_artifact_from_path(artifact, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 107, in upload_artifact_from_path
    return self.upload_artifact_from_stream(stream, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 99, in upload_artifact_from_stream
    content_type=content_type, session=session)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 88, in upload_stream_to_existing_artifact
    timeout=TIMEOUT, backoff=BACKOFF_START, retries=RETRY_LIMIT)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_file_utils/upload.py", line 71, in upload_blob_from_stream
    raise AzureMLException._with_error(azureml_error, inner_exception=e)
azureml._common.exceptions.AzureMLException: AzureMLException:
    Message: Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.
    StorageAccount: mystorage
    ContainerName: azureml
    StatusCode: 403
    InnerException Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5d4e1b7e-c01e-0070-0d47-9bf8a0000000
Time:2021-08-27T13:30:02.2685991Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcw
2021-08-27T13:19:56Z
2021-08-28T13:29:56Z
/blob/mystorage/azureml/ExperimentRun/dcid.98d11a7b-2aac-4bc0-bd64-bb4d72e0e0be/outputs/models/Model.pkl

2019-07-07
b

</AuthenticationErrorDetail></Error>
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.\n\tStorageAccount: mystorage\n\tContainerName: azureml\n\tStatusCode: 403",
        "inner_error": {
            "code": "Auth",
            "inner_error": {
                "code": "Authorization"
            }
        }
    }
}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 318, in <module>
    main()
  File "401_AML_Pipeline_Time_Series_Model_Training_Azure_ML_CPU.py", line 286, in main
    path_or_stream=model_path)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 53, in wrapped
    return func(self, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/core/run.py", line 1989, in upload_file
    datastore_name=datastore_name)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 114, in upload_artifact
    return self.upload_artifact_from_path(artifact, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 107, in upload_artifact_from_path
    return self.upload_artifact_from_stream(stream, *args, **kwargs)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 99, in upload_artifact_from_stream
    content_type=content_type, session=session)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_restclient/artifacts_client.py", line 88, in upload_stream_to_existing_artifact
    timeout=TIMEOUT, backoff=BACKOFF_START, retries=RETRY_LIMIT)
  File "/opt/miniconda/lib/python3.7/site-packages/azureml/_file_utils/upload.py", line 71, in upload_blob_from_stream
    raise AzureMLException._with_error(azureml_error, inner_exception=e)
UserScriptException: UserScriptException:
    Message: Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.
    StorageAccount: mystorage
    ContainerName: azureml
    StatusCode: 403
    InnerException AzureMLException:
    Message: Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.
    StorageAccount: mystorage
    ContainerName: azureml
    StatusCode: 403
    InnerException Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5d4e1b7e-c01e-0070-0d47-9bf8a0000000
Time:2021-08-27T13:30:02.2685991Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcw
2021-08-27T13:19:56Z
2021-08-28T13:29:56Z
/blob/mystorage/azureml/ExperimentRun/dcid.98d11a7b-2aac-4bc0-bd64-bb4d72e0e0be/outputs/models/Model.pkl

2019-07-07
b

</AuthenticationErrorDetail></Error>
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.\n\tStorageAccount: verovisionstorage\n\tContainerName: azureml\n\tStatusCode: 403",
        "inner_error": {
            "code": "Auth",
            "inner_error": {
                "code": "Authorization"
            }
        }
    }
}
    ErrorResponse 
{
    "error": {
        "code": "UserError",
        "message": "Encountered authorization error while uploading to blob storage. Please check the storage account attached to your workspace. Make sure that the current user is authorized to access the storage account and that the request is not blocked by a firewall, virtual network, or other security setting.\n\tStorageAccount: mystorage\n\tContainerName: azureml\n\tStatusCode: 403"
    }
}

据我所知，从 azureml.core.Run 类的代码和随后的函数调用中，Run 对象尝试使用 SAS-Token-Authentication 将文件上传到步骤运行的输出目录（失败） .此文档文章在代码中链接（但我不知道这是否与问题有关）：https://docs.microsoft.com/en-us/rest/api/storageservices/create-service-sas#service-sas-example

有没有人也遇到过这个错误并且知道是什么原因造成的或者如何解决？

最好， 乔纳斯

【问题讨论】：

  • 您能分享一下您是如何创建Run 对象的吗？

标签： python file-upload azure-blob-storage azure-machine-learning-service sas-token


【解决方案1】：

我们以前见过，这很烦人。我认为答案是转到 AML Studio UI 的数据存储页面并再次手动输入存储帐户密钥。

【讨论】：

  • 感谢您的快速回复！可悲的是，这种方法并没有解决问题：-/您还有其他想法吗？ Run-Object 是在 AML 管道 PythonScriptStep 中通过 ` from azureml.core import Run RUN_OBJECT = Run.get_context() ` 创建的
  • 在进一步的思考中，我发现我可以将 Azure ML 中数据存储的身份验证类型设置为“帐户密钥”（默认设置）或“SAS 令牌”。鉴于 SAS 令牌身份验证失败，我考虑将设置更改为“SAS 令牌”。但是，因为这将是我（或者更确切地说是 azureml-sdk）使用 SAS 令牌的唯一情况，因此我猜这可能会破坏脚本中其他位置的身份验证。你怎么看？
  • 我很同情你的经历！绝对不理想。我会先尝试这两种选择，看看你是否可以做任何事情。这将帮助我们了解正在发生的事情。之后，我们可以决定最适合您的方案。 “帐户密钥”对您有用吗？
  • 另外，如果您或您的组织有 Azure 优先支持，我强烈建议为此开立支持票。我也会让一些 Azure ML PM 知道你的问题，因为我也很困惑。
  • 您能否使用问号旁边的笑脸图标从 Azure 机器学习门户 UI 中的数据存储页面提供反馈。此外，请务必在反馈中包含您的电子邮件，以便我们可以跟进反馈以获取更多信息。谢谢。
猜你喜欢
  • 2021-01-07
  • 2021-01-08
  • 1970-01-01
  • 1970-01-01
  • 2015-04-10
  • 2017-11-26
  • 2014-05-04
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode