下载 Azure 存储容器中的所有 Blob

Question

我已经设法编写了一个 python 脚本来列出容器中的所有 blob。

import azure
from azure.storage.blob import BlobService
from azure.storage import *

blob_service = BlobService(account_name='<CONTAINER>', account_key='<ACCOUNT_KEY>')


blobs = []
marker = None
while True:
    batch = blob_service.list_blobs('<CONAINER>', marker=marker)
    blobs.extend(batch)
    if not batch.next_marker:
        break
    marker = batch.next_marker
for blob in blobs:
    print(blob.name)

就像我说的，这仅列出了我要下载的 blob。我已经转移到 Azure CLI，看看它是否有助于我想做的事情。我可以下载单个 blob

azure storage blob download [container]

然后它会提示我指定一个可以从 python 脚本中获取的 blob。我必须下载所有这些 blob 的方法是在上面使用的命令之后将它们复制并粘贴到提示符中。有没有办法：

一个。编写一个 bash 脚本，通过执行命令遍历 blob 列表，然后在提示符中粘贴下一个 blob 名称。

B。指定在 python 脚本或 Azure CLI 中下载容器。有没有我没有看到下载整个容器的东西？

Answer 1

目前，我们似乎无法使用单个 API 从容器中直接下载所有 blob。我们可以通过 https://msdn.microsoft.com/en-us/library/azure/dd179377.aspx 获取所有可用的 blob 操作。

所以我们可以列出 blob 的 ListGenerator，然后循环下载 blob。例如：

result = blob_service.list_blobs(container)
for b in result.items:
    r = blob_service.get_blob_to_path(container,b.name,"folder/{}".format(b.name))

更新

使用azure-storage-python时导入blockblob服务：

from azure.storage.blob import BlockBlobService

Answer 2

@gary-liu-msft 解决方案是正确的。我对它进行了更多更改，现在代码可以遍历容器和其中的文件夹结构（PS - 容器中没有文件夹，只有路径），检查客户端中是否存在相同的目录结构，如果不存在则创建该目录结构并下载这些路径中的 blob。它支持带有嵌入子目录的长路径。

from azure.storage.blob import BlockBlobService
from azure.storage.blob import PublicAccess
import os

#name of your storage account and the access key from Settings->AccessKeys->key1
block_blob_service = BlockBlobService(account_name='storageaccountname', account_key='accountkey')

#name of the container
generator = block_blob_service.list_blobs('testcontainer')

#code below lists all the blobs in the container and downloads them one after another
for blob in generator:
    print(blob.name)
    print("{}".format(blob.name))
    #check if the path contains a folder structure, create the folder structure
    if "/" in "{}".format(blob.name):
        print("there is a path in this")
        #extract the folder path and check if that folder exists locally, and if not create it
        head, tail = os.path.split("{}".format(blob.name))
        print(head)
        print(tail)
        if (os.path.isdir(os.getcwd()+ "/" + head)):
            #download the files to this directory
            print("directory and sub directories exist")
            block_blob_service.get_blob_to_path('testcontainer',blob.name,os.getcwd()+ "/" + head + "/" + tail)
        else:
            #create the diretcory and download the file to it
            print("directory doesn't exist, creating it now")
            os.makedirs(os.getcwd()+ "/" + head, exist_ok=True)
            print("directory created, download initiated")
            block_blob_service.get_blob_to_path('testcontainer',blob.name,os.getcwd()+ "/" + head + "/" + tail)
    else:
        block_blob_service.get_blob_to_path('testcontainer',blob.name,blob.name)

同样的代码也可以在这里https://gist.github.com/brijrajsingh/35cd591c2ca90916b27742d52a3cf6ba

Answer 3

自从@brij-raj-singh-msft 回答后，Microsoft 发布了适用于 Python 的第二代 Azure Storage Blobs 客户端库。 （下面的代码使用版本 12.5.0 测试） 此 sn-p 于 2020 年 9 月 25 日测试

import os
from azure.storage.blob import BlobServiceClient,ContainerClient, BlobClient
import datetime

# Assuming your Azure connection string environment variable set.
# If not, create BlobServiceClient using trl & credentials.
#Example: https://docs.microsoft.com/en-us/python/api/azure-storage-blob/azure.storage.blob.blobserviceclient 

connection_string = os.getenv("AZURE_STORAGE_CONNECTION_STRING")

blob_service_client = BlobServiceClient.from_connection_string(conn_str=connection_string) 
# create container client
container_name = 'test2'
container_client = blob_service_client.get_container_client(container_name)

#Check if there is a top level local folder exist for container.
#If not, create one
data_dir ='Z:/azure_storage'
data_dir = data_dir+ "/" + container_name
if not(os.path.isdir(data_dir)):
    print("[{}]:[INFO] : Creating local directory for container".format(datetime.datetime.utcnow()))
    os.makedirs(data_dir, exist_ok=True)
    
#code below lists all the blobs in the container and downloads them one after another
blob_list = container_client.list_blobs()
for blob in blob_list:
    print("[{}]:[INFO] : Blob name: {}".format(datetime.datetime.utcnow(), blob.name))
    #check if the path contains a folder structure, create the folder structure
    if "/" in "{}".format(blob.name):
        #extract the folder path and check if that folder exists locally, and if not create it
        head, tail = os.path.split("{}".format(blob.name))
        if not (os.path.isdir(data_dir+ "/" + head)):
            #create the diretcory and download the file to it
            print("[{}]:[INFO] : {} directory doesn't exist, creating it now".format(datetime.datetime.utcnow(),data_dir+ "/" + head))
            os.makedirs(data_dir+ "/" + head, exist_ok=True)
    # Finally, download the blob
    blob_client = container_client.get_blob_client(blob.name)
    dowlload_blob(blob_client,data_dir+ "/"+blob.name)

def dowlload_blob(blob_client, destination_file):
    print("[{}]:[INFO] : Downloading {} ...".format(datetime.datetime.utcnow(),destination_file))
    with open(destination_file, "wb") as my_blob:
        blob_data = blob_client.download_blob()
        blob_data.readinto(my_blob)
    print("[{}]:[INFO] : download finished".format(datetime.datetime.utcnow()))    

同样的代码也可以在这里https://gist.github.com/allene/6bbb36ec3ed08b419206156567290b13

Answer 4

我为 Azure CLI 创建了一个Python wrapper，它使我们能够批量下载/上传。这样我们就可以下载一个完整的容器或从容器中下载某些文件。

安装：

pip install azurebatchload

from azurebatchload import Download

Download(
   destination='../pdfs',
   source='blobcontainername',
   extension='.pdf'
).download()

Answer 5

这是一个简单的脚本 (PowerShell)，它将遍历单个容器并将其中的所有内容下载到您提供的 $Destination

# 
# Download All Blobs in a Container
# 

# Connect to Azure Account
Connect-AzAccount

# Set Variables
$Destination="C:\Software"
$ResourceGroupName = 'resource group name'
$ContainerName = 'container name'
$storageAccName = 'storage account name'

# Function to download all blob contents
Function DownloadBlobContents
{
    Write-Host -ForegroundColor Green "Download blob contents from storage container.."
    # Get the storage account  
    $StorageAcc = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccName
    # Get the storage account context
    $Ctx = $StorageAcc.Context
    # Get all containers
    $Containers = Get-AzStorageContainer -Context $Ctx
    Write-Host -ForegroundColor Magenta $Container.Name "Creating or checking folder presence"
    # Create or check folder presence
    New-Item -ItemType Directory -Path $Destination -Force
    # Get the blob contents from the container
    $BlobConents = Get-AzStorageBlob -Container $ContainerName -Context $Ctx
    # Loop through each blob and download each one until they are all complete
    foreach($BlobConent in $BlobConents)
    {
        # Download the blob content
        Get-AzStorageBlobContent -Container $ContainerName -Context $Ctx -Blob $BlobConent.Name -Destination $Destination -Force
        Write-Host -ForegroundColor Green "Downloaded a blob content"
    }
}

DownloadBlobContents

我想学习的是不必使用 Connect-AzAccount，而是调用存储帐户的密钥，因此我可以运行它而无需进行身份验证