GCP - 无法访问 main.py 中 GCP 存储桶中保存的模型文件答案

【问题标题】：GCP - Unable to access saved model file on GCP bucket in main.pyGCP - 无法访问 main.py 中 GCP 存储桶中保存的模型文件
【发布时间】：2021-02-06 05:14:20
【问题描述】：

我已将 PyTorch 检查点文件“checkpoint_ic_d161.pth”上传到 GCP 存储桶。

我正在尝试将 PyTorch Flask 模型上传到 GCP App Engine 以制作简单的网络应用。

但我无法将模型文件从 GCP 存储桶访问到 App Engine 中的 main.py。

MODEL_URL = 'https://storage.googleapis.com/end_to_end_challenge_bucket/checkpoint_ic_d161.pth'

def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)
    model = checkpoint['model']
    model.classifier = checkpoint['classifier']
    model.load_state_dict(checkpoint['state_dict'])
    model.class_to_idx = checkpoint['class_to_idx']
    optimizer = checkpoint['optimizer']
    epochs = checkpoint['epochs']
    
    for param in model.parameters():
        param.requires_grad = False
        
    return model, checkpoint['class_to_idx']


def get_model():
    model, class_to_idx = load_checkpoint(MODEL_URL)
    model.eval()
    return model

我收到 https://storage.googleapis.com/end_to_end_challenge_bucket/checkpoint_ic_d161.pth 的 FileNotFound 错误，尽管这是对存储桶中文件的公开访问。

为什么会这样，我如何访问存储在 GCP 存储桶中的模型/检查点文件，位于 GCP App Engine 的 main.py 中？

【问题讨论】：

标签： python-3.x google-cloud-platform

【解决方案1】：

要下载并上传到 Cloud Storage 文件，您需要使用 Cloud Storage 库。原来是not supported by PyTorch.

pip install --upgrade google-cloud-storage

Upload

from google.cloud import storage

def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # bucket_name = "your-bucket-name"
    # source_file_name = "local/path/to/file"
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_filename(source_file_name)

    print(
        "File {} uploaded to {}.".format(
            source_file_name, destination_blob_name
        )
    )

Download

from google.cloud import storage

def download_blob(bucket_name, source_blob_name, destination_file_name):
    """Downloads a blob from the bucket."""
    # bucket_name = "your-bucket-name"
    # source_blob_name = "storage-object-name"
    # destination_file_name = "local/path/to/file"

    storage_client = storage.Client()

    bucket = storage_client.bucket(bucket_name)

    # Construct a client side representation of a blob.
    # Note `Bucket.blob` differs from `Bucket.get_blob` as it doesn't retrieve
    # any content from Google Cloud Storage. As we don't need additional data,
    # using `Bucket.blob` is preferred here.
    blob = bucket.blob(source_blob_name)
    blob.download_to_filename(destination_file_name)

    print(
        "Blob {} downloaded to {}.".format(
            source_blob_name, destination_file_name
        )
    )

【讨论】：