Python 单元测试 - 'gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket。'答案

【问题标题】：Python Unit Testing - 'gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.'Python 单元测试 - 'gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket。'
【发布时间】：2023-03-16 04:57:01
【问题描述】：

我正在这个脚本 api.py 上运行 Python 单元测试

def download_models(gcs_model_path, local_path):
    filesystem = gcsfs.GCSFileSystem(project=PROJECT)  # , token='creds.json')
    # download models from gcs
    filesystem.get(gcs_model_path, local_path, recursive=True)

这里是单元测试脚本embedding_api_test.py

class EmbeddingAPITest(unittest.TestCase):

    @mock.patch('gcsfs.GCSFileSystem')
    def test_download_models(self, mock_filesystem):
        mock_filesystem.return_value.get.return_value = []
        download_models('gcs_model_path', 'local_path')

我的终端出现此错误：

_______________________ ERROR collecting qa_api/tests/qa_api_test.py ________________________
qa_api/tests/qa_api_test.py:3: in <module>
    from qa_api.api import app
qa_api/api.py:37: in <module>
    download_models(GCS_MODEL_PATH, LOCAL_MODEL_PATH)
qa_api/api.py:21: in download_models
    filesystem.get(gcs_model_path, local_path, recursive=True)
/opt/anaconda3/lib/python3.7/site-packages/fsspec/spec.py:556: in get
    rpaths = self.find(rpath)
/opt/anaconda3/lib/python3.7/site-packages/fsspec/spec.py:371: in find
    for path, dirs, files in self.walk(path, maxdepth, **kwargs):
/opt/anaconda3/lib/python3.7/site-packages/fsspec/spec.py:326: in walk
    listing = self.ls(path, detail=True, **kwargs)
/opt/anaconda3/lib/python3.7/site-packages/gcsfs/core.py:767: in ls
    out = self._list_objects(path)
/opt/anaconda3/lib/python3.7/site-packages/gcsfs/core.py:571: in _list_objects
    items, prefixes = self._do_list_objects(path)
/opt/anaconda3/lib/python3.7/site-packages/gcsfs/core.py:604: in _do_list_objects
    maxResults=max_results,
/opt/anaconda3/lib/python3.7/site-packages/gcsfs/core.py:504: in _call
    raise e
/opt/anaconda3/lib/python3.7/site-packages/gcsfs/core.py:487: in _call
    validate_response(r, path)
/opt/anaconda3/lib/python3.7/site-packages/gcsfs/core.py:130: in validate_response
    raise HttpError(error)
E   gcsfs.utils.HttpError: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.

似乎我在嘲笑它不正确，但我不明白为什么。非常感谢任何帮助。如果需要更多详细信息，请告诉我。谢谢！

【问题讨论】：

标签： python python-3.x unit-testing google-cloud-storage python-unittest

【解决方案1】：

你应该修补'api.gcsfs.GCSFileSystem'。例如

api.py:

import gcsfs

PROJECT = "teresa.teng"


def download_models(gcs_model_path, local_path):
    filesystem = gcsfs.GCSFileSystem(project=PROJECT)
    filesystem.get(gcs_model_path, local_path, recursive=True)

test_api.py:

import unittest
from unittest import mock
from api import download_models


class EmbeddingAPITest(unittest.TestCase):

    @mock.patch('api.gcsfs.GCSFileSystem')
    def test_download_models(self, mock_filesystem):
        mock_filesystem.return_value.get.return_value = []
        download_models('gcs_model_path', 'local_path')
        mock_filesystem.assert_called_once_with(project="teresa.teng")
        mock_filesystem.return_value.get.assert_called_once_with('gcs_model_path', 'local_path', recursive=True)


if __name__ == '__main__':
    unittest.main()

单元测试结果：

.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
Name                                     Stmts   Miss  Cover   Missing
----------------------------------------------------------------------
src/stackoverflow/64673419/api.py            5      0   100%
src/stackoverflow/64673419/test_api.py      11      0   100%
----------------------------------------------------------------------
TOTAL                                       16      0   100%

requirements.txt:

gcsfs==0.7.1

【讨论】：