通过 Python (Jupyter) 从 Google Cloud 存储中读取 .gz 文件答案

【问题标题】：Read a .gz file from Google Cloud storage via Python (Jupyter)通过 Python (Jupyter) 从 Google Cloud 存储中读取 .gz 文件
【发布时间】：2020-10-01 09:30:58
【问题描述】：

我正在尝试通过 Jupyter 笔记本上的 Python 从 Google Cloud 存储中读取 .gz 文件。

我收到第一个代码的错误。

TypeError: can't concat str to bytes

from google.cloud import storage
import pandas as pd
from io import StringIO

client = storage.Client()
bucket = client.get_bucket("nttcomware")
blob = bucket.get_blob(f"test.csv.gz")
df = pd.read_csv(s, compression='gzip', float_precision="high")
df.head()

第二个代码出现第二个错误。

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

from google.cloud import storage
import pandas as pd
from io import StringIO

client = storage.Client()
bucket = client.get_bucket("nttcomware")
blob = bucket.get_blob(f"test.csv.gz")
bt = blob.download_as_string()
s = str(bt, "utf-8")
s = StringIO(s)
df = pd.read_csv(s, compression='gzip', float_precision="high")
df.head()

请提出建议。

【问题讨论】：

问题出在gz文件的解码中，尝试将s = str(bt, "utf-8")替换为s = str(unicode(bt, errors='replace'))，如果可行，请告诉我。
感谢您的回答。但它没有工作..

标签： python pandas google-cloud-storage

【解决方案1】：

幸运的是我自己解决了。我希望它对其他人有所帮助。

client = storage.Client()

# get the bucket
bucket = client.get_bucket("nttcomware")

# get the blob object
blob_name = "test.csv.gz"
blob = bucket.get_blob(blob_name)

# convert blob into string and consider as BytesIO object. Still compressed by gzip
data = io.BytesIO(blob.download_as_string())

# open gzip into csv
with gzip.open(data) as gz:
    #still byte type string
    file = gz.read()
    # erase the .gz extension and get the blob object
    blob_decompress = bucket.blob(blob_name.replace('.gz',''))
    # convert into byte type again
    blob_decompress = blob_decompress.download_as_string()
    # decode the byte type into string by utf-8
    blob_decompress = blob_decompress.decode('utf-8')
    # StringIO object
    s = StringIO(blob_decompress)
    

df = pd.read_csv(s, float_precision="high")
df.head()

【讨论】：

您好，您已经定义了“文件”并读取了 gzip 的内容，之后您就不再使用了。请您详细说明一下。我试过这个方法，但它给我一个错误。

【解决方案2】：

这适用于我直接从 GCS 读取 json.gz 到数据帧。

client = storage.Client()

def gcs_read_json_gz(gcs_filepath, nrows=None):
    
    # Validate input path
    if not gcs_filepath.startswith("gs://") or not gcs_filepath.endswith(".json.gz"):
        raise ValueError(F"Invalid path: {gcs_filepath}")

    # Get the bucket
    bucket_name = gcs_filepath.split("/")[2]
    bucket = client.get_bucket(bucket_name)

    # Get the blob object
    blob_name = "/".join(gcs_filepath.split("/")[3:])
    blob = bucket.get_blob(blob_name)

    # Convert blob into string and consider as BytesIO object. Still compressed by gzip
    data = io.BytesIO(blob.download_as_string())

    # Open gzip into csv
    with gzip.open(data) as gz:
        # Read compressed file as a file object
        file = gz.read()
        # Decode the byte type into string by utf-8
        blob_decompress = file.decode('utf-8')
        # StringIO object
        s = io.StringIO(blob_decompress)

    df = pd.read_json(s, precise_float="high", nrows=nrows, lines=True)

    return df

【讨论】：