BigQuery 结果到 Panda DataFrame 中的块

Question

我正在尝试使用 bigquery.Client.query.to_dataframe() 将 BigQuery 查询的结果保存到 Panda DataFrame

此查询可以返回数百万行。

鉴于 Panda 到 BQ (Dataframe.to_gbq()) 有一个块参数，BQ 到 Pandas 是否有类似的东西可以增量添加到数据帧，而不必多次运行查询并使用限制和偏移量？

Answer 1

@William 提到过，您可以对 BigQuery 结果进行分块并对它们进行分页，查询只会收取一次执行费用。我根据官方文档使用公共数据集完成了这段代码：'bigquery-public-data.baseball.games_wide' 作为演示：

import pandas as pd
import math

bq_client = bigquery.Client()

class BqToDfChunker(object):


    def __init__(self, query_job, results_per_page):
        bq_result = query_job.result()
        destination = query_job.destination
        destination =  bq_client.get_table(destination)

        self.destination = destination
        self.results_per_page = results_per_page
        self.num_pages = math.ceil(float(destination.num_rows/results_per_page))
        self.index = 0
        self.next_token = None


    def get_next_df_page(self):
        rows = bq_client.list_rows(self.destination,
           max_results = self.results_per_page,
           page_token = self.next_token)

        if self.index < self.num_pages:

            df = pd.DataFrame(rows)
            self.index += 1
            self.next_token = rows.next_page_token 

            return df

        else:
            return None


    def has_next(self):
        if self.index != self.num_pages:
            return True
        else:
            return False

if __name__ == '__main__':

    query = """
        SELECT homeTeamName FROM `bigquery-public-data.baseball.games_wide` group by homeTeamName
    """

    query_job = bq_client.query(query) 

    #initialize the class with the query_job and number_of_results_per_page
    bq_test = BqToDfChunker(query_job, 10)

    while bq_test.has_next():
        print(bq_test.get_next_df_page())