在循环结束时合并数据框以进行网页抓取答案

【问题标题】：Combine Dataframes at end of a loop for Web Page scrape在循环结束时合并数据框以进行网页抓取
【发布时间】：2020-08-06 22:14:38
【问题描述】：

我已经开始编写一个程序来从多个 URL 中抓取一个数据表。我已经到了通过导入 Excel 电子表格创建 url 列表、遍历 url 列表并通过搜索表头来抓取网页上的特定表的地步。

循环末尾的 print 语句分别为每个 url 打印出单独的表格。有没有一种简单的方法可以将 DataFrame 行 append() 在一起，类似于简单的列表生成？表格的布局相同。

from urllib.request import urlopen
from bs4 import BeautifulSoup, NavigableString, Tag
import requests
import pandas as pd
import re
import ssl
import lxml
import xlrd
import csv

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

#auto import excel file
ex = pd.read_excel(r'/Users/adamsewell/Desktop/GB_Basketball/Data/GB_Player_Tracking_Document.xlsm', sheet_name='Player URL')
yr = '2019-20'

#list urls from excel sheet
url_list = ex['URL'].tolist()

for url in url_list:
    #first header as a reference point
    table_title = 'International Regular Season Stats - Per Game'
    #replace to gain second header title to end loop
    second_header = (table_title.replace(' Per Game',' Totals'))

    html = urlopen(url, context=ctx).read()
    soup = BeautifulSoup(html, "html.parser")

    #find the table in the whole HTML
    start = soup.find('h2', text=table_title)
    end = soup.find('h2', text=second_header)
    content = '' #prime content as nothing
    item = start.nextSibling

    #while not at the end header, add content to the item
    while item != end:
        content += str(item)
        item = item.nextSibling

    #create a list and concat to a dataframe table
    dfs = pd.read_html(content)
    df = pd.concat(dfs)

    #remove unwanted row (if not year of interest)
    indexNames = df[(df['Season'] != yr) & (df['Season'] != yr + ' *')].index
    df.drop(indexNames, inplace=True)

    #abstract players name from GM URL
    name_split = url.split('/')
    players_name = (name_split[4].replace('-', ' '))

    #Add column of player name, add player name from URL, and move to first column
    df['Player Name'] = players_name
    col_name = 'Player Name'
    first_col = df.pop(col_name)
    df.head
    df.insert(0,'Player Name', first_col)
    print(df)

我真的是编程新手，大约 3 周前才开始使用 python，所以答案越简单越好！谢谢

【问题讨论】：

标签： python pandas dataframe

【解决方案1】：

如果我理解要求，您只想将所有数据帧合并到一个数据帧。

试试这个代码：

dfFull = None
for url in url_list:
    .........
    print(df)
    if dfFull == None: 
        dfFull = df  # start with first set
    else:  # append new df
        dfFull.append(df, ignore_index=True) 

print(dfFull)

【讨论】：

嗨，迈克，感谢您提供代码。不幸的是，它返回了一个回溯错误Traceback (most recent call last): File "Real_GM_v2.py", line 65, in <module> if dfFull == None: File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pandas/core/generic.py", line 1326, in __nonzero__ raise ValueError( ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). 我可以将它附加到一个列表（a.item()）然后将列表连接回一个 DataFrame 吗？

【解决方案2】：

我已经做了更多的挖掘工作，并在以下link的上一个答案中找到了解决方案

#set new df 
real_gm_append = []

for url in url_list:
........................

    #append to new data frame
    real_gm_append.append(df)

real_gm_append = pd.concat(real_gm_append)
print(real_gm_append)

【讨论】：