Python Pandas - 从多个 URL 中抓取多个表到一个组合数据框中

Question

我是 Python 新手。

我正在尝试从网站上的页面中抓取多个表格。我想从每个表中创建一个数据框，并将所有数据框组合成一个大的组合数据框。

以下是我目前已完成的步骤：

1. 导入包
2. 打开网站并登录
3. 跳转到包含数据的页面
4. 将数字添加到字典并循环遍历它们以构建每个页面的 url。以零开头的数字是字符串，因此不会丢弃零。

这就是我卡住的地方。我可以创建一个数据框，但我无法为每个表创建一个数据框并将它们组合在一起。我已经尝试了 append 函数，但我最终得到了一个数据帧，所以我一定遗漏了一些东西。

澄清一下，我有大约 700 个网址。网址位于同一网站，但每个页面的网址末尾都有一个唯一的 4 位数字。我可以使用下面的代码遍历每个 url。当每个 url 打开时，我想从该 url 的表中创建一个数据框。然后我想将所有数据帧合并成一个大数据帧。

我希望这很清楚。任何帮助表示赞赏！

from bs4 import BeautifulSoup
import pandas as pd
from selenium import webdriver
DRIVER_PATH = 'C:\\Path to chromedriver.exe'
from selenium.webdriver import chrome
from selenium.webdriver.support.select import Select
driver = webdriver.Chrome(executable_path='C:\\Path to chromedriver.exe')
import numpy as np
from openpyxl import load_workbook

#open website 
driver.get('https://website.com')

#Login to website
element = driver.find_element_by_id('txtUID')
element.click()
element.clear()
element.send_keys("userID")
element = driver.find_element_by_id('txtPWD')
element.click()
element.clear()
element.send_keys("userPW")
import time
time.sleep(1)
element = driver.find_element_by_id('Send')
element.click()

#Jump to detail page
driver.get('https://DetailPage')
   
numbers = ['0101','0110','0130','0140','0150','0160','0170','0172','0180','0184','0185','0186','0190','0193','0203','0211','0213','0240','0260','0261','0270','0274','0280','0281','0285','0290','0300','0302','0310','0320','0340','0380','0412','0440','0490','0500','0540','0560','0603','0610','0630','0635','0652','0660','0670','0680','0681','0690','0713','0721','0740','0742','0750','0761','0776','0784','0790','0794','0801','0802','0804','0810','0851','0863','0870','0871','0881','0884','0886',1030,1056,1060,1090,1100,1170,1176,1180,1183,1194,1220,1230,1240,1251,1255,1260,1270,1290,1321,1360,1370,1405,1430,1441,1460,1501,1513,1522,1530,1560,1632,1650,1680,1686,1701,1711,1720,1740,1750,1760,1780,1790,1800,1806,1810,1820,1850,1860,1870,1900,1901,1903,1905,1917,1931,1940,1943,1950,1970,1972,1974,2002,2003,2060,2070,2072,2083,2100,2108,2110,2118,2121,2124,2127,2142,2170,2171,2180,2202,2203,2211,2212,2240,2260,2290,2302,2315,2321,2322,2332,2340,2341,2350,2360,2362,2380,2410,2440,2450,2510,2600,2720,2721,2725,2740,2760,2764,2801,2810,2811,2821,2824,2825,2830,2832,2850,2855,2860,2861,2862,2870,2874,2912,2930,2940,2943,2960,2961,2965,2970,2971,2992,3000,3003,3004,3012,3013,3014,3015,3022,3024,3025,3026,3027,3031,3032,3034,3040,3044,3045,3050,3051,3060,3070,3090,3120,3144,3160,3171,3172,3176,3190,3202,3203,3210,3220,3223,3250,3260,3261,3271,3275,3276,3283,3291,3296,3301,3306,3311,3313,3316,3326,3333,3340,3342,3343,3344,3350,3351,3352,3354,3357,3362,3363,3364,3371,3386,3390,3398,3461,3491,3523,3524,3531,3540,3560,3570,3581,3610,3620,3660,3661,3700,3702,3711,3715,3720,3725,3731,3740,3741,3783,3800,3810,3811,3830,3836,3850,3880,3881,3950,3961,3970,4016,4020,4021,4022,4023,4050,4052,4110,4122,4200,4210,4230,4243,4271,4305,4310,4313,4322,4324,4400,4403,4404,4411,4414,4416,4421,4422,4431,4432,4440,4470,4490,4501,4510,4511,4512,4521,4526,4530,4532,4541,4560,4580,4607,4620,4621,4622,4623,4624,4635,4636,4650,4651,4660,4681,4684,4690,4695,4715,4730,4737,4754,4770,4790,4793,4805,4806,4810,4813,4821,4822,4823,4830,4833,4834,4835,4836,4850,4881,4884,4902,4920,4942,4952,4955,5000,5010,5040,5070,5080,5110,5200,5220,5300,5301,5310,5320,5321,5340,5354,5370,5371,5390,5430,5431,5440,5441,5460,5486,5490,5510,5511,5521,5540,5541,5590,5594,5595,5630,5710,5850,5990,6001,6002,6004,6013,6014,6016,6017,6020,6030,6031,6043,6045,6051,6062,6066,6071,6090,6111,6135,6160,6182,6183,6220,6230,6265,6270,6300,6322,6334,6410,6412,6414,6480,6520,6560,6601,6603,6660,6780,6811,6814,6850,6860,6880,7000,7011,7031,7070,7081,7190,7222,7236,7360,7401,7414,7472,7500,7521,7525,7526,7534,7535,7552,7553,7560,7561,7570,7583,7587,7602,7610,7620,7623,7670,7702,7704,7706,7707,7711,7737,7738,7745,7753,7756,7763,7770,7780,7800,7820,7821,7823,7826,7837,7843,7850,7851,7852,7860,7872,7875,7876,7880,7882,7884,7940,7960,7970,7976,7986,7990,8000,8003,8007,8008,8010,8012,8013,8022,8025,8030,8040,8045,8046,8050,8091,8130,8137,8140,8150,8161,8200,8240,8290,8320,8321,8331,8332,8340,8346,8367,8370,8413,8440,8450,8451,8461,8465,8501,8502,8514,8520,8521,8522,8526,8530,8546,8565,8570,8571,8572,8582,8630,8640,8750,8820,8902,8904,8910,8914,8947,8970,9004,9010,9013,9014,9016,9040,9048,9050,9060,9061,9062,9080,9081,9120,9122,9132,9150,9153,9171,9172,9175,9200,9201,9202,9206,9211,9214,9221,9232,9234,9240,9246,9251,9253,9262,9270,9272,9300,9310,9321,9330,9331,9340,9371,9390,9402,9440,9450,9452,9453,9454,9455,9456,9460,9461,9470,9490,9491,9500,9505,9510,9511,9514,9530,9545,9550,9580,9581,9590,9591,9593,9701,9706,9723,9725,9730,9732,9740,9743,9750,9760,9770,9800,9802,9820,9822,9831,9844,9846,9861,9876,9910,9921,9930]

str = 'https://DetailPage={}'

for number in numbers:
    urls = str.format(number)
    print(urls)
    driver.get(urls)
    html = driver.page_source
    df = pd.read_html(html)

print(df)

Answer 1

在代码 df = pd.read_html(html) 中，您基本上创建了一个新的 data.frame 并覆盖了以前的结果。

所以你应该这样做：

result_df = []  # Create an empty list

for number in numbers:
    # Create url and get html page here

    result_df.append( pd.read_html(html) ) # Append generated dataframe from each html page to the list

现在检查每个生成的数据框，看看它们是否正确。您可以通过手动检查result_df[0] 等来做到这一点。

如果一切正确，请尝试将每个数据框合并在一起： ` final_df = pd.DataFrame() 对于 result_df 中的 df： final_df = final_df.append(df)

final_df 就是你要找的。​​h1>

`

顺便说一句，str 是 python 中的保留字符串类型，str = 'https://DetailPage={}' 不是最佳实践。创建 url 的最简单方法是在 for 循环中：

url = f'https://DetailPage={number}'

字符串的f 前缀告诉python 使用变量number 创建字符串。例如，如果 number 为 '0101'，则 url 将为 'https://DetailPage=0101'