【发布时间】：2022-10-20 18:16:12
【问题描述】：
我对网络报废非常陌生，也许 python3 也是如此，希望得到帮助以解决我的问题。
我正在从以下网页抓取：http://ets.aeso.ca/ets_web/ip/Market/Reports/CSDReportServlet
我收到的代码来自：New to Beautiful Soup. Need to scrape tables from an online report
这是我所拥有的代码示例。 （见下文）
这会产生：
GAS GAS.1 GAS.2 \
0 Simple Cycle Simple Cycle Simple Cycle
1 ASSET MC TNG
2 AB Newsprint (ANC1) 63 65
3 Bantry (ALP1) 7 6
4 Bellshill (BHL1) 5 5
5 Carson Creek (GEN5) 15 12
6 Cloverbar #1 (ENC1) 48 35
7 Cloverbar #2 (ENC2) 101 93
8 Cloverbar #3 (ENC3) 101 91
9 Crossfield Energy Centre #1 (CRS1) 48 0
10 Crossfield Energy Centre #2 (CRS2) 48 41
11 Crossfield Energy Centre #3 (CRS3)^ 48 41
问题：不知道为什么代码将它分成不同的列。 GAS.2 \ 这会将 GAS.3 的数据拆分到完全不同的行上。
我正在寻找的是如何将这些信息写入excel？ IE。需要删除前 3 行，GAS & Simple Cycle & ASSET 不是我需要显示的信息。
GAS GAS.1 GAS.2 \
0 Simple Cycle Simple Cycle Simple Cycle
1 ASSET MC TNG
2 AB Newsprint (ANC1) 63 65
3 Bantry (ALP1) 7 6
我需要具有两个值的字典中名称的前两项。
IE afc = {'AB Newsprint (ANC1)': {'MC':63,'TNG': 65}, 'Bantry (ALP1)': {'MC': 7,'TNG': 6}}
对于从 GAS 表中抓取的整个 td 列表，放入上面的字典中。
然后我需要以 DataPipe (excel) 方式显示它。
IE
A1 = AB Newsprint (ANC1)
B1 = 63
C1 = 65
A2 = Bantry (ALP1)
B2 = 7
C2 = 6
我应该如何编码或继续以完成此操作？
import requests, sys, re
import pandas as pd
from bs4 import BeautifulSoup
import numpy as np
#np.set_printoptions(threshold=sys.maxsize)
#np.set_printoptions(threshold='nan')
pd.set_option('display.max_rows', 100000)
pd.set_option('display.max_columns', 100000)
def get_summary(soup):
summary = soup.select_one(
"table:has(b:-soup-contains(SUMMARY)):not(:has(table))"
)
summary.tr.extract()
return pd.read_html(str(summary))[0]
def get_generation(soup):
generation = soup.select_one(
"table:has(b:-soup-contains(GENERATION)):not(:has(table))"
)
generation.tr.extract()
for td in generation.tr.select("td"):
td.name = "th"
return pd.read_html(str(generation))[0]
def get_interchange(soup):
interchange = soup.select_one(
"table:has(b:-soup-contains(INTERCHANGE)):not(:has(table))"
)
interchange.tr.extract()
for td in interchange.tr.select("td"):
td.name = "th"
return pd.read_html(str(interchange))[0]
def get_gas(soup):
gas = soup.select_one(
"table:has(b:-soup-contains(GAS)):not(:has(table))"
)
#for td in gas.tr.select("td"):
#td.name = "th"
return pd.read_html(str(gas))[0]
'''
def print_full(x):
dim = x.shape
pd.set_option('display.max_rows', dim[0])#dim[0] = len(x)
pd.set_option('display.max_columns', dim[1])
#print(x)
pd.reset_option('display.max_rows')
pd.reset_option('display.max_columns')
'''
url = "http://ets.aeso.ca/ets_web/ip/Market/Reports/CSDReportServlet"
#html5lib: A pure-python library for parsing HTML. It is designed to conform to the WHATWG HTML specification
soup = BeautifulSoup(requests.get(url).content, "html5lib")
#print(get_summary(soup))
#print(get_generation(soup))
#print(get_interchange(soup))
print (get_gas(soup))
#print(get_hydro(soup))
#print(get_biomass(soup))
#print(get_energystorage(soup))
#print(get_solar(soup))
#print(get_wind(soup))
#print(get_coal(soup))
【问题讨论】：
标签： python pandas beautifulsoup