Python - Web-Scraping - 解析 HTML 表 - 将多个 href 合并为一列

Question

我正在从我的客户网站中提取一个表格，我需要将此 HTML 解析为 Pandas 数据框。但是，在表格上，我想将所有 HREF 存储到我的数据框中。 我的 HTML 具有以下架构：

<table>
    <tr>
         <th>Col_1</th>
         <th>Col_2</th>
         <th>Col_3</th>
         <th>Col_4</th>
         <th>Col_5</th>
         <th>Col_6</th>
         <th>Col_7</th>
         <th>Col_8</th>
         <th>Col_9</th>
    </tr>
    <tr>
         <td>Office</td>
         <td>Office2</td>
         <td>Customer</td>
         <td></td>
         <td><a href="test12345_163">New Doc</a><br><a href="test12345_163">my_work.jpg</a></td>
         <td><a href="test12345_163">Person_2</a><br><a href="test12345_163">Person_3</a><br><a href="test12345_163">Person 3</a></td>
         <td><a href="test12345_163">Person_1</a><br><a href="test12345_163">Person_1</a><br><a href="test12345_163">Person_1</a><br><a href="test12345_163">Person_1</a></td>
         <td>STATUS</td>
         <td>9030303</td>
    </tr>
</table>

我有这个代码：

soup = BeautifulSoup(page.content, "html.parser")

html_table = soup.find('table')

df = pd.read_html(str(html_table), header=0)[0]
df['Link'] = [link.get('href') for link in html_table.find_all('a')]

我只是想创建一个包含来自每个索引的所有链接的列（如果有多个链接，则将其分组）。但是当我运行这段代码时，我得到了：

Length of values (1102) does not match length of index (435)

我做错了什么？

谢谢！

Answer 1

你不需要read_html，Dataframe 应该这样定义：

html_table = soup.find('table')
hyperlinks=soup.find_all("a")
l=[]
for a in hyperlinks:
    l.append([a.text,a.get("href")])
pd.DataFrame(l,columns=["Names","Links"])

更新：

#here we get headers:
headers=[]
html_table = soup.find('table')
trs=html_table.find_all("tr")
headers=[th.text for th in trs[0].find_all("th")]
#an empty dataframe with all headers as columns and one row index:
df=pd.DataFrame(columns=headers,index=[0])
#here we get contents:
body_td=trs[1].find_all("td")
i=0
for td in body_td:
    HyperLinks=td.find_all("a")
    cell=[a.get("href") for a in HyperLinks]
    df.iloc[0,i]=cell
    i+=1

Answer 2

您可以在循环 tds 之前获取链接，使用列表推导获取给定行的所有 href；将所有 td 文本抓取到一个列表中，并使用包含一项的嵌套列表扩展该列表，这是您之前收集的 href 列表：

from bs4 import BeautifulSoup as bs
import pandas as pd

html = '''<table>
    <tr>
         <th>Col_1</th>
         <th>Col_2</th>
         <th>Col_3</th>
         <th>Col_4</th>
         <th>Col_5</th>
         <th>Col_6</th>
         <th>Col_7</th>
         <th>Col_8</th>
         <th>Col_9</th>
    </tr>
    <tr>
         <td>Office</td>
         <td>Office2</td>
         <td>Customer</td>
         <td></td>
         <td><a href="test12345_163">New Doc</a><br><a href="test12345_163">my_work.jpg</a></td>
         <td><a href="test12345_163">Person_2</a><br><a href="test12345_163">Person_3</a><br><a href="test12345_163">Person 3</a></td>
         <td><a href="test12345_163">Person_1</a><br><a href="test12345_163">Person_1</a><br><a href="test12345_163">Person_1</a><br><a href="test12345_163">Person_1</a></td>
         <td>STATUS</td>
         <td>9030303</td>
    </tr>
</table>'''
soup = bs(html, 'lxml')
results = []
headers = [i.text for i in soup.select('table th')]
headers.append('Links')

for _row in soup.select('table tr')[1:]:
    row = []
    links = [i['href'] for i in _row.select('a')]
    for _td in _row.select('td'):
        row.append(_td.text)
    row.extend([links])
    results.append(row)

df = pd.DataFrame(results, columns = headers)
df