如何使用 Python csv 仅为两个特定列编写多个 beautifulsoup 表行？

Question

我想使用 beautifulsoup 来抓取 HTML，以便从一个表的每一行中提取 仅 两列。但是，每个“tr”行都有 10 个“td”单元格，我只想要每行中的 [1] 和 [8] 个“td”单元格。最pythonic的方法是什么？

根据我在下面的输入，我有一个表格、一个正文、三行和每行 10 个单元格。

输入

<table id ="tblMain">
 <tbody>
  <tr>
   <td "text"</td>
   <td "data1"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "data2"</td>
   <td "text"</td>
  <tr>
   <td "text"</td>
   <td "data1"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "data2"</td>
   <td "text"</td>
  <tr>
   <td "text"</td>
   <td "data1"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "text"</td>
   <td "data2"</td>
   <td "text"</td>

我尝试过的事情

我了解如何使用单元格的索引来循环并在 [1] 和 [8] 处获取“td”。但是，当我试图将一行上的数据写回 csv 时，我感到很困惑。

table = soup.find('table', {'id':'tblMain'} )
table_body = table.find('tbody')
rows = table_body.findAll('tr')
data1_columns = []
data2_columns = []
for row in rows[1:]:
    data1 = row.findAll('td')[1]
    data1_columns.append(data1.text)
    data2 = row.findAll('td')[8]
    data2_columns.append(data2.text)

这是我当前的代码，它查找表、行和所有“td”单元格并将它们正确打印到 .csv。但是，我不想将每行的所有十个“td”单元格写回 csv 行，而只想抓取“td”[1] 和“td”[8]。

html = browser.page_source
soup = BeautifulSoup(html, 'html.parser')
table = soup.find('table', {'id':'tblMain'} )
table_body = table.find('tbody')
rows = table_body.findAll('tr')
filename = '%s.csv' % reportname
with open(filename, "wt+", newline="") as f:
    writer = csv.writer(f)
    for row in rows:
        csv_row = []
        for cell in row.findAll("td"):
            csv_row.append(cell.get_text())
        writer.writerow(csv_row)

预期结果

我希望能够将“td”[1] 和“td”[8] 写回我的 csv_row，以便将每个列表写回 csv writer.writerow。

将行写回 csv_row，然后写入我的 csv 文件：

['data1', 'data2']
['data1', 'data2']
['data1', 'data2']

Answer 1

你几乎做到了

for row in rows:
    row = row.findAll("td")
    csv_row = [row[1].get_text(), row[8].get_text()]
    writer.writerow(csv_row)

完整代码

html ='''<table id ="tblMain">
 <tbody>
  <tr>
   <td>text</td>
   <td>data1</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>data2</td>
   <td>text</td>
  <tr>
   <td>text</td>
   <td>data1</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>data2</td>
   <td>text</td>
  <tr>
   <td>text</td>
   <td>data1</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>text</td>
   <td>data2</td>
   <td>text</td>
'''

from bs4 import BeautifulSoup
import csv

soup = BeautifulSoup(html, 'html.parser')

table = soup.find('table', {'id':'tblMain'} )
table_body = table.find('tbody')

rows = table_body.findAll('tr')

reportname = 'output'
filename = '%s.csv' % reportname

with open(filename, "wt+", newline="") as f:
    writer = csv.writer(f)
    for row in rows:
        row = row.findAll("td")
        csv_row = [row[1].get_text(), row[8].get_text()]
        writer.writerow(csv_row)

Answer 2

你应该可以使用nth-of-type pseudo class css 选择器

from bs4 import BeautifulSoup as bs
import pandas as pd
html = 'actualHTML'
soup = bs(html, 'lxml')
results = []
for row in soup.select('#tblMain tr'):
    out_row = [item.text.strip() for item in row.select('td:nth-of-type(2), td:nth-of-type(9)')]
    results.append(out_row)
df = pd.DataFrame(results)
print(df)
df.to_csv(r'C:\Users\User\Desktop\data.csv', sep=',', encoding='utf-8-sig',index = False )

Answer 3

每当我需要拉出一张带有<table> 标签的表格时，我都会让 Pandas 为我完成这项工作，然后在需要时操作它返回的数据框。这就是我在这里要做的：

html = '''<table id ="tblMain">
 <tbody>
  <tr>
   <td> text</td>
   <td> data1</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> data2</td>
   <td> text</td>
  <tr>
   <td> text</td>
   <td> data1</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> data2</td>
   <td> text</td>
  <tr>
   <td> text</td>
   <td> data1</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> text</td>
   <td> data2</td>
   <td> text</td>'''


import pandas as pd

# .read_html() returns a list of dataframes
tables = pd.read_html(html)[0]

# we want the dataframe from that list in position [0]
df = tables[0]

# Use .iloc to say I want all the rows, and columns 1, 8
df = df.iloc[:,[1,8]]

# Write the dataframe to file
df.to_csv('path.filename.csv', index=False)