从表格中获取特定单元格的内容

Question

我正在尝试解析来自例如this page 的专利数据。最终输出应该是一个 CSV 文件，每个专利有一行，每行都有（受让人、家族 ID 和申请日期）。 我正在使用 BeautifulSoup，我可以很好地检索大部分信息并编写 CSV 文件。

我的问题是我注意到表格的结构会随着时间而变化；并非所有字段都一直存在。例如。在给出的示例中，没有家庭 ID。 因此，我无法将每个单元格分配给特定变量（例如here）。根据报告的字段数，行/列数也会发生变化。

我想编写代码，使其足够灵活，可以执行以下操作： 如果 header == "Assignee"，则从该单元格获取文本。否则，请将其留空。 如果 header == "Family ID"，则从该单元格获取文本。否则，请将其留空。

这样最终的输出将类似于：

Assignee, Family ID, filing date
"Potomac Aviation",         , "June 11, 2002"
"Anonymous Co", 40432687, "June 5, 2016"
etc. 

这段代码让我最接近，但我离最终的目标还很远。

fourth_table=table.find_next("table")
header_1 = fourth_table.find('th')
if header_1.get_text() == "Inventors:":
   inventors=fourth_table.find('td').get_text()

header1=fourth_table.th
header_2 = header1.find_next('th')
cell1=fourth_table.td
cell2=cell1.find_next('td')
if header_2.get_text() == "Applicant:":
   applicant= cell2.find('td').get_text()

显然很罗嗦；一旦我确定我理解每一位是如何工作的，我将尝试让代码更高效。

编辑：我认为这是一个让我更接近的替代方案。然而，虽然它适用于“Assignee”，python 为 print(family_id) 行返回“None”。我检查了拼写。

 fourth_table=table.find_next("table")
 assignee=fourth_table.find(text="Assignee:").findNext('td').get_text().replace("\n","").strip()
#family_id=fourth_table.find(text="Family ID:").findNext('td').get_text().replace("\n","").strip()
 family_id=fourth_table.find(text="Family ID:")
 print(family_id)

如果我遗漏了一些明显的东西，我们深表歉意。蒂亚！

Answer 1

这有点复杂，你真的需要使用正则表达式来找到你想要的行，因为文本中有换行符，所以你不能只指定text="Family ID": 等等，这将从中获取所有链接该站点并写入感兴趣的三行（如果可用），如果该行不存在，则 csv 将有一个缺失数据的空条目：

import csv
import re


def get_nxt_link(soup):
    nxt = soup.select_one("img[src=/netaicon/PTO/nextdoc.gif]")
    if nxt:
        return urljoin(base, nxt.parent["href"])
    return False

start = 'http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO2&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-bool.html&r=1&f=G&l=50&co1=AND&d=PTXT&s1=boston&s2=2005$.PD.&OS=boston%20AND%20ISD/2005&RS=boston%20AND%20ISD/2005'

base = "http://patft.uspto.gov/"
def get_text(soup, txt):
    for t in txt:
        _tag = soup.find("th", text=t)
        yield _tag.parent.td.text.replace("\n", "") if _tag else ""


with open("out.csv", "w") as o:
    wr = csv.writer(o)
    soup = BeautifulSoup(requests.get(start).content, "lxml")
    table = soup.find("th", text="Assignee:").find_previous("table")
    nxt = get_nxt_link(soup)
    wr.writerow(["Assignee", "Family ID", "filing date"])
    text = [re.compile("Assignee:"), re.compile("Family ID:"), re.compile("Filed:")]
    wr.writerow(tuple(get_text(table, text)))
    while nxt:
        soup = BeautifulSoup(requests.get(nxt).content, "lxml")
        table = soup.find("th", text="Assignee:").find_previous("table")
        wr.writerow(tuple(get_text(table, text)))
        nxt = get_nxt_link(soup)

在我们获得的前几个 url 上运行：

Assignee,Family ID,filing date
"Potomac Aviation Technology Corp. (Boston, MA)",,"June 11, 2002"
"Microsoft Corp. (Redmond, WA)",33563549,"December 31, 1998"
"Teradyne, Inc. (Boston, MA)",32029308,"September 27, 2002"
"First Data Corporation (Greenwood Village, CO)",22696181,"December 5, 2001"
"Micron Technology, Inc. (Boise, ID)",35482789,"July 8, 2003"
"Digital River, Inc. (Eden Prairie, MN)",38226542,"December 11, 2000"
"Oracle International Corporation (Redwood Shores, CA)",35482753,"October 1, 2002"
"The United States of America as represented by the Secretary of the Navy (Washington, DC)",35482734,"October 6, 2003"
Garmin Ltd. (KY),21836040,"November 19, 2004"
"Lucent Technologies Inc. (Murray Hill, NJ)",25439780,"July 30, 2001"
"The Chamberlain Group, Inc. (Elmhurst, IL)",23942291,"October 17, 2001"
"Inplane Photonics, Inc. (South Plainfield, NJ)",32824042,"February 7, 2003"
"Motorola, Inc (Horsham, PA)",22314958,"November 17, 2000"
"Xerox Corporation (Stamford, CT)",35482631,"July 7, 2004"
"General Hospital Corporation (Boston, MA)",35480219,"October 16, 2002"