使用beautifulsoup从带有ID的网站抓取表格

Question

我在抓取该网站的表格时遇到问题，我应该得到标题，但得到的是

AttributeError: 'NoneType' object has no attribute 'tbody'

我对网络抓取有点陌生，所以如果你能帮助我，那就太好了

import requests
from bs4 import BeautifulSoup

URL = "https://www.collincad.org/propertysearch?situs_street=Willowgate&situs_street_suffix" \
      "=&isd%5B%5D=any&city%5B%5D=any&prop_type%5B%5D=R&prop_type%5B%5D=P&prop_type%5B%5D=MH&active%5B%5D=1&year=2021&sort=G&page_number=1"

s = requests.Session()

page = s.get(URL)
soup = BeautifulSoup(page.content, "lxml")

table = soup.find("table", id="propertysearchresults")
table_data = table.tbody.find_all("tr")

headings = []
for td in table_data[0].find_all("td"):
    headings.append(td.b.text.replace('\n', ' ').strip())

print(headings)

Answer 1

会发生什么？

注意： 总是先看看你的汤——这就是事实。内容总是与开发工具中的视图略有不同。

访问被撤销

您的 IP 地址已被阻止。

我们 检测到对我们的 Property Search 发起的不规则、类似机器人的使用 从您的 IP 地址。设置这个块是为了减轻我们的压力 网络服务器，以确保我们提供最佳的网站性能 科林县的纳税人。

我们有 没有阻止您下载our data exports，您仍然可以使用它来获取批量财产 数据。

如何解决？

将user-agent 添加到您的请求中，使其看起来像您使用“浏览器”进行请求。

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}
page = s.get(URL,headers=headers)

或者作为替代方法，只需下载结果。

示例（刮表）

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

URL = "https://www.collincad.org/propertysearch?situs_street=Willowgate&situs_street_suffix" \
      "=&isd%5B%5D=any&city%5B%5D=any&prop_type%5B%5D=R&prop_type%5B%5D=P&prop_type%5B%5D=MH&active%5B%5D=1&year=2021&sort=G&page_number=1"

s = requests.Session()

page = s.get(URL,headers=headers)
soup = BeautifulSoup(page.content, "lxml")

data = []
for row in soup.select('#propertysearchresults tr'):
    data.append([c.get_text(' ',strip=True) for c in row.select('td')])

pd.DataFrame(data[1:], columns=data[0])

输出

	Property ID ↓ Geographic ID ↓	Owner Name	Property Address	Legal Description	2021 Market Value
1	2709013 R-10644-00H-0010-1	PARTHASARATHY SURESH & ANITHA HARIKRISHNAN	12209 Willowgate Dr Frisco, TX\xa0 75035	Ridgeview At Panther Creek Phase 2, Blk H, Lot 1	$513,019
2	2709018 R-10644-00H-0020-1	JOSHI PRASHANT & SHWETA PANT	12235 Willowgate Dr Frisco, TX\xa0 75035	Ridgeview At Panther Creek Phase 2, Blk H, Lot 2	$546,254
3	2709019 R-10644-00H-0030-1	THALLAPUREDDY RAVENDRA & UMA MAHESWARI VEMULA	12261 Willowgate Dr Frisco, TX\xa0 75035	Ridgeview At Panther Creek Phase 2, Blk H, Lot 3	$550,768
4	2709020 R-10644-00H-0040-1	KULKARNI BHEEMSEN T & GOURI R	12287 Willowgate Dr Frisco, TX\xa0 75035	Ridgeview At Panther Creek Phase 2, Blk H, Lot 4	$509,593
5	2709021 R-10644-00H-0050-1	BALAM GANESH & SHANTHIREKHA LOKULA	12313 Willowgate Dr Frisco, TX\xa0 75035	Ridgeview At Panther Creek Phase 2, Blk H, Lot 5	$553,949

...

Answer 2

import requests
from bs4 import BeautifulSoup

URL = "https://www.collincad.org/propertysearch?situs_street=Willowgate&situs_street_suffix" \
      "=&isd%5B%5D=any&city%5B%5D=any&prop_type%5B%5D=R&prop_type%5B%5D=P&prop_type%5B%5D=MH&active%5B%5D=1&year=2021&sort=G&page_number=1"

s = requests.Session()
headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"}
page = s.get(URL,headers=headers)
soup = BeautifulSoup(page.content, "lxml")

查找表数据：

column_data=soup.find("table").find_all("tr")[0]
column=[i.get_text() for i in column_data.find_all("td") if i.get_text()!=""]

row=soup.find("table").find_all("tr")[1:]
main_lst=[]
for row_details in row:
    lst=[]
    for i in row_details.find_all("td")[1:]:
        if i.get_text()!="":
            lst.append(i.get_text())
    main_lst.append(lst)

转换为pandas DataFrame：

import pandas as pd
df=pd.DataFrame(main_lst,columns=column)

输出：

Property ID↓ Geographic ID ↓    Owner Name  Property Address    Legal Description   2021 Market Value
0   2709013R-10644-00H-0010-1   PARTHASARATHY SURESH & ANITHA HARIKRISHNAN  12209 Willowgate DrFrisco, TX  75035    Ridgeview At Panther Creek Phase 2, Blk H, Lot 1    $513,019
.....

Answer 3

如果您查看page.content，您将看到“您的 IP 地址已被阻止”。

您应该在请求中添加一些标头，因为该网站阻止了您的请求。在您的具体情况下，添加User-Agent 就足够了：

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
}

URL = "https://www.collincad.org/propertysearch?situs_street=Willowgate&situs_street_suffix" \
      "=&isd%5B%5D=any&city%5B%5D=any&prop_type%5B%5D=R&prop_type%5B%5D=P&prop_type%5B%5D=MH&active%5B%5D=1&year=2021&sort=G&page_number=1"

s = requests.Session()

page = s.get(URL, headers=headers)
soup = BeautifulSoup(page.content, "lxml")

table = soup.find("table", id="propertysearchresults")
table_data = table.tbody.find_all("tr")

headings = []
for td in table_data[0].find_all("td"):
    headings.append(td.b.text.replace('\n', ' ').strip())

print(headings)

如果加了表头，还是会报错，但是在行里：

headings.append(td.b.text.replace('\n', ' ').strip())

你应该把它改成

headings.append(td.text.replace('\n', ' ').strip())

因为td 并不总是有b。