在循环 Python/BS4/Selenium 中，抓取结果不同

【问题标题】：Scraping results differ when in a loop Python/BS4/Selenium在循环 Python/BS4/Selenium 中，抓取结果不同
【发布时间】：2020-04-20 19:36:55
【问题描述】：

我有一个 csv 文件，其中包含我需要抓取的链接。我还设置了使用相同的 chrome 浏览器进行登录（我需要的元素仅在登录时可用）。当我在循环之外抓取单个页面时，我会从页面中获得我需要的结果。当我将相同的代码放入循环中以抓取所有链接时，我会得到不同的结果。我认为这与“source =”和或“soup =”有关

CSV 文件包含 3 个链接：

https://www.redfin.com/UT/Murray/875-E-Arrowhead-Ln-84107/unit-44/home/77418264
https://www.redfin.com/UT/Murray/35-W-American-Ave-84107/home/86446505
https://www.redfin.com/UT/Murray/4551-S-200-E-84107/home/86457987

单页代码：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
#####################################
chrome_driver = "C:/chromedriver.exe"
Chrome_options = Options()
Chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9015")
driver = webdriver.Chrome(chrome_driver, options=Chrome_options)
#####################################
driver.get("https://www.redfin.com/UT/Murray/4551-S-200-E-84107/home/86457987")
source = driver.page_source
soup = BeautifulSoup(source, "html.parser")
#####################################
address = soup.find('span', class_='street-address').text
print("      Address: " + address)
city = soup.find('span', class_='locality').text
print("         City: " + city)
state = soup.find('span', class_='region').text
print("        State: " + state)
zipcode = soup.find('span', class_='postal-code').text
print("      ZipCode: " + zipcode)
soldPrice = soup.find('div', class_='price-col number').text
print("   Sold Price: " + soldPrice)
ln = soup.find('div', class_='listing-agent-item')
Lname = ln.find_all('span')[1].text
print("Listing Agent: " + Lname)
bn = soup.find('div', class_='buyer-agent-item')
Bname = bn.find_all('span')[1].text
print(" Buying Agent: " + Bname)
date = soup.find('div',attrs={"class":"col-4"})
sDate = date.find_all('p')[0].text
print("         Date: " + sDate)
mls = soup.find('div', class_='sourceContent').text
print("   MLS Source: " + mls)
for span in soup.find_all('span'):
    if span.find(text='MLS#'):
            mlsNum = span.nextSibling.text
print("         MLS#: " + mlsNum)
driver.quit()

单页结果完美显示：

      Address: 4551 S 200 E 
         City: Murray, 
        State: UT
      ZipCode: 84107
   Sold Price: $262,000 
Listing Agent: Jerold Ivie
 Buying Agent: Zac Eldridge
         Date: Dec 20, 2019
   MLS Source: WFRMLS
         MLS#: 1635000
[Finished in 3.3s]

循环前带有“source=”和“driver=”的循环代码：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
import csv
#####################################
chrome_driver = "C:/chromedriver.exe"
Chrome_options = Options()
Chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9015")
driver = webdriver.Chrome(chrome_driver, options=Chrome_options)
#####################################
#driver.get("https://www.redfin.com/UT/Murray/4551-S-200-E-84107/home/86457987")
source = driver.page_source
soup = BeautifulSoup(source, "html.parser")
#####################################
with open('UTlinks.csv') as file:
    readCSV = csv.reader(file)
    for row in readCSV:
        url = str(row).replace("['","").replace("']","")
        print("_________________________________")
        print("Scraping: " + url)        
        driver.get(url)
        #source = driver.page_source
        #soup = BeautifulSoup(source, "html.parser")
####################################
        try:
                address = soup.find('span', class_='street-address').text
                print("      Address: " + address)
        except:
                print("      Address: " + "NA")
        try:
                city = soup.find('span', class_='locality').text
                print("         City: " + city)
        except:
                print("         City: " + "NA")
        try:
                state = soup.find('span', class_='region').text
                print("        State: " + state)
        except:
                print("        State: " + "NA")
        try:
                zipcode = soup.find('span', class_='postal-code').text
                print("      ZipCode: " + zipcode)
        except:
                print("      ZipCode: " + "NA")
        try:
                soldPrice = soup.find('div', class_='price-col number').text
                print("   Sold Price: " + soldPrice)
        except:
                print("   Sold Price: " "NA")            
        try:
                ln = soup.find('div', class_='listing-agent-item')
                Lname = ln.find_all('span')[1].text
                print("Listing Agent: " + Lname)
        except:
                print("Listing Agent: " + "NA")
        try:
                bn = soup.find('div', class_='buyer-agent-item')
                Bname = bn.find_all('span')[1].text
                print(" Buying Agent: " + Bname)
        except:
                print(" Buying Agent: " + "NA")
        try:
                date = soup.find('div',attrs={"class":"col-4"})
                sDate = date.find_all('p')[0].text
                print("         Date: " + sDate)
        except:
                print("         Date: " + "NA")
        try:
                mls = soup.find('div', class_='sourceContent').text
                print("   MLS Source: " + mls)
        except:
                print("   MLS Source: " + "NA")
        try:
                for span in soup.find_all('span'):
                        if span.find(text='MLS#'):
                                mlsNum = span.nextSibling.text
                print("         MLS#: " + mlsNum)
        except:
                print("         MLS#: " + "NA")

循环结果： 您可以看到它打印文件中的 url，然后刮掉当前打开的浏览器结果 3 次......但抓取打开的 url 所需的所有信息。

_________________________________
Scraping: https://www.redfin.com/UT/Murray/875-E-Arrowhead-Ln-84107/unit-44/home/77418264
      Address: 4551 S 200 E 
         City: Murray, 
        State: UT
      ZipCode: 84107
   Sold Price: $262,000 
Listing Agent: Jerold Ivie
 Buying Agent: Zac Eldridge
         Date: Dec 20, 2019
   MLS Source: WFRMLS
         MLS#: 1635000
_________________________________
Scraping: https://www.redfin.com/UT/Murray/35-W-American-Ave-84107/home/86446505
      Address: 4551 S 200 E 
         City: Murray, 
        State: UT
      ZipCode: 84107
   Sold Price: $262,000 
Listing Agent: Jerold Ivie
 Buying Agent: Zac Eldridge
         Date: Dec 20, 2019
   MLS Source: WFRMLS
         MLS#: 1635000
_________________________________
Scraping: https://www.redfin.com/UT/Murray/4551-S-200-E-84107/home/86457987
      Address: 4551 S 200 E 
         City: Murray, 
        State: UT
      ZipCode: 84107
   Sold Price: $262,000 
Listing Agent: Jerold Ivie
 Buying Agent: Zac Eldridge
         Date: Dec 20, 2019
   MLS Source: WFRMLS
         MLS#: 1635000
[Finished in 6.9s]

如果我将 'source=' 和 'soup =' 放入循环中：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
#####################################
chrome_driver = "C:/chromedriver.exe"
Chrome_options = Options()
Chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9015")
driver = webdriver.Chrome(chrome_driver, options=Chrome_options)
#####################################
#driver.get("https://www.redfin.com/UT/Murray/4551-S-200-E- 84107/home/86457987")
#source = driver.page_source
#soup = BeautifulSoup(source, "html.parser")
#####################################
with open('UTlinks.csv') as file:
    readCSV = csv.reader(file)
    for row in readCSV:
        url = str(row).replace("['","").replace("']","")
        print("_________________________________")
        print("Scraping: " + url)        
        driver.get(url)
        source = driver.page_source
        soup = BeautifulSoup(source, "html.parser")
####################################
        try:
                address = soup.find('span', class_='street-address').text
                print("      Address: " + address)
        except:
                print("      Address: " + "NA")
        try:
                city = soup.find('span', class_='locality').text
                print("         City: " + city)
        except:
                print("         City: " + "NA")
        try:
                state = soup.find('span', class_='region').text
                print("        State: " + state)
        except:
                print("        State: " + "NA")
        try:
                zipcode = soup.find('span', class_='postal-code').text
                print("      ZipCode: " + zipcode)
        except:
                print("      ZipCode: " + "NA")
        try:
                soldPrice = soup.find('div', class_='price-col number').text
                print("   Sold Price: " + soldPrice)
        except:
                print("   Sold Price: " "NA")            
        try:
                ln = soup.find('div', class_='listing-agent-item')
                Lname = ln.find_all('span')[1].text
                print("Listing Agent: " + Lname)
        except:
                print("Listing Agent: " + "NA")
        try:
                bn = soup.find('div', class_='buyer-agent-item')
                Bname = bn.find_all('span')[1].text
                print(" Buying Agent: " + Bname)
        except:
                print(" Buying Agent: " + "NA")
        try:
                date = soup.find('div',attrs={"class":"col-4"})
                sDate = date.find_all('p')[0].text
                print("         Date: " + sDate)
        except:
                print("         Date: " + "NA")
        try:
                mls = soup.find('div', class_='sourceContent').text
                print("   MLS Source: " + mls)
        except:
                print("   MLS Source: " + "NA")
        try:
                for span in soup.find_all('span'):
                        if span.find(text='MLS#'):
                                mlsNum = span.nextSibling.text
                print("         MLS#: " + mlsNum)
        except:
                print("         MLS#: " + "NA")

'source =' & 'soup =' 在循环结果中：

Scraping: https://www.redfin.com/UT/Murray/875-E-Arrowhead-Ln-84107/unit-44/home/77418264
      Address: 875 E Arrow Head Ln S #44 
         City: Salt Lake City, 
        State: UT
      ZipCode: 84107
   Sold Price: NA
Listing Agent: Joe Olschewski
 Buying Agent: James Corey
         Date: NA
   MLS Source: WFRMLS
         MLS#: 1654937
_________________________________
Scraping: https://www.redfin.com/UT/Murray/35-W-American-Ave- 84107/home/86446505
      Address: 35 American Ave 
         City: Murray, 
        State: UT
      ZipCode: 84107
   Sold Price: NA
Listing Agent: Dana Conway
 Buying Agent: Rich Varga
         Date: NA
   MLS Source: WFRMLS
         MLS#: 1660023
_________________________________
Scraping: https://www.redfin.com/UT/Murray/4551-S-200-E-84107/home/86457987
      Address: 4551 S 200 E 
         City: Murray, 
        State: UT
      ZipCode: 84107
   Sold Price: NA
Listing Agent: Jerold Ivie
 Buying Agent: Zac Eldridge
         Date: NA
   MLS Source: WFRMLS
         MLS#: 1635000
[Finished in 8.6s]

现在它工作正常，但没有抓取“Sold Price:”或“Sold Date:”。如果我关闭错误处理，它会抛出这个错误：

soldPrice = soup.find('div', class_='price-col number').text
AttributeError: 'NoneType' object has no attribute 'text'

我在这里做错了什么？

【问题讨论】：

  • 查看第一个链接时，我发现这个元素不包含数字。 soup.find('div', class_='price-col number')
  • @Sri，不确定您的意思？使用相同的选择器在循环外也能正常工作。
  • 我刚刚运行它，但它在这一行不起作用。 soldPrice = soup.find('div', class_='price-col number').text.我得到 NoneType 对象没有属性文本，这意味着它找不到该元素。
  • @Sri，ZipCode后面的数据只有在登录的情况下才可用。你可以从第一块代码和结果中看到，它在那里并找到它。甚至是第二个代码块。不明白为什么只有这两个元素让我在循环中感到悲伤

标签： python-3.x selenium beautifulsoup


【解决方案1】：

有一个 API 可以获取数据。不过有点棘手。我能够在不注册/登录的情况下提取数据（尽管由于某种原因我在 html 或 json 响应中唯一找不到的是采购代理）。但是，如果您登录，它似乎会提供该数据。看起来其他一切（以及更多）都在那里。

import requests
from bs4 import BeautifulSoup
import json

links = ['https://www.redfin.com/UT/Murray/875-E-Arrowhead-Ln-84107/unit-44/home/77418264',
'https://www.redfin.com/UT/Murray/35-W-American-Ave-84107/home/86446505',
'https://www.redfin.com/UT/Murray/4551-S-200-E-84107/home/86457987']

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'}

payload = {
'email':'username@email.com',
'pwd':'thisIsThePassword'}

#cookiesStr = ''
with requests.Session() as s:       
    login = s.post('https://www.redfin.com/stingray/do/api-login', headers=headers, params=payload)
    #cookies = s.cookies.get_dict()
    #for k, v in cookies.items():
    #    cookiesStr += '%s=%s;' %(k,v)

#headers.update({'cookie':cookiesStr}) 

for url in links:
    response = s.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')

    scripts = soup.find_all('script')
    for script in scripts:
        if '_tLAB.wait(function()' in script.text and '/stingray/api/home/details/belowTheFold' in script.text:
            jsonStr = script.text
            jsonStr = '{' + jsonStr.split('{',2)[-1].rsplit(')',2)[0]

            jsonData2 = json.loads(jsonStr)
            jsonData2 = json.loads(jsonData2['res']['text'].split('&&')[-1])

            address = jsonData2['payload']['amenitiesInfo']['addressInfo']['street']
            city = jsonData2['payload']['amenitiesInfo']['addressInfo']['city']
            state = jsonData2['payload']['amenitiesInfo']['addressInfo']['state']
            zipcode = jsonData2['payload']['amenitiesInfo']['addressInfo']['zip']

            mlsSource = jsonData2['payload']['amenitiesInfo']['provider']
            listingAgent = jsonData2['payload']['amenitiesInfo']['mlsDisclaimerInfo']['listingAgentName']

        if 'InitialContext = ' in script.text:
            jsonStr = script.text.split('InitialContext = ')[-1].split('root.__reactServerState.Config')[0].rsplit(';',1)[0]
            jsonData = json.loads(jsonStr)

            dataAPIs = jsonData['ReactServerAgent.cache']['dataCache']
            jsonData2 = json.loads(dataAPIs['/stingray/api/home/details/aboveTheFold']['res']['text'].split('&&')[-1])
            soldPrice = jsonData2['payload']['addressSectionInfo']['priceInfo']['amount']
            soldDate = jsonData2['payload']['mediaBrowserInfo']['sashes'][0]['lastSaleDate']

            jsonData2 = json.loads(dataAPIs['/stingray/api/home/details/initialInfo']['res']['text'].split('&&')[-1])
            mls = jsonData2['payload']['mlsId']

            jsonData2 = json.loads(dataAPIs['/stingray/api/home/details/mainHouseInfoPanelInfo']['res']['text'].split('&&')[-1])
            buyingAgents = jsonData2['payload']['mainHouseInfo']['buyingAgents'][0]['agentInfo']['agentName']

    print("_________________________________")
    print("Scraping: " + url)  
    print('%15s: %s' %('Address',address))
    print('%15s: %s' %('City',city))
    print('%15s: %s' %('State',state))
    print('%15s: %s' %('Zipcode',zipcode))
    print('%15s: $' %('Sold Price') + f'{soldPrice:,}')
    print('%15s: %s' %('Listing Agent',listingAgent))
    print('%15s: %s' %('Buying Agent',buyingAgents))
    print('%15s: %s' %('Date',soldDate))
    print('%15s: %s' %('MLS Source',mlsSource))
    print('%15s: %s' %('MLS#',mls))

输出：

_________________________________
Scraping: https://www.redfin.com/UT/Murray/875-E-Arrowhead-Ln-84107/unit-44/home/77418264
        Address: 875 E Arrow Head Lane South Unit 44
           City: Salt Lake City
          State: UT
        Zipcode: 84107
     Sold Price: $179,900
  Listing Agent: Joe Olschewski
   Buying Agent: James Corey
           Date: MAR 12, 2020
     MLS Source: WFRMLS
           MLS#: 1654937
_________________________________
Scraping: https://www.redfin.com/UT/Murray/35-W-American-Ave-84107/home/86446505
        Address: 35 American Ave
           City: Murray
          State: UT
        Zipcode: 84107
     Sold Price: $317,500
  Listing Agent: Dana Conway
   Buying Agent: Rich Varga
           Date: MAR 25, 2020
     MLS Source: WFRMLS
           MLS#: 1660023
_________________________________
Scraping: https://www.redfin.com/UT/Murray/4551-S-200-E-84107/home/86457987
        Address: 4551 South 200 E
           City: Murray
          State: UT
        Zipcode: 84107
     Sold Price: $262,000
  Listing Agent: Jerold Ivie
   Buying Agent: Zac Eldridge
           Date: DEC 20, 2019
     MLS Source: WFRMLS
           MLS#: 1635000

【讨论】：

  • 是的，@chitown88。不过，我真的需要买家信息。我尝试了 API。而不是时间延迟，我尝试循环直到找到运行脚本的元素......它总是抛出相同的错误，就像它找不到元素文本一样。困惑
  • @RyanEvans。我明天再看一遍。我将创建一个登录名并按此方式进行操作。
  • @RyanEvans 好的。是的，您确实需要登录才能获得。我调整了代码。你只需要输入你的用户名，通过。
  • 一直在玩它。它不会抛出任何错误，但不会打印任何结果。试图将结果发送到 csv 但给了我一个空白文件。试图破译你的代码，但它比我的尝试更先进......有什么想法吗？
猜你喜欢
  • 1970-01-01
  • 2019-08-04
  • 2020-07-30
  • 1970-01-01
  • 2019-07-02
  • 2021-06-07
  • 1970-01-01
  • 2013-12-04
  • 1970-01-01
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode