Google Maps 一些 XPATH 选择器返回一些不是 Selenium Python 的数据答案

【问题标题】：Google Maps some XPATH selectors return data some not Selenium PythonGoogle Maps 一些 XPATH 选择器返回一些不是 Selenium Python 的数据
【发布时间】：2020-08-25 10:01:38
【问题描述】：

我试图抓取谷歌地图。电话和小时变量不返回任何数据。其他变量工作正常并返回数据。 XPATH 是正确的。我不确定这里有什么问题。

这里是LINK

姓名、地址、职务、网站等其他选择器可以正常返回数据，但电话和小时数不返回任何数据。

希望得到一些答案。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
from scrapy.selector import Selector
import csv
from tqdm import tqdm
import time

driver = webdriver.Firefox()


linksFile=open("links.txt",'r')
allLinks = linksFile.readlines()
    
for link in tqdm(allLinks):

    try:
        driver.get(link)
    except Exception:
        print('Something went wrong with the URL: ')
    
    # time.sleep(15)
    
    while True:
        WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.XPATH, '//div[contains(text(), "Directions")] | //div[contains(text(), "Website")]'))
        )

        results = driver.find_elements_by_xpath('//div[contains(text(), "Directions")] | //div[contains(text(), "Website")]')
        for result in results:
            # writing to the CSV file
            outFile =  open("data.csv",'a+',newline="")
            writer = csv.writer(outFile)

            business = driver.find_element_by_xpath('//div[@role="heading"]/div')
            business.click()

            # waiting for the page to load
            WebDriverWait(driver, 15).until(
                EC.presence_of_element_located((By.XPATH, '//div[@class="immersive-container"]'))
            )


            # parcing response to the scrapy selector
            response = Selector(text=driver.page_source)

            name = response.xpath('//h2[@data-attrid="title"]/span/text()').get()
            title = response.xpath('(//span[contains(text(), "Google reviews")])/parent::a/parent::span/parent::span/parent::div/parent::div/parent::div/following-sibling::div/div/span/span/text()').get()
            address = response.xpath('//a[contains(text(), "Address")]/parent::span/following-sibling::span/text()').get()
            website = response.xpath('(//a[contains(text(), "Website")])/@href').get()
            phone = response.xpath('//a[contains(text(), "Phone")]/parent::span/following-sibling::span/a/span/text()').get()
            hours = response.xpath('//a[contains(text(), "Hours")]/parent::span/following-sibling::div/label/span//btext()').get()
            total_reviews = response.xpath('(//span[contains(text(), "Google reviews")])[1]/text()').get()
            total_rating = response.xpath('(//span[contains(text(), "Google reviews")])/parent::a/parent::span/parent::span/parent::div/span/text()').get()


            input('Check: ')




            outFile =  open("data.csv",'a+',newline="")
            writer = csv.writer(outFile)
            
            vals = [name, title, address, website, phone, hours, total_reviews, total_rating]
            writer.writerow(vals)
            outFile.close()

【问题讨论】：

您是否可以分享您正在点击的链接。由于谷歌地图中的每个位置都没有电话和营业时间。
嗨 Rahul，我已经更新了问题中的链接。

标签： python selenium selenium-webdriver scrapy

【解决方案1】：

你可以使用 Java 脚本 outerHTML 而不是 pageSource。

response = Selector( driver.execute_script("return document.documentElement.outerHTML"))

在 xpath of Hours 中也存在问题：

hours = response.xpath('//a[contains(text(), "Hours")]/parent::span/following-sibling::div/label/span//b/text()').get()

【讨论】：