【问题标题】:Extracting information from same page popup in python Selenium webscraping从python Selenium webscraping中的同一页面弹出窗口中提取信息
【发布时间】:2021-01-24 05:47:50
【问题描述】:

注意:我在 python 方面经验丰富,但刚开始使用 selenium 和 webscraping。如果这是一个不好的问题,或者我在硒方面的基础知识似乎有问题,请原谅。我在几个小时的搜索中找不到答案,所以我在这里问

目标:提取企业 Yelp 页面中的“About the Business”信息 某些页面在基于“阅读更多”按钮的弹出窗口中包含有关业务信息(例如:https://www.yelp.com/biz/and-pizza-bethesda-bethesda) 某些页面在基于阅读更多按钮的弹出窗口中没有他们的业务信息(例如:https://www.yelp.com/biz/pneuma-fashions-upper-marlboro-3

问题:无法导航到单击“阅读更多”按钮后出现的“关于业务”弹出窗口并提取其中的文本。

目前的尝试:通过谷歌搜索,我找到了有关如何处理警报弹出窗口或窗口弹出窗口的解释。但是代码不起作用。单击Read More按钮时出现的弹出窗口不会导致window_handles发生变化

    import re
    # getting all sections of the page
    result=driver.find_elements_by_tag_name("section")
    About = None
    for sec in result:
    if sec.text.startswith("About the Business"):
        # this pertains only to the About the business section
        
        main_page=driver.current_window_handle
        print(main_page) # Returns the current handle
        
        sec.find_element_by_tag_name("button").click()
        popup=None
        for handle in driver.window_handles: # is an iterable with only one handle 
            # The only handle present is the main_page handle
            print(handle) 
            if handle!=main_page:
                popup = handle
        print(popup) # returns None
        driver.switch_to.window(popup) # Throws error because popup=None

# THE FOLLOWING SECTION IS NOT EXECUTED BECAUSE OF THE ERROR ABOVE
#////////////////////////////////////////////////////

        button_contents=driver.find_elements_by_tag_name("p")
        for b in button_contents:
            print(b.text) # intended to print text contents
        close=driver.find_element_by_tag_name("button")
        close.click()
        driver.switch_to.window(main_page)
        

请帮忙

感谢所有阅读此问题并提供建议和答案的人

【问题讨论】:

    标签: selenium selenium-webdriver dom web-scraping beautifulsoup


    【解决方案1】:

    这是一个自定义弹出窗口,因此您无需切换到它。我建议研究一下获取relative xpath。使用循环导航到您的网址并包含以下代码

    driver.get(your_URL)
    
    readMoreBtnXpath= "//h4[text()='About the Business']/ancestor::section//button"
    aboutTheBusinessSec = "//h4[text()='About the Business']/ancestor::section"
    fromTheBusinessSec = "((//h2[text()='From the business']/parent::div/following-sibling::div//div)[5]/div)[last()]/preceding-sibling::div"
    
    try:
        driver.find_element(By.XPATH, readMoreBtnXpath).click()
        button_contents = driver.find_elements(By.XPATH, fromTheBusinessSec)
        for b in button_contents:
                print(b.text)
    except:
        print(driver.find_element(By.XPATH, aboutTheBusinessSec).text)

    【讨论】:

    【解决方案2】:

    您应该知道的一件事是弹出窗口不会显示在新窗口中。相反,它显示在同一页面本身。以下是从弹出窗口中提取文本的完整代码:

    from selenium import webdriver
    
    driver = webdriver.Chrome()
    
    driver.get('https://www.yelp.com/biz/and-pizza-bethesda-bethesda')
    
    try:
        driver.find_element_by_xpath('//*[@id="wrap"]/div[3]/div/div[4]/div/div/div[2]/div/div/div[1]/div/div[1]/section[5]/div[2]/button').click()
    
        p1 = driver.find_element_by_xpath('//*[@id="modal-portal-container"]/div[2]/div/div/div/div[2]/div/div[2]/div/div[2]/div/div/div[1]/p').text
    
        p2 = driver.find_element_by_xpath('//*[@id="modal-portal-container"]/div[2]/div/div/div/div[2]/div/div[2]/div/div[2]/div/div/div[2]/p[2]').text
    
        print("Specialties --",p1)
        print("History --",p2)
    
    except:
        print('Read more button not found')
    

    输出:

    Specialties -- Award-winning pizza: Named one of Fast Company's "World's Most Innovative Companies" in 2018, third-place in the Washington Post Express's of "Best Fast Casual" in 2018, third place in the Washington City Paper's "Best Gluten-Free Menu" in 2018 and won its "Best Pizza in D.C." in 2017, 11th on TripAdvisor's "Best Fast Casual Restaurants -- United States" in 2018.
    History -- Since 2012, we've built pizza shops with an edge to their craft pies, beverages and shop design, created an environment where ALL of our Tribe can thrive, supported our local communities and now we'll text you back, if you want. Started with a pizza shop. Became a culture. That's &pizza.
    

    编辑:

    由于这不适用于 this 网站,请将第一个 find_element_by_xpath 替换为:

    driver.find_element_by_xpath("//div[@class='lemon--div__373c0__1mboc border-color--default__373c0__3-ifU']/button[.='Read more']").click()
    

    这适用于两个网站。

    【讨论】:

    • 嗨,我尝试了这种使用 xpath 的方法,但问题是这不能很好地概括和在其他业务页面上工作。例如,此业务页面链接 yelp.com/biz/asian-bakery-cafe-rockville> 中断
    • 将第一个find_element_by_xpath替换为driver.find_element_by_xpath("//div[@class='lemon--div__373c0__1mboc border-color--default__373c0__3-ifU']/button[.='Read more']").click()
    • 这绝对可以。我也在我的帖子中添加了相同的内容。看看吧。
    • 先生,更改后的版本在单击按钮方面有效,但无法提取专业和历史的文本(即 p1 和 p2)。我相信这是因为 xpath 在企业的不同网页之间发生变化。是否可以使用类名之类的更通用的方式获取文本数据?
    • 大声笑,我尝试使用``` driver.find_elements_by_class_name("lemon--div__373c0__1mboc padding-t2__373c0__11Iek padding-r2__373c0__28zpp padding-b2__373c0__34gV1 padding-l2__373c0__1 default__373c0__3-ifU")``` 但它不返回任何内容并产生错误
    猜你喜欢
    • 2021-01-11
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2018-12-18
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多