我正在尝试从网页中抓取一些数据并不断收到 selenium.common.exceptions.TimeoutException 错误

Question

我已经用请求 + 美丽的汤从第一页抓取了我想要的元素，并尝试使用 selenium 单击一个按钮，该按钮加载另一个页面，其中包含我想要抓取的其余数据。新的 selenium 窗口打开第一页，然后打开第二页，但总是返回某种 selenium.common.exceptions 错误。 File "c:\Users\ArkPr\yelp\yelp_learner.py", line 122, in <module> WebDriverWait(driver, 10).until( File "C:\Users\ArkPr\AppData\Roaming\Python\Python39\site-packages\selenium\webdriver\support\wait.py", line 80, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: 我四处搜索并尝试使用隐式和显式等待 time.sleep()，使用 find_element_by_class_name 和 xpath，使用请求获取数据，我不知道下一步该做什么。

url = "https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=London&ns=1" 试图从该站点抓取数据

next_page = driver.find_element_by_class_name("css-ac8spe")

try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "css-ac8spe"))
    )
    # element.clear()
    element.click()

except TimeoutException as u:
    print(u)

        # if EC.presence_of_element_located((By.CLASS_NAME, "css-ac8spe")) == True:
# next_page.click()
            # next_page.back()
for li in ul:

    #driver.implicitly_wait(3)

    

    WebDriverWait(driver, 10).until(
    EC.visibility_of_all_elements_located((By.CLASS_NAME, 'stickySidebar__373c0__3MPss border-color--default__373c0__r305k')))
    # EC.presence_of_element_located((By.CLASS_NAME, 'stickySidebar__373c0__3MPss border-color--default__373c0__r305k')))
    
    
    
    sidebar = driver.find_element_by_class_name('stickySidebar__373c0__3MPss border-color--default__373c0__r305k')
    

    
    if sidebar != None:


        for side in sidebar:
            print(side)
            if side != None:
                try: 


                    address = side.find("p", {"class": " css-chtywg"})
                    # address = side.find_element_by_class_name("css-chtywg")
                    print(address.text)

                except TimeoutException as t:
                    print(t)```

Answer 1

您需要在每个帖子上单击更多，然后它会导航到新页面，提取地址并返回原始页面。

driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.implicitly_wait(50)
driver.get("https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=London&ns=1")
wait = WebDriverWait(driver, 50)
length = len(wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//li[contains(@class,'border-color')]/div[contains(@class, 'container')]"))))
j = 0
for i in range(length):
  elements = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//li[contains(@class,'border-color')]/div[contains(@class, 'container')]")))
  ActionChains(driver).move_to_element(elements[j]).perform()
  time.sleep(1)
  elements[j].find_element(By.XPATH, ".//descendant::a[text()='more']").click()
  time.sleep(1)
  print(wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "p.css-chtywg"))).text)
  driver.execute_script("window.history.go(-1)")
  time.sleep(2)
  j = j +1

Answer 2

这段代码对我有用。

driver.implicitly_wait(10)
driver.get("https://www.yelp.co.uk/search?find_desc=Restaurants&find_loc=London&ns=1")
res_leng = len(driver.find_elements_by_xpath("//li/div[contains(@class,'container__09f24')]"))
for i in range(res_leng):
    resturants = driver.find_elements_by_xpath("//li/div[contains(@class,'container__09f24')]")
    resturants[i].find_element_by_xpath(".//a[text()='more']").click()
    name = driver.find_element_by_tag_name("h1").text
    address = driver.find_element_by_xpath("//p[@class=' css-chtywg']").text
    print("{} : {}".format(name,address))
    time.sleep(2)
    driver.back()