在电子商务网站上抓取嵌套元素

Question

当我访问特定的产品页面时，我试图用 Selenium 从 Target 的网站上抓取产品 img url，但没有返回任何内容。

这是我的那部分代码：

# ADD THE IMAGE URL
    j = 0
    found = False
    while(j < 5 and not found):
        try:
            img_panel = driver.find_element_by_class_name('slideDeckPicture')
            img_panel = img_panel.find_element_by_tag_name('img')
            img_name = img.get_attribute('alt')
            img_url = img_panel.get_attribute('src')

            # img_urls.append(img_url)
            line += ',"' + img_url + '"'
            found = True
            break
        # if it can't find the image, it probably hasn't loaded. wait and try again.
        except:
            j += 1
            time.sleep(4)
            # img_urls.append('NO URL')
            # pass
    # if we've tried 5 times add no url
    if found == False:
        line += ',NO IMG URL'

HTML 截图：

Link to example product

Answer 1

url 列表包含您要查找的 url：

url = "https://www.target.com/p/revolution-beauty-conceal-define-concealer-0-11-fl-oz/-/A-82003638?preselect=81551727#lnk=sametab"
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0"}
resp = rq.get(url, headers=headers)
soup = bs(resp.content)

divs_img = soup.find_all("div", attrs={"data-test": "product-image"})[0]
urls = [i["src"] for i in divs_img.find_all("img") if i["src"].startswith("https")]