使用 Selenium 滚动对网页进行不完整的抓取答案

【问题标题】：Incomplete Scraping on Webpage using Selenium Scrolling使用 Selenium 滚动对网页进行不完整的抓取
【发布时间】：2021-04-28 19:24:38
【问题描述】：

我正在尝试从使用 JS 加载呈现 HTML 的网站上抓取产品数据。我使用了 Selenium，具有滚动到页面末尾的功能以及重新加载页面的时间，但仍然只能抓取网站中排名前 8 的产品。

这是我的代码：

url = 'https://www.faire.com/retailer/r_9vkjixqbpq/category/Beauty%20&%20Wellness/subcategory/Bath%20&%20Body?filters=sorting%3Afeatured'
wd.get(url)

last_height = wd.execute_script("return document.documentElement.scrollHeight")
time.sleep(3)


while True:
    # Scroll down to bottom
  wd.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
  time.sleep(20)

    # Calculate new scroll height and compare with last scroll height
  new_height = wd.execute_script("return document.body.scrollHeight")
  if new_height == last_height:
      break
  last_height = new_height

html = wd.page_source
soup = BS(html, 'lxml')
listings = soup.find('div', {'class': 'MarketplaceProductList__Wrapper-sc-3mfb9g-0 fWFKvm'}).findAll('div', {'class':'MarketplaceProductList__TileWrapper-sc-3mfb9g-2 fQbbTY'})

for item in listings: 
  product_name = item.find('span', {"class":"FallbackHandler__ContentParentWrapper-sk18il-2 jHwpkw"}).get_text(strip=True)
  print(product_name)

如何从页面上的每个产品中提取信息？谢谢！

【问题讨论】：

标签： javascript html selenium web-scraping beautifulsoup

【解决方案1】：

问题是您必须加载部分页面然后到达页面末尾才能获得整个页面源。所以我把页面分成了 3 个部分，滚动了 2/3 的页面。你也可以滚动到页面的末尾，但是为什么要浪费时间和内存。

所有的抓取几乎都可以用Selenium 本身完成，因为与BS 相比，它提供了更清晰的输出，但是如果你将re 模块与BS 一起使用，你可以让它看起来也很漂亮（我留下它任你选择）！

使用硒

url = 'https://www.faire.com/retailer/r_9vkjixqbpq/category/Beauty%20&%20Wellness/subcategory/Bath%20&%20Body?filters=sorting%3Afeatured'
wd.get(url)
sleep(3)
height = wd.execute_script("return document.documentElement.scrollHeight;")

#first scroll to load 1/3 rd of the page
wd.execute_script('window.scrollTo(0, arguments[0]);',height/3)
sleep(3)

#scroll to the next 1/3 rd of the page
wd.execute_script('window.scrollTo(0, arguments[0]);',height/3 + height/3)
sleep(3)

# by now all the scripts would've been loaded

#span tag has all the details you'll need
details = [i.text for i in wd.find_elements_by_tag_name('span') if len(i.text) > 0 and '\n' in i.text]
wd.quit()

for i in details:
    print(i)
    print()

输出

使用美丽的汤

url = 'https://www.faire.com/retailer/r_9vkjixqbpq/category/Beauty%20&%20Wellness/subcategory/Bath%20&%20Body?filters=sorting%3Afeatured'
wd.get(url)
sleep(3)
height = wd.execute_script("return document.documentElement.scrollHeight;")

#first scroll to load 1/3 rd of the page
wd.execute_script('window.scrollTo(0, arguments[0]);',height/3)
sleep(3)

#scroll to the next 1/3 rd of the page
wd.execute_script('window.scrollTo(0, arguments[0]);',height/3 + height/3)
sleep(3)

#by now all the scripts would've been loaded

soup = BS(wd.page_source, 'lxml')
wd.quit()
#span tag has all the details you'll need
details = [i.text for i in soup.find_all('span') if len(i.text) >20 and 'MSRP $' in i.text]

#removing duplicates
details = list(dict.fromkeys(details))

for i in details:
    print(i)
    print()

输出

【讨论】：