网页抓取评论 -Flipkart答案

【问题标题】：Web Scraping reviews -Flipkart网页抓取评论 -Flipkart
【发布时间】：2018-09-24 18:35:12
【问题描述】：

我正在尝试删除产品的整个评论（剩余一半的评论在单击阅读更多后显示。但我仍然无法这样做。它没有显示评论的全部内容，这会被显示点击阅读更多选项后。下面是代码，点击阅读更多选项并从网站获取数据

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

response = requests.get("https://www.flipkart.com/poco-f1-graphite-black-64-gb/product-reviews/itmf8fyjyssnt25c?page=2&pid=MOBF85V7A6PXETAX")    
data = BeautifulSoup(response.content, 'lxml')
chromepath = r"C:\Users\Mohammed\Downloads\chromedriver.exe"
driver=webdriver.Chrome(chromepath)
driver.get("https://www.flipkart.com/poco-f1-graphite-black-64-gb/product-reviews/itmf8fyjyssnt25c?page=2&pid=MOBF85V7A6PXETAX")
d = driver.find_element_by_class_name("_1EPkIx")
d.click()
title = data.find_all("p",{"class" : "_2xg6Ul"})
text1 = data.find_all("div",{"class" : "qwjRop"})
name = data.find_all("p",{"class" : "_3LYOAd _3sxSiS"})
for t2, t , t1 in zip(title,text1,name) :
    print(t2.text,'\n',t.text,'\n',t1.text)

【问题讨论】：

标签： python-3.x web-scraping beautifulsoup python-requests selenium-chromedriver

【解决方案1】：

要获得完整的评论，有必要点击那些READ MORE 按钮来解开其余部分。由于您已经将selenium 与BeautifulSoup 结合使用，因此我已修改脚本以遵循逻辑。该脚本将首先单击那些READ MORE 按钮。完成后，它将从那里解析所有titles 和reviews。您现在可以从多个页面（最多 4 个页面）获取 titles 和 reviews。

import time
from bs4 import BeautifulSoup
from selenium import webdriver

link = "https://www.flipkart.com/poco-f1-graphite-black-64-gb/product-reviews/itmf8fyjyssnt25c?page={}&pid=MOBF85V7A6PXETAX"

driver = webdriver.Chrome() #If necessary, define the chrome path explicitly
for page_num in range(1,5):
    driver.get(link.format(page_num))
    [item.click() for item in driver.find_elements_by_class_name("_1EPkIx")]
    time.sleep(1)
    soup = BeautifulSoup(driver.page_source, 'lxml')
    for items in soup.select("._3DCdKt"):
        title = items.select_one("p._2xg6Ul").text
        review = ' '.join(items.select_one(".qwjRop div:nth-of-type(2)").text.split())
        print(f'{title}\n{review}\n')

driver.quit()

【讨论】：