BeautifulSoup 不会在动态页面中提取评论标签

Question

我需要什么：统计所有语言的 Chrome 商店 中扩展程序下的评论数量。 我做了什么： 尝试用 BeautifulSoup 提取某个标签。我重新搜索了页面的 html-code 并找到了一个评论标签：

试过这段代码：

    from bs4 import BeautifulSoup
    import requests
    page = requests.get(url)
    soup = BeautifulSoup(page.content, 'html5lib')
    comments = soup.find_all('div', class_ = 'ba-bc-Xb ba-ua-zl-Xb')

但print(comments) 显示数组为空。
我现在被困住了，我发现我需要进一步处理两个问题：

1. 如何应对选择语言按钮？如果默认情况下仅选择一种语言，如何计算 所有 种语言的评论。

2. 评论存储在不同的选项卡中。我读到了动态提取它，但没有得到一点。

Answer 1

您可以使用 selenium 执行任务并等待页面更改并从 PaginationMessage 中提取评论计数。用几个链接测试。您可能需要为没有评论的项目添加错误处理。似乎还有一些 POST XHR 活动会产生您可能希望探索的审查 JSON 字符串。

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains

url = 'https://chrome.google.com/webstore/detail/evernote-web-clipper/pioclpoplcdbaefihamjohnefbikjilc?hl=en/'
#url = 'https://chrome.google.com/webstore/detail/https-everywhere/gcbommkclmclpchllfjekcdonpmejbdp?hl=en/'
d = webdriver.Chrome()
d.get(url)
WebDriverWait(d, 5).until(EC.visibility_of_element_located((By.ID, ':21'))).click()
ActionChains(d).click_and_hold(WebDriverWait(d, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.h-z-Ba-ca.ga-dd-Va.g-aa-ca')))).perform()
languageSelection = WebDriverWait(d, 5).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '.g-aa-ca-ma-x-L')))
languageSelection[1].click()

s= WebDriverWait(d, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.Aa.dc-tf + span'))).text
print(s.split()[-1])
d.quit()

Answer 2

试试这个

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
import time

driver = webdriver.Chrome()

driver.get('https://chrome.google.com/webstore/detail/evernote-web-clipper/pioclpoplcdbaefihamjohnefbikjilc?hl=en')
wait = WebDriverWait(driver, 5)
wait.until(EC.visibility_of_element_located((By.ID, ':21'))).click()
wait.until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, '.h-z-Ba-ca.ga-dd-Va.g-aa-ca'))
    ).click()

english = driver.find_element_by_xpath('//div[@class="ah-mg-j"]/span').text
print('English: ' + english.split()[-1])

wait.until(
        EC.visibility_of_element_located((By.XPATH, '//div[@class="g-aa-ca-ma-x-L" and text() = "All languages"]'))
    ).click()
wait.until_not(EC.text_to_be_present_in_element((By.XPATH, '//div[@class="ah-mg-j"]/span'), english))
time.sleep(2)

AllCount = driver.find_element_by_xpath('//div[@class="ah-mg-j"]/span').text
print('All languages: ' + AllCount.split()[-1])
driver.close()