这些视图由 javascript 显示,并且总是很有可能在“网络”选项卡中找到与该请求相关的 XHR 请求。
我发现这个网站以 JSON 格式返回这些视图。 https://collector.sikayetvar.com/complaints/view-count?complaints=14793971%2C14792534%2C14791327%2C14789107%2C14787731%2C14787333%2C14787160%2C14784871%2C14784832%2C14784823%2C14784535%2C14783714%2C14762547%2C14783312%2C14783264%2C14782822
这些投诉是每个投诉的 id,用逗号分隔,它们可以在 https://www.sikayetvar.com/sikayetler?brand=bosch 的源中找到,我们只需将逗号替换为 '%2C' 并制作完整的字符串
米
import requests
import lxml
from bs4 import BeautifulSoup
headers = {'user-agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.125'}
brand = 'bosch'
site = f'https://www.sikayetvar.com/sikayetler?brand={brand}'
req = requests.get(site, headers=headers)
soup = BeautifulSoup(req.content, 'lxml')
complaints = soup.find_all('span', {'class':'js-view-count'})
complaints = '%2C'.join(i['data-id'] for i in complaints)
print(complaints)
site = f'https://collector.sikayetvar.com/complaints/view-count?complaints={complaints}'
r = requests.get(site)
print(r.json())
输出:-
[{'id': 14793971, 'viewCount': 43, 'uniqueViewCount': 29}, {'id': 14792534,'viewCount': 501, 'uniqueViewCount': 263}, {'id': 14791327, 'viewCount': 218,'uniqueViewCount': 134}, {'id': 14789107, 'viewCount': 492, 'uniqueViewCount': 286},{'id': 14787731, 'viewCount': 581, 'uniqueViewCount': 342}, {'id': 14787333, 'viewCount':506, 'uniqueViewCount': 287}, {'id': 14787160, 'viewCount': 428,'uniqueViewCount': 249}, {'id': 14784871, 'viewCount': 25, 'uniqueViewCount': 16},{'id': 14784832, 'viewCount': 29, 'uniqueViewCount': 15}, {'id': 14784823, 'viewCount':20, 'uniqueViewCount': 8}, {'id': 14784535, 'viewCount': 99, 'uniqueViewCount': 57}, {'id':14783714, 'viewCount': 510, 'uniqueViewCount': 280}, {'id': 14762547, 'viewCount':139, 'uniqueViewCount': 85}, {'id': 14783312, 'viewCount': 142,'uniqueViewCount': 78}, {'id': 14783264, 'viewCount': 190, 'uniqueViewCount': 114},{'id': 14782822, 'viewCount': 216, 'uniqueViewCount': 120}]
另一种方法是进行每一次投诉,这将花费大量时间,而且如果您继续抓取,网站可能会阻止您