在父 div beautifulsoup 中打印文本

Question

我正在尝试从中获取每个产品的名称和价格 https://www.daraz.pk/catalog/?q=risk 但什么也没显示。

containers = page_soup.find_all("div",{"class":"c2p6A5"})

for container in containers:
  pname = container.findAll("div", {"class": "c29Vt5"})
  name = pname[0].text
  price1 = container.findAll("span", {"class": "c29VZV"})
  price = price1[0].text
  print(name)
  print(price)

Answer 1

页面中有JSON数据，你可以使用beautifulsoup在<script>标签中获取，但我认为不需要，因为你可以直接使用json和re获取它

import requests, json, re

html = requests.get('https://.......').text

jsonStr = re.search(r'window.pageData=(.*?)</script>', html).group(1)
jsonObject = json.loads(jsonStr)

for item in jsonObject['mods']['listItems']:
    print(item['name'])
    print(item['price'])

Answer 2

如果页面是动态的，Selenium 应该处理好这个

from bs4 import BeautifulSoup
import requests
from selenium import webdriver

browser = webdriver.Chrome()
browser.get('https://www.daraz.pk/catalog/?q=risk')

r = browser.page_source
page_soup = bs4.BeautifulSoup(r,'html.parser')

containers = page_soup.find_all("div",{"class":"c2p6A5"})

for container in containers:
  pname = container.findAll("div", {"class": "c29Vt5"})
  name = pname[0].text
  price1 = container.findAll("span", {"class": "c29VZV"})
  price = price1[0].text
  print(name)
  print(price)

browser.close() 

输出：

Risk Strategy Game
Rs. 5,900
Risk Classic Board Game
Rs. 945
RISK - The Game of Global Domination
Rs. 1,295
Risk Board Game
Rs. 1,950
Risk Board Game - Yellow
Rs. 3,184
Risk Board Game - Yellow
Rs. 1,814
Risk Board Game - Yellow
Rs. 2,086
Risk Board Game - The Game of Global Domination
Rs. 975
...

Answer 3

我错了。用于计算页数的信息存在于 json 中，因此您可以获得所有结果。不需要正则表达式，因为您可以提取相关的脚本标签。此外，您可以在循环中创建页面 url。

import requests
from bs4 import BeautifulSoup
import json
import math

def getNameAndPrice(url):
    res = requests.get(url)
    soup = BeautifulSoup(res.content,'lxml')
    data = json.loads(soup.select('script')[2].text.strip('window.pageData='))
    if url == startingPage:
        resultCount = int(data['mainInfo']['totalResults'])
        resultsPerPage = int(data['mainInfo']['pageSize'])
        numPages = math.ceil(resultCount/resultsPerPage)
    result = [[item['name'],item['price']] for item in data['mods']['listItems']]   
    return result

resultCount = 0
resultsPerPage = 0
numPages = 0
link = "https://www.daraz.pk/catalog/?page={}&q=risk"
startingPage = "https://www.daraz.pk/catalog/?page=1&q=risk"
results = []
results.append(getNameAndPrice(startingPage))

for links in [link.format(page) for page in range(2,numPages + 1)]: 
    results.append(getNameAndPrice(links))

Answer 4

向像我这样的新手参考 JSON 答案。 您可以使用 Selenium 导航到搜索结果页面，如下所示：

PS：非常感谢@ewwink。你拯救了我的一天！

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time #time delay when load web
import requests, json, re

keyword = 'fan'

opt = webdriver.ChromeOptions()
opt.add_argument('headless')
driver = webdriver.Chrome(options = opt)

# driver = webdriver.Chrome()

url = 'https://www.lazada.co.th/'
driver.get(url)

search = driver.find_element_by_name('q')
search.send_keys(keyword)
search.send_keys(Keys.RETURN)

time.sleep(3) #wait for web load for 3 secs

page_html = driver.page_source #Selenium way of page_html = webopen.read() for BS

driver.close()

jsonStr = re.search(r'window.pageData=(.*?)</script>', page_html).group(1)
jsonObject = json.loads(jsonStr)

for item in jsonObject['mods']['listItems']:
    print(item['name'])
    print(item['sellerName'])