未找到 Bs4 标签

Question

我正在尝试为目标网站制作网络爬虫。我正在使用下面的代码，它可以很好地查找产品信息，但查找价格不会返回。我不明白为什么找不到价格，但其他所有标签都可以。提前感谢您的帮助！

网址：https://www.target.com/p/rold-gold-fat-free-tiny-twists-pretzels-16oz/-/A-13325504#lnk=sametab

for data in soup.findAll('span',{'data-test':'product-price'}):
    global price
    price = str(data.text)
    print(price)

Answer 1

我认为问题在于它在捕获后加载，就像您打印汤一样，由于它是由 javascript 加载的，所以您不会在其中找到价格，我对此很陌生，所以不要相信我的话它，但我认为解决方案是使用 selenium 或 scrapy &scrapy-playwright 如果我有空并且没有其他人帮助你，我可能会为它做一个刮刀。

编辑 我已经尝试为该页面做一个简单的scrapy-playwright scraper 似乎工作正常我可能应该选择 wait_for_selector 而不是 timeout 但不明白为剧作家做这件事的正确形式所以只是做了 timeout 并得到了结果工作正常 名称、价格 Rold Gold 无脂肪小扭曲椒盐脆饼 - 16 盎司，3.19 美元

'''

import scrapy
from scrapy_playwright.page import PageCoroutine


class TargetSpider(scrapy.Spider):
    name = 'target'

    def start_requests(self):
        yield scrapy.Request(
            url='https://www.target.com/p/rold-gold-fat-free-tiny-twists-pretzels-16oz/-/A-13325504',
            meta=dict(
                playwright=True,
                playwright_include_page=True,
                playwright_page_coroutines=[
                    PageCoroutine('wait_for_timeout', 3000)]
            ),
            dont_filter=True,
            callback=self.parse_c)

        pass

    async def parse_c(self, response):
        yield {

            'name' : response.xpath("/html/body/div[1]/div/div[4]/div/div[1]/div[2]/h1/span//text()").get(),
            'price' : response.css('span.web-migration-tof__PriceFontSize-sc-14z8sos-16.iyRiQZ::text').get()
        }
'''

也做了硒

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

driver = webdriver.Firefox()
url = "https://www.target.com/p/rold-gold-fat-free-tiny-twists-pretzels-16oz/-/A-13325504"
driver.get(url)
try:

    elem = WebDriverWait(driver, 30).until(
        EC.presence_of_element_located((By.XPATH, "//span[@data-test='product-price']"))
    )

    name = driver.find_elements(By.XPATH, "/html/body/div[1]/div/div[4]/div/div[1]/div[2]/h1/span")
    price = driver.find_elements(By.XPATH, "//span[@data-test='product-price']")
    for n in name:
        print(n.text)
    for i in price:
        print(i.text)
    time.sleep(5)
finally:
    driver.quit()

希望这有助于萌芽..