Python bs4 .find 未检测到文章

Question

我正在尝试获取产品的名称，但是当它到达赞助产品时，它会返回 None。这是我的代码；

    next_page_url = 'https://www.jumia.com.ng/catalog/?q=oraimo&shipped_from=country_local&page=1#catalog-listing'
    result_nextpage = requests.get(next_page_url, headers=headers).text # headers are generated from default python 'fake_headers' module.
    doc_nextpage = BeautifulSoup(result_nextpage, 'lxml') # I also tried other parsers
    divs = doc_nextpage.find('div', class_='-paxs row _no-g _4cl-3cm-shs')
    result_articles = divs.select('h3.name')
    for i in result_articles:
        print(i.string)

结果；

Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds
Oraimo 27000mAh Massive Power Charing Bank Traveller 3 Byte
Oraimo OPB-P116DN 10000 Mah Power-Bank Dual Fast Charging
Oraimo FreePods3 True Wireless Stereo Earbuds IPX5 & Sweat Proof
Oraimo Smart Watch 1.69'' IPS Screen IP68 Waterproof
Oraimo FreePods-2 2Baba-version True Wireless Earbuds
Oraimo Silver Edition Smart Watch 1.69'' IPS Screen IP68 Waterproof
Oraimo Charger UKDualUSB OCW-U63D White
Oraimo Portable Wireless Speaker Subwoofer Outdoor Sound Box
Oraimo Charger Oraimo UKDualUSB OCW-U81F White
Oraimo Power Oraimo Bank OPB-P206DN 20KmAh
Oraimo SoundPro Wireless Speaker Muti-Model Music Play
Oraimo Tempo-W3 Smart Watch Health Monitor IP67 Waterproof
Oraimo Car Charger Oraimo OCC-21DML Black
Oraimo SoundPro-2C 10W Portable Wireless Bluetooth Speaker
Oraimo Necklace 5C Neckband Wireless Earphone
Oraimo COMPACT 10000mAh Ultra Slim Fast Charging Power Bank
Oraimo 10000mAh OPTIMIZED SLIM Power-bank With LED Light
Oraimo Mermaid Half In-ear Earphone With Mic
Oraimo Necklace 3 Lite Neckband BT 5.0 Wireless Earphone
Oraimo Senior BT5.0 Single Wireless Bluetooth Headsets
Oraimo True Wireless Bluetooth Earbuds- Freepods 2
Oraimo FreePods-2 2Baba-version True Wireless Earbuds
Oraimo Air-Buds-2S Super Bass Wireless Stereo Earbuds
Oraimo 20000MAH Powerbank -long Lasting PowerBank
Oraimo Bluetooth Wireless SOUNDBAR SPEAKER
Oraimo Shark-2 BT5.0 In-Ear Wireless Bluetooth Headphones
Oraimo BoomPop Over-Ear Bluetooth Wireless Headphone
Oraimo  20000MAH Powerbank -long Lasting Power For Days
Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud
Oraimo 2021 Latest Edition Smart Function Waterproof Smart Watch
Oraimo OCW-U36S Efficient And Durable USB Charger - Black
Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud-white
Oraimo 10000MAh Ultimate Slim Power Bank - Black
Oraimo 20000MAH Powerbank - Power For Days
Oraimo 10000mAh Ultra Slim Fast Charging Power Bank
Oraimo 2020 Edition Tempo S - OSW-11 Multi Function Smart Watch
Oraimo SOLID 27000mAh Massive Powerbank OPB-P271D Traveller 3 Byte
Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds
Oraimo Tempo-S IP67 Waterproof Smart Watch WITH AMAZING FUNCTIONS
None
None
None
None
None
None
None
None

文章标签 41-48 是赞助产品，产品名称从浏览器的检查元素中显示，但 bs4 没有检测到它，但它检测到其他非赞助产品。 请帮忙。

Answer 1

注意 首先，看看你的汤/doc_nextpage - 这是你处理数据的真相。

会发生什么？

在您的 doc_nextpage 中，您的赞助产品的 html 是空的，这就是您获得这些 None 的原因。

它们是空的，因为它们将由网站动态提供，requests 无法处理。它不是浏览器，它会解释/操作数据。

如何解决？

一种选择是使用selenium 模拟浏览器行为，并让page_source 使用beautifulsoup 或selenium 本身对其进行处理。

示例（硒 4）

from bs4 import BeautifulSoup 
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
service = ChromeService(executable_path='ENTER YOUR PATH TO CHROMEDRIVER')
driver = webdriver.Chrome(service=service, options=options)
driver.get('https://www.jumia.com.ng/catalog/?q=oraimo&shipped_from=country_local&page=1#catalog-listing')

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '[data-list="sponsored"]')))

soup = BeautifulSoup(driver.page_source, 'lxml')

print([x.text for x in soup.select('article h3.name')])

driver.close()

输出

['Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds',
 'Oraimo 27000mAh Massive Power Charing Bank Traveller 3 Byte',
 'Oraimo OPB-P116DN 10000 Mah Power-Bank Dual Fast Charging',
 'Oraimo FreePods3 True Wireless Stereo Earbuds IPX5 & Sweat Proof',
 "Oraimo Smart Watch 1.69'' IPS Screen IP68 Waterproof",
 'Oraimo FreePods-2 2Baba-version True Wireless Earbuds',
 "Oraimo Silver Edition Smart Watch 1.69'' IPS Screen IP68 Waterproof",
 'Oraimo Charger UKDualUSB OCW-U63D White',
 'Oraimo Portable Wireless Speaker Subwoofer Outdoor Sound Box',
 'Oraimo Charger Oraimo UKDualUSB OCW-U81F White',
 'Oraimo Portable Source 10000mAh Po Wer Ba Nk Oraimo OPB-P110D',
 'Oraimo Power Oraimo Bank OPB-P206DN 20KmAh',
 'Oraimo SoundPro Wireless Speaker Muti-Model Music Play',
 'Oraimo Tempo-W3 Smart Watch Health Monitor IP67 Waterproof',
 'Oraimo Car Charger Oraimo OCC-21DML Black',
 'Oraimo SoundPro-2C 10W Portable Wireless Bluetooth Speaker',
 'Oraimo Necklace 5C Neckband Wireless Earphone',
 'Oraimo 10000mAh OPTIMIZED SLIM Power-bank With LED Light',
 'Oraimo COMPACT 10000mAh Ultra Slim Power Fast Charging Bank',
 'Oraimo Mermaid Half In-ear Earphone With Mic',
 'Oraimo Necklace 3 Lite Neckband BT 5.0 Wireless Earphone',
 'Oraimo Senior BT5.0 Single Wireless Bluetooth Headsets',
 'Oraimo True Wireless Bluetooth Earbuds- Freepods 2',
 'Oraimo Pilot 20000mAh 2.1A Fast  Power Charging Bank',
 'Oraimo FreePods-2 2Baba-version True Wireless Earbuds',
 'Oraimo Air-Buds-2S Super Bass Wireless Stereo Earbuds',
 'Oraimo 20000MAH Powerbank -long Lasting PowerBank',
 'Oraimo Bluetooth Wireless SOUNDBAR SPEAKER',
 'Oraimo Shark-2 BT5.0 In-Ear Wireless Bluetooth Headphones',
 'Oraimo BoomPop Over-Ear Bluetooth Wireless Headphone',
 'Oraimo  20000MAH Powerbank -long Lasting Power For Days',
 'Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud',
 'Oraimo OCW-U36S Efficient And Durable USB Charger - Black',
 'Oraimo 2021 Latest Edition Smart Function Waterproof Smart Watch',
 'Oraimo OCW-U36S Efficient And Durable USB Charger - Black',
 'Oraimo Firefly-2 5.0V/2.1A Dual USB Fast Wall Charger',
 'Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud-white',
 'Oraimo 10000MAh Ultimate Slim Power Bank - Black',
 'Oraimo SOLID 27000mAh Massive Powerbank OPB-P271D Traveller 3 Byte',
 'Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds',
 'Oraimo Massive 27000mAh Travellers 3 Byte OPB-P271D Power Bank',
 'Oraimo 1.69" IPS Screen IP68 Waterproof Smart Watch Pro-Silver',
 'Oraimo Tempo-S IP67 Waterproof Smart Watch WITH AMAZING FUNCTIONS',
 'Oraimo FreePods-3 E104D 2Baba Edition BT 5.2 Wireless Earbuds',
 'Oraimo Tempo-S IP67 Waterproof Smart Watch',
 'Oraimo 2020 Edition Tempo S - OSW-11 Multi Function Smart Watch',
 'Oraimo 10000mAh Ultra Slim Fast Charging Power Bank',
 'Oraimo 20000MAH Powerbank - Power For Days']