使用 BeautifulSoup 抓取 shopee.co.id 时得到一个空列表答案

【问题标题】：Got an empty list when scraping shopee.co.id using BeautifulSoup使用 BeautifulSoup 抓取 shopee.co.id 时得到一个空列表
【发布时间】：2020-07-19 20:31:37
【问题描述】：

我试图使用 BeautifulSoup 包在 https://shopee.co.id/top_products 上获取顶级产品的名称和销售额。但无论我尝试什么，它总是返回一个空的产品列表。你能解释一下幕后发生的事情吗？这是我的代码：

import requests
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}

shopee_url = 'https://shopee.co.id/top_products'
response = requests.get(shopee_url, headers=headers)
soup = bs(response.text, "html.parser")

products = soup.find_all("._3S8sOC _2QfAXF")
print(type(products))
print(products)

感谢您阅读本文。我已经阅读了一些类似的答案，但它们似乎不适用于这里。

【问题讨论】：

我发现了一个类似的线程，这个人试图解析一个使用 API 获取数据的网站。这也可能是这种情况，但我在使用网络工具的请求中没有发现任何明显的东西。我不是专家，所以我很容易遗漏一些东西……您是否尝试过解析同一页面的任何其他元素？
我发现唯一可以证明 API 案例的是这个文件：deo.shopeemobile.com/shopee/shopee-pcmall-live-sg/assets/… 有趣的是，在编辑器中格式化代码并查看第 181 行：D

标签： python beautifulsoup web-crawler

【解决方案1】：

如果您愿意使用 selenium，您可以使用 selenium 提出请求：

from selenium import webdriver
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}
driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')
shopee_url = 'https://shopee.co.id/top_products'
driver.get(shopee_url)
driver.implicitly_wait(15)
response = driver.page_source
driver.close()
soup = bs(response, "html.parser")

products = soup.select(".shopee-item-card__lower-padding")

#print(products)

for product in products:
    name = product.select_one('.shopee-item-card__text-name span')
    print(name.get_text(strip=True))
    price = product.select_one('.shopee-item-card__current-price')
    print(price.text)

请注意，您需要安装 Firefox、geckodriver 和 selenium。

输出：

Kuota Telkomsel Cek Dadakan
Rp300
Aigo 1GB 5Hari
Rp7.375
Paket Data Tri Mix Small 5GB
Rp4.500
Paket Data Tri Mix Small 2.75GB
Rp9.500
*PROMO MURAH* SURPRISE DEAL PAKET DATA INTERNET KUOTA TELKOMSEL SIMPATI AS LOOP OMG 20GB 30GB 50GB
Rp1.000
Kuota Telkomsel Cek Dadakan
Rp500
Paket Data Telkomsel 12GB
Rp98.000
Paket Data Tri AON Unlimited + 6GB
Rp58.000
KUOTA INTERNET PAKET DATA INTERNET TELKOMSEL KUOTA DATA FLASH Kartu Simpati As Loop (KUOTA MURAH)
Rp1.000
[MURAH] PAKET DATA TELKOMSEL ∣ KUOTA INTERNET ∣ PAKET DATA ∣ SIMPATI  ∣ AS  ∣ LOOOP
Rp1.000
Paket Data Tri AddOn 12GB
Rp48.000
Paket Data  Indosat Freedom U 7GB+20GB Apps / 30 Hari
Rp66.500
PAKET DATA TELKOMSEL ISI ULANG SIMPATI 14GB/27GB/37GB - AS 6,5GB/10GB/17GB/32GB - LOOP 3GB/8GB/15GB
Rp38.000
Paket Data Indosat Freedom Internet 4GB / 30 Hari
Rp24.000
Paket Data Indosat Freedom U 1GB+4.5GB Apps / 30 Hari
Rp24.000
KUOTA TELKOMSEL paket kuota data internet telkomsel termurah 52gb injek injeck kuota  promo
Rp1.000
Kuota Telkomsel Simpati 52GB 50GB 24jam Langsung Masuk
Rp1.000
Paket Data Telkomsel 50GB
Rp195.000
Paket Data Indosat Freedom Internet 10GB /30 Hari
Rp48.000
KUOTA TELKOMSEL paket data internet telkomsel kuota internet TELKOMSEL  termurah simpati as loop
Rp1.000
Paket Data Telkomsel Data 25.000
Rp23.500
VOCHER VOUCHER KUOTA AXIS AIGO 1GB 5HARI NASIONAL
Rp7.390
PAKET DATA KUOTA INDOSAT [DISKON 15%] TERMURAH
Rp1.000
VOUCHER Axis 1gb 5 hari 24 Jam KUOTA 1 GB
Rp7.500
Paket Data Indosat Freedom Combo 14GB, 30 Hari
Rp48.500
[PROMO] KUOTA BESAR PAKET OMG TELKOMSEL MURAH
Rp1.000
Paket Data Indosat Freedom Combo 8GB/ 30 Hari
Rp33.500
TELKOMSEL PEMBAYARAN RESELLER/ AGEN atau Penambahan Selisih Harga Produk ThalhahCell
Rp1.000
Aigo 3GB 15Hari
Rp18.500
Paket Data Indosat Freedom U 3GB+15GB Apps / 30 Hari
Rp49.900

【讨论】：