是否可以通过请求和 BeautifulSoup 获得 eBay 项目描述？

Question

我正在尝试从 eBay 上的鞋子收集数据。对于每个项目，我想收集所有数据，包括自定义描述以建立数据库。我已经收集了所有方面的信息，例如价格、运输标题等，以及请求和 BS4。不幸的是，唯一缺少的是自定义项目描述。

这似乎是一个事件 html，它在浏览器中是自动加载的，但不是请求和 BS4。我更喜欢用 requests 和 BS4 来做，因为脚本几乎已经准备好了，例如 Selenium 的抓取速度要慢得多。我正在处理的示例如下：

from bs4 import BeautifulSoup as soup
import requests

source=requests.get("https://www.ebay.com/itm/SIGNED-2000-NIKE-AIR-JORDAN-1-HIGH-BANNED-1985-ROOKIE-RETRO-SHOES-AUTOGRAPH-UDA/392861887827?hash=item5b7864ad53:g:njcAAOSw9rpfALFX")
Nike_shoe = soup(source.text, "lxml")

我试图过滤的描述部分包含以下摘录： 这可以在 eBay 页面上找到。此描述是以下 HTML 结构的一部分：

当我浏览 Nike_shoe 汤时，此文本不存在。我尝试将 source.text 解析为 lxml、html.parser、html5lib 和 xml。

我也尝试过使用 Requests-HTML 包，它应该有完整的 JavaScript 支持：

from requests_html import HTMLSession
session = HTMLSession()
source = session.get('https://www.ebay.com/itm/SIGNED-2000-NIKE-AIR-JORDAN-1-HIGH-BANNED-1985-ROOKIE-RETRO-SHOES-AUTOGRAPH-UDA/392861887827?hash=item5b7864ad53:g:njcAAOSw9rpfALFX')
Nike_shoe=soup(source.text, "html5lib")

但不幸的是，我仍然无法检索这些数据。另外我对这个包不熟悉，所以也许我做错了什么。

编辑 22/08/2020 13:41： 下面的两个答案（@Andrej Kesely 和@p1xel）都给出了正确的结果。 p1xel 他的回答可以实现如下：

source=requests.get("https://www.ebay.com/itm/SIGNED-2000-NIKE-AIR-JORDAN-1-HIGH-BANNED-1985-ROOKIE-RETRO-SHOES-AUTOGRAPH-UDA/392861887827?hash=item5b7864ad53:g:njcAAOSw9rpfALFX")
Nike_shoe = soup(source.text, "lxml")
iframe=requests.get(Nike_shoe.find(id="desc_ifr")["src"])
Custom_description = soup(iframe.text, "html5lib")
print(Custom_description.find("td").text

                            SIGNED 2000NIKE AIR JORDAN 1 HIGH BANNED 1985 ROOKIE RETRO SHOES AUTOGRAPH UDA SIGNED IN PRESENCE OF UPPER DECK REPRESENTATIVES•  SHOES ARE OFFICIAL RETRO FROM 2000, BRAND NEW WITH ORIGINAL BOX AND RETRO CARD Beautiful signature accompanied by  CERTIFICATE OF AUTHENTICITY FROM THE UPPER DECK COMPANY, which currently HOLDS AN EXCLUSIVE RIGHTS to ALL authorized authentic Jordan autographed memorabilia & trading cards (No 3rd party authentication here!)Have a peace of mind knowing that YOU ARE GETTING THE REAL DEAL
RECENTLY ACQUIRED BIG COLLECTION FROM A PRIVATE COLLECTOR, PLEASE CHECK OUR AUCTION PERIODICALLY AS WE WILL CONTINUE TO POST NEW ITEMS DAILYIt says on the certificate that "Each individual product that bears the original autograph is signed in the presence of an Upper Deck Authenticated representative and registered by its numbered hologram and kept on permanent file", as part of UDA's patented 5-Step Hologram process. (NO LETTER OF OPINION HERE!)
Pictures are from the actual shoe you are bidding on....  BUY FROM A REPUTABLE COLLECTOR, Please check my feedbacks from previous satisfied buyers and bid with confidence.BUYER TO PAY $100 FOR FULLY INSURED shipping with tracking number & signature confirmation. International buyer are responsible for any import/customs duty fee that might be charged upon delivery of the packageRECENTLY ACQUIRED BIG COLLECTION FROM A PRIVATE COLLECTOR, PLEASE CHECK OUR AUCTION PERIODICALLY AS WE WILL CONTINUE TO POST NEW ITEMS DAILYALL SALES ARE FINAL. MAKE SURE TO CHECK MY OTHER AUCTIONS FOR MORE GREAT MJ MEMORABILIA

由于p1xel他的回答是通过同一页面上的请求格式完成的，这将被选为接受的解决方案，但两种解决方案都很好。

Answer 1

描述似乎在 iframe 中。

您需要找到 ID 为 desc_ifr 的 iframe 并简单地向其 src 发出请求。

这应该做你想要的（未经测试）：

requests.get(Nike_shoe.find(id="desc_ifr")["src"])

Answer 2

描述是从不同的 URL 加载的，您只需要项目 ID，在这种情况下编号为 392861887827：

import requests
from bs4 import BeautifulSoup


url = 'https://www.ebay.com/itm/SIGNED-2000-NIKE-AIR-JORDAN-1-HIGH-BANNED-1985-ROOKIE-RETRO-SHOES-AUTOGRAPH-UDA/392861887827?hash=item5b7864ad53:g:njcAAOSw9rpfALFX'
item_descr_url = 'https://vi.vipr.ebaydesc.com/ws/eBayISAPI.dll?item={item_id}'

item_id = url.split('?')[0].split('/')[-1]  # this should be `392861887827`

soup = BeautifulSoup(requests.get(item_descr_url.format(item_id=item_id)).content, 'html.parser')
print(soup.get_text(strip=True, separator='\n'))

打印：

eBay
SIGNED 2000NIKE AIR JORDAN 1 HIGH BANNED 1985 ROOKIE RETRO SHOES AUTOGRAPH UDA
SIGNED IN PRESENCE OF UPPER DECK REPRESENTATIVES
•  SHOES ARE OFFICIAL RETRO FROM 2000, BRAND NEW WITH ORIGINAL BOX AND RETRO CARD
Beautiful signature accompanied by  CERTIFICATE OF AUTHENTICITY FROM THE UPPER DECK COMPANY, which currently HOLDS AN EXCLUSIVE RIGHTS to ALL authorized authentic Jordan autographed memorabilia & trading cards (No 3rd party authentication here!)
Have a peace of mind knowing that YOU ARE GETTING THE REAL DEAL
RECENTLY ACQUIRED BIG COLLECTION FROM A PRIVATE COLLECTOR, PLEASE CHECK OUR AUCTION PERIODICALLY AS WE WILL CONTINUE TO POST NEW ITEMS DAILY
It says on the certificate that "Each individual product that bears the original autograph is signed in the presence of an Upper Deck Authenticated representative and registered by its numbered hologram and kept on permanent file", as part of UDA's patented 5-Step Hologram process. (NO LETTER OF OPINION HERE!)
Pictures are from the actual shoe you are bidding on....  BUY FROM A REPUTABLE COLLECTOR, Please check my feedbacks from previous satisfied buyers and bid with confidence.
BUYER TO PAY $100 FOR FULLY INSURED shipping with tracking number & signature confirmation. International buyer are responsible for any import/customs duty fee that might be charged upon delivery of the package
RECENTLY ACQUIRED BIG COLLECTION FROM A PRIVATE COLLECTOR, PLEASE CHECK OUR AUCTION PERIODICALLY AS WE WILL CONTINUE TO POST NEW ITEMS DAILY
ALL SALES ARE FINAL. MAKE SURE TO CHECK MY OTHER AUCTIONS FOR MORE GREAT MJ MEMORABILIA