【发布时间】：2020-04-12 23:16:47
【问题描述】：
我正在从网站上抓取数据。我发现表格数据在页面的源代码中显示为加载。我想知道如何使用 python 收集这些数据。它似乎是一个反应 js 网络应用程序。
【问题讨论】：
-
在开发工具下查看 XHR 请求。如果您分享网址，我也许可以向您展示我的意思
标签： python web-scraping beautifulsoup scrapy
【发布时间】：2020-04-12 23:16:47
【问题描述】：
如果您转到网络选项卡，您会发现下面的 API 以 json 格式返回数据。
你不需要selenium 或beautifulsoup。
这是下面的代码。
import requests
res=requests.get("https://api.ycombinator.com/companies/export.json?").json()
for item in res:
try:
print('name:' + item['name'])
except:
continue
try:
print('URL:' + item['url'])
except:
continue
try:
print('batch:' + item['batch'])
except:
continue
try:
print('Description:' + item['description'])
except:
continue
API 快照
回应：
【讨论】：
在 XHR 下找不到它作为请求，所以你可以使用 Selenium 来渲染页面，然后用 pandas 抓取表格：
from selenium import webdriver
import pandas as pd
driver = webdriver.Chrome('C:/chromedriver_win32/chromedriver.exe')
url = 'https://www.ycombinator.com/companies/'
driver.get(url)
df = pd.read_html(driver.page_source)[0]
driver.close()
输出：
print (df)
[ 0 1 2
0 Actiondesk s2019 Google Sheets meets Zapier. Actiondesk lets no...
1 Alana s2019 Helping large companies in LATAM hire blue-col...
2 Apero Health s2019 Modern medical billing.
3 Apurata s2019 Small loans for the Latin American middle clas...
4 Arpeggio Bio s2019 Arpeggio builds technology to watch and learn ...
5 Asayer s2019 Asayer is a session replay tool for developers...
6 Asher Bio s2019 We build better immunotherapies
7 AudioFocus s2019 NaN
8 Axite Labs s2019 A modern IP licensing platform to accelerate t...
9 basis s2019 Software to automate construction workflows, s...
10 Beacons AI s2019 Helping creators monetize through short video ...
11 Binks s2019 Binks is a chain of trusted micro-boutiques th...
12 Blair s2019 Financing college education through Income Sha...
13 Boost Biomes s2019 NaN
14 Bouncer s2019 SDK for scanning and verifying credit cards an...
15 Brave Care s2019 Modern healthcare for kids. We do that with a ...
16 Breadfast s2019 Breadfast delivers fresh bread, milk and eggs ...
17 BuildStream s2019 A market network for industrial labor
18 Business Score s2019 Connecting startups with the things they need.
19 Canix s2019 Canix makes it easy to get and stay compliant ...
20 Carry s2019 Carry plans, books, and supports corporate tra...
21 Carve s2019 NaN
22 Cloosiv s2019 Cloosiv is an order-ahead app for independent ...
23 Coco s2019 The Venezuelan Instacart - allowing Venezuelan...
24 CoLab Software s2019 Jira for Mechanical Engineering Teams
25 Compound s2019 Compound helps people who work at startups und...
26 Courier s2019 Send your product's user notifications to the ...
27 Covela s2019 The digital insurance broker for SMEs in LATAM
28 Cuboh s2019 Cuboh helps restaurants use several delivery p...
29 Curri s2019 We provide on-demand material delivery for the...
... ... ...
2009 Zenter w2007 NaN
2010 Jamglue s2006 NaN
2011 Jumpchat s2006 NaN
2012 Likebetter s2006 NaN
2013 Omgpop s2006 NaN
2014 Pollground s2006 Online polls.
2015 Scribd s2006 World's largest online library.
2016 Shoutfit s2006 NaN
2017 Talkito s2006 NaN
2018 Thinkature s2006 NaN
2019 Xobni s2006 NaN
2020 Zanbazaar s2006 NaN
2021 Audiobeta w2006 NaN
2022 Clustrix w2006 NaN
2023 Flagr w2006 NaN
2024 Inkling w2006 NaN
2025 Project Wedding w2006 NaN
2026 Snipshot w2006 We sold Snipshot to Ansa in 2013.
2027 Wufoo w2006 Online form builder.
2028 Airtime s2005 NaN
2029 Clickfacts s2005 NaN
2030 Infogami s2005 NaN
2031 Kiko s2005 We're the best online calendar solution to eve...
2032 Loopt s2005 NaN
2033 Memamp s2005 NaN
2034 Parakey s2005 NaN
2035 Posthaven s2005 Blogging forever
2036 Reddit s2005 The frontpage of the internet.
2037 Simmery s2005 NaN
2038 TextPayMe s2005 NaN
[2039 rows x 3 columns]]
【讨论】：
-
我遇到了未解决的 selenium 导入错误
-
您是否安装了 selenium 和正确的驱动程序？
-
@chitown88 有一个API，你去NetWork下的All Tab你会发现。
-
@KunduK，很好的发现！是的，这就是我要找的！但只是在 XHR 下查看。以后我必须记得在“所有”选项卡下进行检查。感谢您发布！ Srinivas，接受 Kunduks 解决方案。虽然硒会起作用，但他们的是更好的选择。
-
@chitown88 ：当你说什么都没有从 XHR 中找到时，我有一个疑问，然后我检查了所有选项卡并找到了那个 API 链接。但是我感谢你的努力。