我创建了函数get_soup(page)，它接受来自1 的page 参数并返回带有相关数据的汤。你可以把这个函数放在一个循环中来抓取更多的页面：
import requests
from bs4 import BeautifulSoup
def get_soup(page=1):
headers = {
'Accept-Language' : 'en-US,en;q=0.5',
'Host' : 'angel.co',
'User-Agent' : 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'
}
payload={'sort':'signal','page':str(page)}
url = 'https://angel.co/company_filters/search_data'
data = requests.get(url, headers=headers, data=payload).json()
new_url = 'https://angel.co/companies/startups?' + '&'.join('ids[]={}'.format(_id) for _id in data['ids'])
new_url += '&sort=' + data['sort']
new_url += '&total=' + str(data['total'])
new_url += '&page=' + str(data['page'])
new_url += '&new=' + str(data['new']).lower()
new_url += '&hexdigest=' + data['hexdigest']
data = requests.get(new_url, headers=headers).json()
return BeautifulSoup(data['html'], 'lxml')
soup = get_soup(1)
rows = []
for company, joined, location, market, website, company_size, stage, raised in zip(soup.select('.column.company'),
soup.select('.column.joined .value'),
soup.select('.column.location .value'),
soup.select('.column.market .value'),
soup.select('.column.website .value'),
soup.select('.column.company_size .value'),
soup.select('.column.stage .value'),
soup.select('.column.raised .value')):
company = company.get_text(strip=True, separator=" ")
joined = joined.get_text(strip=True)
location = location.get_text(strip=True)
market = market.get_text(strip=True)
website = website.get_text(strip=True)
company_size = company_size.get_text(strip=True)
stage = stage.get_text(strip=True)
raised = raised.get_text(strip=True)
rows.append([company, joined, location, market, website, company_size, stage, raised])
from textwrap import shorten
print(''.join('{: <25}'.format(shorten(d, 25)) for d in ['Company', 'Joined', 'Location', 'Market', 'Website', 'Company Size', 'Stage', 'Raised']))
print('-' * (25*8))
for row in rows:
print(''.join('{: <25}'.format(shorten(d, 25)) for d in row))
打印：
Company Joined Location Market Website Company Size Stage Raised
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Nutanix Your [...] May ’14 San Jose Virtualization nutanix.com 1001-5000 IPO $312,200,000
EverFi Oct ’12 Washington DC Education everfi.com 51-200 Series C $61,000,000
Butter Make friends [...]Jun ’14 San Francisco Messaging getbutter.me 1-10 Seed $371,500
Fluent The future [...] Mar ’12 Sydney Curated Web fluent.io - - -
Belly Sep ’12 Chicago Small and Medium [...] bellycard.com Series B $24,975,000
Autotech Ventures [...] Apr ’14 Menlo Park Internet of Things autotechvc.com 1-10 - -
Oscar Health [...] Jun ’14 Tempe Technology hioscar.com 1001-5000 $1,267,500,000
Tovala Smart oven [...] Feb ’16 Chicago Home Automation tovala.com 11-50 Series A $10,800,000
GiftRocket Online [...] Mar ’16 San Francisco Gift Card giftrocket.com 1-10 Seed $520,000
Elemeno Health B2B [...] Apr ’16 Oakland Training elemenohealth.com 1-10 Seed $1,635,000
Sudo Technologies [...] Apr ’16 Menlo Park - sudo.ai - -
Stypi Sep ’16 - - Acquired -
Amazon Alexa Amazon [...]Sep ’16 Cambridge Speech Recognition developer.amazon.com 11-50 - -
Altos Ventures A [...] Oct ’16 Menlo Park Technology altos.vc 1-10 - -
Flirtey Making [...] Oct ’16 Reno - flirtey.com 11-50 Series A $16,000,000
SV Liquidity Fund [...] Oct ’16 San Francisco B2B svlq.io 1-10 - -
Princeton Ventures [...] Jan ’17 Princeton Technology princetonventures.com 1-10 - -
hulu - Beijing [...] Jan ’17 Beijing TV Production hulu.com - - -
Distributed Systems [...]Jan ’17 San Francisco Identity pavlov.ai 1-10 - -
Fetch Marketplace [...] May ’17 Atlanta Technology fetchtruck.com 1-10 Seed -
编辑：为了获取链接，您可以这样做：
soup = get_soup(1)
for a in soup.select('.website a[href]'):
print(a['href'])
打印：
http://www.fuelpowered.com
http://www.slide.com
http://www.mparticle.com
http://www.matter.io
http://www.smartling.com
https://stensul.com
https://avametric.com/
https://ledgerinvesting.com
http://www.relativityspace.com
http://teamdom.co
http://www.wonderschool.com
http://www.upcall.com
http://focal.systems
https://asktetra.com
https://www.subdreamstudios.com/
http://www.stedi.com
http://www.magnarapp.com/
http://www.kylie.ai
http://clipboardhealth.com