【发布时间】：2019-08-23 21:56:13
【问题描述】：
我正在尝试用漂亮的汤来计算行程时间：
https://betaplanner.trimet.org/map/#/?fromPlace=10255%20SW%20CANYON%20RD%3A%3A45.493227%2C-122.782138&toPlace=4809%20N%20KERBY%20AVE%3A%3A45.557817%2C-122.673216&date=2019-08-14&time=16%3A56&arriveBy=false&mode=BUS%2CTRAM%2CRAIL%2CGONDOLA%2CCAR_RENT&showIntermediateStops=true&optimize=QUICK&ignoreRealtimeUpdates=true&companies=NaN&minTransitDistance=50%25&searchTimeout=10000&onlyTransitTrips=true&ui_activeItinerary=0
问题似乎与 javascript 渲染有关，但是，当我尝试执行 html-requests 时，出现错误
对于 Bs4：
r = session.get(*linkfromabove*)
soup = BeautifulSoup(r.content, 'html.parser')
soup.select
对于 HTML_Requests：
import requests
from requests_html import HTMLSession
session = HTMLSession()
r = session.get(linkfromabove)
r.html.render(wait=8, sleep=8)
使用 BS4 的实际结果：
<html lang="en">
<head>
<meta charset="utf-8"/>
<meta content="ie=edge" http-equiv="x-ua-compatible"/>
<title>TriMet MOD OTP Demo</title>
<meta content="width=device-width, initial-scale=1" name="viewport"/>
<link href="https://fonts.googleapis.com/css?
family=Hind:300,400,500,600,700"
rel="stylesheet"/>
<link href="main.css" rel="stylesheet"/></head>
<body>
<div id="main"></div>
<script src="bundle.js" type="text/javascript"></script></body>
</html>
来自 Html 请求：
pyppeteer.errors.NetworkError: Protocol error Target.closeTarget: Target closed.
【问题讨论】：
-
bs4 和 requests 都不做 javascript 渲染。除了硒，我没有任何想法。
标签： html web-scraping beautifulsoup