Python 网页抓取 Javascript 与 Await

Question

我有一个关于使用 Python 进行网页抓取的问题。我正在尝试使用from requests_html import AsyncHTMLSession 从https://www.nyse.com/ipo-center/filings 的第一个表中获取数据。

我的代码在这里：

from bs4 import BeautifulSoup
from requests_html import AsyncHTMLSession

#first define the URL and start the session
url = 'http://www.nyse.com/ipo-center/filings'
session = AsyncHTMLSession()

#then get the URL content, and load the html content after parsing through the javascript
r = await session.get(url)
await r.html.arender()

#then we create a beautifulsoup object based on the rendered html
soup = BeautifulSoup(r.html.html, "lxml")

#then we find the first datatable, which is the one that contains upcoming IPO data
table1 = soup.find('table', class_='table table-data table-condensed spacer-lg')

现在我有两个问题：

1. 网站通常不会从table1 返回任何有效信息，因此我无法获得表内的基础信息。到目前为止，我通过简单地等待几秒钟来绕过它，然后再次运行循环，直到加载数据帧。不过可能不是最佳选择。
2. 该代码在 Jupyter Notebook 中确实可以工作，但是一旦我以 .py 格式将其上传到我的服务器上，我就会收到 SyntaxError: 'await' outside async function 的错误消息。

有没有人能解决上面提到的两个问题？

Answer 1

由于您使用的是协程，因此您需要将它们包装在 async 函数中。见下例

from bs4 import BeautifulSoup
from requests_html import AsyncHTMLSession

#first define the URL and start the session
url = 'http://www.nyse.com/ipo-center/filings'
session = AsyncHTMLSession()

#then get the URL content, and load the html content after parsing through the javascript
async def get_page():
    r = await session.get(url)
    await r.html.arender(timeout=20)
    return r.text

data = session.run(get_page)

#then we create a beautifulsoup object based on the rendered html
soup = BeautifulSoup(data[0], "lxml")

#then we find the first datatable, which is the one that contains upcoming IPO data
table1 = soup.find_all('table', class_='table table-data table-condensed spacer-lg')
print(table1)