编辑
由于RecursionError,旧代码无法腌制soup对象:
Traceback (most recent call last):
File "soup.py", line 20, in <module>
pickle.dump(soup, f)
RecursionError: maximum recursion depth exceeded while calling a Python object
解决办法是increase the recursion limit。他们在 this answer 中做同样的事情,而 docs 又引用了 docs。
但是,您尝试加载和保存的特定网站是非常嵌套的。我的计算机无法超过 50000 的递归限制,这对您的站点来说还不够,并且崩溃:10008 segmentation fault (core dumped) python soup.py。
因此,如果您需要下载 HTML 并稍后使用,您可以这样做:
from bs4 import BeautifulSoup
from urllib.request import urlopen
url = "https://coinmarketcap.com/all/views/all/"
html = urlopen(url)
# Save HTML to a file
with open("soup.html", "wb") as f:
while True:
chunk = html.read(1024)
if not chunk:
break
f.write(chunk)
然后你可以读取你保存的 HTML 文件并用它实例化 bs4 对象:
# Read HTML from a file
with open("soup.html", "rb") as f:
soup = BeautifulSoup(f.read(), "lxml")
print(soup.title)
# <title>All Cryptocurrencies | CoinMarketCap</title>
此外,这是我将用于嵌套较少的网站的代码:
import pickle
from bs4 import BeautifulSoup
from urllib.request import urlopen
import sys
url = "https://stackoverflow.com/questions/52973700/how-to-save-the-beautifulsoup-object-to-a-file-and-then-read-from-it-as-beautifu"
html = urlopen(url)
soup = BeautifulSoup(html,"lxml")
sys.setrecursionlimit(8000)
# Save the soup object to a file
with open("soup.pickle", "wb") as f:
pickle.dump(soup, f)
# Read the soup object from a file
with open("soup.pickle", "rb") as f:
soup_obj = pickle.load(f)
print(soup_obj.title)
# <title>python - How to save the BeautifulSoup object to a file and then read from it as BeautifulSoup? - Stack Overflow</title>.