【问题标题】:How to save the BeautifulSoup object to a file and then read from it as BeautifulSoup?如何将 BeautifulSoup 对象保存到文件中,然后将其作为 BeautifulSoup 读取?
【发布时间】:2019-03-29 03:31:18
【问题描述】:

我想将 BeautifulSoup 对象保存到文件中。因此,我将其更改为字符串,然后将其写入文件。然后将其作为字符串读取后,我将字符串转换为 BeautifulSoup 对象。这将有助于我的测试,因为我正在抓取的数据是动态的。

url = "https://coinmarketcap.com/all/views/all/"
html = urlopen(url)
soup = BeautifulSoup(html,"lxml")

这样写汤对象:

  new_soup = str(soup)
  with open("coin.txt", "w+") as f:
      f.write(new_soup)

产生这个错误:

UnicodeEncodeError: 'charmap' codec can't encode 
characters in position 28127-28132: character maps to <undefined>

另外,如果我能够将它保存到文件中,我将如何读取作为 BeautifulSoup 对象返回的字符串?

【问题讨论】:

标签: python beautifulsoup


【解决方案1】:

编辑

由于RecursionError,旧代码无法腌制soup对象:

Traceback (most recent call last):
  File "soup.py", line 20, in <module>
    pickle.dump(soup, f)
RecursionError: maximum recursion depth exceeded while calling a Python object

解决办法是increase the recursion limit。他们在 this answer 中做同样的事情,而 docs 又引用了 docs。

但是,您尝试加载和保存的特定网站是非常嵌套的。我的计算机无法超过 50000 的递归限制,这对您的站点来说还不够,并且崩溃:10008 segmentation fault (core dumped) python soup.py。

因此,如果您需要下载 HTML 并稍后使用,您可以这样做:

from bs4 import BeautifulSoup
from urllib.request import urlopen

url = "https://coinmarketcap.com/all/views/all/"
html = urlopen(url)

# Save HTML to a file
with open("soup.html", "wb") as f:
    while True:
        chunk = html.read(1024)
        if not chunk:
            break
        f.write(chunk)

然后你可以读取你保存的 HTML 文件并用它实例化 bs4 对象:

# Read HTML from a file
with open("soup.html", "rb") as f:
    soup = BeautifulSoup(f.read(), "lxml")

print(soup.title)
# <title>All Cryptocurrencies | CoinMarketCap</title>

此外,这是我将用于嵌套较少的网站的代码:

import pickle
from bs4 import BeautifulSoup
from urllib.request import urlopen
import sys

url = "https://stackoverflow.com/questions/52973700/how-to-save-the-beautifulsoup-object-to-a-file-and-then-read-from-it-as-beautifu"
html = urlopen(url)
soup = BeautifulSoup(html,"lxml")

sys.setrecursionlimit(8000)

# Save the soup object to a file
with open("soup.pickle", "wb") as f:
    pickle.dump(soup, f)

# Read the soup object from a file
with open("soup.pickle", "rb") as f:
    soup_obj = pickle.load(f)

print(soup_obj.title)

# <title>python - How to save the BeautifulSoup object to a file and then read from it as BeautifulSoup? - Stack Overflow</title>.

【讨论】:

  • 我在运行这段代码时得到RecursionError: maximum recursion depth exceeded while pickling an object,因为对象很大。
  • 我也收到“递归错误:超出最大深度”错误
  • 你说得对。我编辑了我的答案。结果是生成的汤对象太嵌套而无法腌制它,但有一些解决方法。
猜你喜欢
  • 2021-11-03
  • 2022-11-16
  • 2020-12-07
  • 1970-01-01
  • 1970-01-01
  • 2021-05-23
  • 1970-01-01
  • 2020-02-24
  • 2013-02-27
相关资源
最近更新 更多