pickle 在 Ubuntu 14.04 上引发 EOFError 但适用于 Mac OS 和 Linux Mint答案

【问题标题】：pickle raises EOFError on Ubuntu 14.04 but works on Mac OS and Linux Mintpickle 在 Ubuntu 14.04 上引发 EOFError 但适用于 Mac OS 和 Linux Mint
【发布时间】：2016-08-07 13:57:41
【问题描述】：

我正在使用 Python 的 pickle 模块保存一些大数据。更准确地说，我将两个对象转储到同一个泡菜文件中，如下所示：

def save(sim):
    if sim.tstamp > 0:
        with open(filepath(sim.identifier), 'wb') as f:
            pickle.dump(sim.serialize(), f)
            pickle.dump(sim, f)
    else:
        raise ValueError("Simulation not yet simulated")

sim是一个自写的类，具有不同的属性和一个名为serialize()的方法。此方法通过使用以下内容将所有属性及其值转换为字典：

def serialize(self):
    keys = ['height', 'diameter', 'rpm', 'packaging']
    serialized = dict((key, getattr(self, key)) for key in keys)
    serialized['sectors'] = [sector.serialize() for sector in self.sectors]
    return serialized

所以基本上sim.serialize() 返回一个字典，该字典作为第一个对象转储到pickle 文件中。第二个对象是模拟对象本身。这是由于每个模拟对象的尺寸很大。提供字典可以进行一种汇总，以防止在非必要时加载完整的模拟。

从我使用的泡菜文件中加载数据：

def load(fname, params_only=False):
    simulation = find(fname) # find returns file name
    try:
        with open(simulation, 'r') as json_file:
            return json.load(json_file)
    except (UnicodeDecodeError, ValueError) as e:
        with open(simulation, 'rb') as pickle_file:
            params = pickle.load(pickle_file)
            if params_only:
                return params
            return pickle.load(pickle_file)

由于较旧的 API 也能够处理 JSON 文件，因此我需要同时处理 JSON 文件和 pickle 文件。

转储对象按需要工作（不会引发错误消息）。但是，在请求某些数据图时，API 需要取消提取数据，这会引发以下错误：

[2016-08-07 14:39:34,637] ERROR in app: Exception on /api/simulations/20160807_123707581/heatflux_distribution.png [GET]
Traceback (most recent call last):
 File "/home/abkos/abkosproject/abkos/_tools.py", line 36, in load
   return json.load(json_file)
 File "/usr/lib/python3.4/json/__init__.py", line 265, in load
   return loads(fp.read(),
 File "/usr/lib/python3.4/codecs.py", line 319, in decode
   (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1988, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1641, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1544, in handle_user_exception
   reraise(exc_type, exc_value, tb)
 File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
   raise value
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1639, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1625, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "./app.py", line 118, in plot_figure
   img = plotter(identifier, figname, request.args.items())
 File "./app.py", line 122, in plotter
   sim = Simulation.from_file(identifier)
 File "/home/abkos/abkosproject/abkos/calculation.py", line 56, in from_file
   params_or_simulation = load(fname, params_only=new)
 File "/home/abkos/abkosproject/abkos/_tools.py", line 42, in load
   return pickle.load(pickle_file)
_pickle.UnpicklingError
10.0.2.2 - - [07/Aug/2016 14:39:34] "GET /api/simulations/20160807_123707581/heatflux_distribution.png HTTP/1.1" 500 -
[2016-08-07 14:39:38,691] ERROR in app: Exception on /api/simulations/20160807_123707581/heatmap_axial_temperatures.png [GET]
Traceback (most recent call last):
 File "/home/abkos/abkosproject/abkos/_tools.py", line 36, in load
   return json.load(json_file)
 File "/usr/lib/python3.4/json/__init__.py", line 265, in load
   return loads(fp.read(),
 File "/usr/lib/python3.4/codecs.py", line 319, in decode
   (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1988, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1641, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1544, in handle_user_exception
   reraise(exc_type, exc_value, tb)
 File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
   raise value
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1639, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1625, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "./app.py", line 118, in plot_figure
   img = plotter(identifier, figname, request.args.items())
 File "./app.py", line 122, in plotter
   sim = Simulation.from_file(identifier)
 File "/home/abkos/abkosproject/abkos/calculation.py", line 56, in from_file
   params_or_simulation = load(fname, params_only=new)
 File "/home/abkos/abkosproject/abkos/_tools.py", line 42, in load
   return pickle.load(pickle_file)
EOFError
10.0.2.2 - - [07/Aug/2016 14:39:38] "GET /api/simulations/20160807_123707581/heatmap_axial_temperatures.png HTTP/1.1" 500 -

代码在使用 Python 3.4.3 的 MacOS 和 Linux Mint 上运行良好。但是，它不适用于 Ubuntu 14.04 (Python 3.4.3)。

关于从哪里开始调查错误有什么想法吗？

【问题讨论】：

标签： python macos python-3.x ubuntu pickle

【解决方案1】：

由于我找不到任何编码错误，我将这三个系统相互比较，发现从软件的角度来看几乎相同。它们都使用 Python 3.4.3 64 位，pickle 是 Python 默认模块的一部分。

所以我得出结论，这可能是与硬件相关的问题。由于 MacOS 机器和 Mint 机器都是“真正的”机器，而 Ubuntu 机器只是虚拟机，我比较了一般硬件规格，例如 CPU 和 RAM 的数量。

真机分别配备 8 GB RAM。薄荷的 12 GB。所以我将 VM 的 RAM 从大约 1 GB 增加到 3 GB，一切正常。

所以即使没有MemoryError 或类似的提出，EOFError 也可以与内存相关。

【讨论】：