泡菜会在大文件上随机失败并出现 OSError 吗？

Question

问题陈述

我正在使用 python3 并试图腌制一个重约 2 到 3 GB 的 IntervalTrees 字典。这是我的控制台输出：

10:39:25 - project: INFO - Checking if motifs file was generated by pickle...
10:39:25 - project: INFO -   - Motifs file does not seem to have been generated by pickle, proceeding to parse...
10:39:38 - project: INFO -   - Parse complete, constructing IntervalTrees...
11:04:05 - project: INFO -   - IntervalTree construction complete, saving pickle file for next time.
Traceback (most recent call last):
  File "/Users/alex/Documents/project/src/project.py", line 522, in dict_of_IntervalTree_from_motifs_file
    save_as_pickled_object(motifs, output_dir + 'motifs_IntervalTree_dictionary.pickle')
  File "/Users/alex/Documents/project/src/project.py", line 269, in save_as_pickled_object
    def save_as_pickled_object(object, filepath): return pickle.dump(object, open(filepath, "wb"))
OSError: [Errno 22] Invalid argument

我尝试保存的行是

def save_as_pickled_object(object, filepath): return pickle.dump(object, open(filepath, "wb"))

错误可能在调用save_as_pickled_object 15 分钟后出现（在 11:20）。

我用一个小得多的主题文件小节尝试了这个，它工作正常，所有的代码都完全相同，所以它一定是一个规模问题。 在 python 3.6 中是否有任何已知的泡菜错误与您尝试泡菜的规模有关？一般来说，酸洗大文件是否存在已知错误？是否有任何已知的解决方法？

谢谢！

更新：这个问题可能与Python 3 - Can pickle handle byte objects larger than 4GB? 重复

解决方案

这是我使用的代码。

def save_as_pickled_object(obj, filepath):
    """
    This is a defensive way to write pickle.write, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    bytes_out = pickle.dumps(obj)
    n_bytes = sys.getsizeof(bytes_out)
    with open(filepath, 'wb') as f_out:
        for idx in range(0, n_bytes, max_bytes):
            f_out.write(bytes_out[idx:idx+max_bytes])


def try_to_load_as_pickled_object_or_None(filepath):
    """
    This is a defensive way to write pickle.load, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    try:
        input_size = os.path.getsize(filepath)
        bytes_in = bytearray(0)
        with open(filepath, 'rb') as f_in:
            for _ in range(0, input_size, max_bytes):
                bytes_in += f_in.read(max_bytes)
        obj = pickle.loads(bytes_in)
    except:
        return None
    return obj

Answer 1

Alex，如果我没记错的话，这个错误报告完美地描述了你的问题。

http://bugs.python.org/issue24658

作为一种解决方法，我认为您可以pickle.dumps 而不是pickle.dump，然后以小于 2**31 的块写入文件。