泡菜文件太大而无法加载

Question

我遇到的问题是我试图打开一个非常大的 pickle 文件 (2.6 Gb)，但每次打开时都会出现内存错误。我现在意识到我应该使用数据库来存储所有信息，但现在为时已晚。 pickle 文件包含从互联网上抓取的美国国会记录中的日期和文本（运行大约需要 2 周时间）。

有什么方法可以访问我以增量方式转储到 pickle 文件中的信息，或者有什么方法可以将 pickle 文件转换为 SQL 数据库或其他我可以打开而无需重新输入所有数据的方法。我真的不想再花两周时间重新抓取国会记录并将数据输入数据库。

非常感谢您的帮助

编辑*

对象如何被腌制的代码：

def save_objects(objects): 
    with open('objects.pkl', 'wb') as output: 
        pickle.dump(objects, output, pickle.HIGHEST_PROTOCOL)

def Main():   
    Links()
    file = open('datafile.txt', 'w')
    objects = []
    with open('links2.txt', 'rb') as infile:
        for link in infile: 
            print(link)
            title, text, date = Get_full_text(link)
            article=Doccument(title, date, text)
            if text != None:
                write_to_text(date, text)
                objects.append(article)
                save_objects(objects)

这是有错误的程序：

def Main():
    file = open('objects1.pkl', 'rb') 
    object = pickle.load(file)

Answer 1

您没有以增量方式腌制您的数据。您一次又一次地单独腌制您的数据。每次循环时，您都会销毁您拥有的所有输出数据（open(...,'wb') 会销毁输出文件），然后再次重新写入所有数据。此外，如果您的程序曾经停止然后使用新的输入数据重新启动，那么旧的输出数据就会丢失。

我不知道为什么 objects 在您进行酸洗时没有导致内存不足错误，因为它增长到与 pickle.load() 想要创建的对象相同的大小。

您可以通过以下方式逐步创建 pickle 文件：

def save_objects(objects): 
    with open('objects.pkl', 'ab') as output:  # Note: `ab` appends the data
        pickle.dump(objects, output, pickle.HIGHEST_PROTOCOL)

def Main():
    ...
    #objects=[] <-- lose the objects list
    with open('links2.txt', 'rb') as infile:
        for link in infile: 
            ... 
            save_objects(article)

那么您可以像这样逐步读取 pickle 文件：

import pickle
with open('objects.pkl', 'rb') as pickle_file:
    try:
        while True:
            article = pickle.load(pickle_file)
            print article
    except EOFError:
        pass

我能想到的选择是：

• 试试 cPickle。这可能会有所帮助。
• 尝试流式泡菜
• 在具有大量 RAM 的 64 位环境中读取您的 pickle 文件
• 重新抓取原始数据，这次实际上是增量存储数据，或者将其存储在数据库中。 如果没有不断重写您的 pickle 输出文件的低效率，那么这次您的抓取速度可能会大大加快。

Answer 2

看来你有点吃不消了！ ;-)。希望在此之后，你永远不会使用 PICKLE。这不是一种很好的数据存储格式。

无论如何，对于这个答案，我假设您的 Document 课程看起来有点像这样。如果没有，请使用您的实际 Document 类评论：

class Document(object): # <-- object part is very important! If it's not there, the format is different!
    def __init__(self, title, date, text): # assuming all strings
        self.title = title
        self.date = date
        self.text = text

不管怎样，我用这个类做了一些简单的测试数据：

d = [Document(title='foo', text='foo is good', date='1/1/1'), Document(title='bar', text='bar is better', date='2/2/2'), Document(title='baz', text='no one likes baz :(', date='3/3/3')]

用2 格式腌制它（pickle.HIGHEST_PROTOCOL 用于 Python 2.x）

>>> s = pickle.dumps(d, 2)
>>> s
'\x80\x02]q\x00(c__main__\nDocument\nq\x01)\x81q\x02}q\x03(U\x04dateq\x04U\x051/1/1q\x05U\x04textq\x06U\x0bfoo is goodq\x07U\x05titleq\x08U\x03fooq\tubh\x01)\x81q\n}q\x0b(h\x04U\x052/2/2q\x0ch\x06U\rbar is betterq\rh\x08U\x03barq\x0eubh\x01)\x81q\x0f}q\x10(h\x04U\x053/3/3q\x11h\x06U\x13no one likes baz :(q\x12h\x08U\x03bazq\x13ube.'

并用pickletools拆解它：

>>> pickletools.dis(s)
    0: \x80 PROTO      2
    2: ]    EMPTY_LIST
    3: q    BINPUT     0
    5: (    MARK
    6: c        GLOBAL     '__main__ Document'
   25: q        BINPUT     1
   27: )        EMPTY_TUPLE
   28: \x81     NEWOBJ
   29: q        BINPUT     2
   31: }        EMPTY_DICT
   32: q        BINPUT     3
   34: (        MARK
   35: U            SHORT_BINSTRING 'date'
   41: q            BINPUT     4
   43: U            SHORT_BINSTRING '1/1/1'
   50: q            BINPUT     5
   52: U            SHORT_BINSTRING 'text'
   58: q            BINPUT     6
   60: U            SHORT_BINSTRING 'foo is good'
   73: q            BINPUT     7
   75: U            SHORT_BINSTRING 'title'
   82: q            BINPUT     8
   84: U            SHORT_BINSTRING 'foo'
   89: q            BINPUT     9
   91: u            SETITEMS   (MARK at 34)
   92: b        BUILD
   93: h        BINGET     1
   95: )        EMPTY_TUPLE
   96: \x81     NEWOBJ
   97: q        BINPUT     10
   99: }        EMPTY_DICT
  100: q        BINPUT     11
  102: (        MARK
  103: h            BINGET     4
  105: U            SHORT_BINSTRING '2/2/2'
  112: q            BINPUT     12
  114: h            BINGET     6
  116: U            SHORT_BINSTRING 'bar is better'
  131: q            BINPUT     13
  133: h            BINGET     8
  135: U            SHORT_BINSTRING 'bar'
  140: q            BINPUT     14
  142: u            SETITEMS   (MARK at 102)
  143: b        BUILD
  144: h        BINGET     1
  146: )        EMPTY_TUPLE
  147: \x81     NEWOBJ
  148: q        BINPUT     15
  150: }        EMPTY_DICT
  151: q        BINPUT     16
  153: (        MARK
  154: h            BINGET     4
  156: U            SHORT_BINSTRING '3/3/3'
  163: q            BINPUT     17
  165: h            BINGET     6
  167: U            SHORT_BINSTRING 'no one likes baz :('
  188: q            BINPUT     18
  190: h            BINGET     8
  192: U            SHORT_BINSTRING 'baz'
  197: q            BINPUT     19
  199: u            SETITEMS   (MARK at 153)
  200: b        BUILD
  201: e        APPENDS    (MARK at 5)
  202: .    STOP

看起来很复杂！但实际上，它并没有那么糟糕。 pickle 基本上是一个堆栈机器，您看到的每个 ALL_CAPS 标识符都是一个操作码，它以某种方式操纵内部“堆栈”进行解码。如果我们试图解析一些复杂的结构，这将更重要，但幸运的是我们只是制作了一个简单的本质元组列表。所有这些“代码”所做的就是在堆栈上构造一堆对象，然后将整个堆栈推入一个列表中。

我们确实需要关心的一件事是您看到的散布在各处的“BINPUT”/“BIGET”操作码。基本上，这些是用于“记忆化”，以减少数据占用，pickle 用BINPUT <id> 保存字符串，然后如果它们再次出现，而不是重新转储它们，只需放置一个BINGET <id> 从缓存。

另外，还有一个并发症！不仅仅是SHORT_BINSTRING - 对于> 256字节的字符串，还有普通的BINSTRING，还有一些有趣的unicode变体。我只是假设您使用的是带有所有 ASCII 字符串的 Python 2。如果这不是一个正确的假设，请再次评论。

好的，所以我们需要流式传输文件，直到达到 '\81' 字节 (NEWOBJ)。然后，我们需要向前扫描，直到我们遇到一个 '(' (MARK) 字符。然后，直到我们遇到一个 'u' (SETITEMS)，我们读取键/值字符串对 - 应该有 3 对总计，每个字段一个。

那么，让我们这样做吧。这是我以流方式读取泡菜数据的脚本。它远非完美，因为我只是为了这个答案而将它一起破解，你需要对其进行大量修改，但这是一个好的开始。

pickledata = '\x80\x02]q\x00(c__main__\nDocument\nq\x01)\x81q\x02}q\x03(U\x04dateq\x04U\x051/1/1q\x05U\x04textq\x06U\x0bfoo is goodq\x07U\x05titleq\x08U\x03fooq\tubh\x01)\x81q\n}q\x0b(h\x04U\x052/2/2q\x0ch\x06T\x14\x05\x00\x00bar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterq\rh\x08U\x03barq\x0eubh\x01)\x81q\x0f}q\x10(h\x04U\x053/3/3q\x11h\x06U\x13no one likes baz :(q\x12h\x08U\x03bazq\x13ube.'

# simulate a file here
import StringIO
picklefile = StringIO.StringIO(pickledata)

import pickle # just for opcode names
import struct # binary unpacking

def try_memo(f, v, cache):
    opcode = f.read(1)
    if opcode == pickle.BINPUT:
        cache[f.read(1)] = v
    elif opcode == pickle.LONG_BINPUT:
        print 'skipping LONG_BINPUT to save memory, LONG_BINGET will probably not be used'
        f.read(4)
    else:
        f.seek(f.tell() - 1) # rewind

def try_read_string(f, opcode, cache):
    if opcode in [ pickle.SHORT_BINSTRING, pickle.BINSTRING ]:
        length_type = 'b' if opcode == pickle.SHORT_BINSTRING else 'i'
        str_length = struct.unpack(length_type, f.read(struct.calcsize(length_type)))[0]
        value = f.read(str_length)
        try_memo(f, value, memo_cache)
        return value
    elif opcode == pickle.BINGET:
        return memo_cache[f.read(1)]
    elif opcide == pickle.LONG_BINGET:
        raise Exception('Unexpected LONG_BINGET? Key ' + f.read(4))
    else:
        raise Exception('Invalid opcode ' + opcode + ' at pos ' + str(f.tell()))

memo_cache = {}
while True:
    c = picklefile.read(1)
    if c == pickle.NEWOBJ:
        while picklefile.read(1) != pickle.MARK:
            pass # scan forward to field instantiation
        fields = {}
        while True:
            opcode = picklefile.read(1)
            if opcode == pickle.SETITEMS:
                break
            key = try_read_string(picklefile, opcode, memo_cache)
            value = try_read_string(picklefile, picklefile.read(1), memo_cache)
            fields[key] = value
        print 'Document', fields
        # insert to sqllite
    elif c == pickle.STOP:
        break

这可以正确读取我的pickle格式2的测试数据（修改为长字符串）：

$ python picklereader.py
Document {'date': '1/1/1', 'text': 'foo is good', 'title': 'foo'}
Document {'date': '2/2/2', 'text': 'bar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is betterbar is better', 'title': 'bar'}
Document {'date': '3/3/3', 'text': 'no one likes baz :(', 'title': 'baz'}

祝你好运！

Answer 3

我最近遇到了非常相似的情况 - 一个 11 GB 的泡菜。我没有尝试将它增量加载到我的机器上，因为我没有足够的时间来实现自己的增量加载器或为我的案例改进现有的加载器。

我所做的是，我在云托管提供商中启动了一个具有足够内存的大型实例（如果仅在几个小时之类的短时间内启动，价格并不高），通过 SSH (SCP) 将该文件上传到该服务器并简单地将其加载到该实例上以在那里对其进行分析 + 将其重写为更合适的格式。

不是一种编程解决方案，但时间有效（省力）。