将 30 Gb json 文件拆分为更小的文件

Question

我在读取 30 GB 大小的 json 文件时遇到内存问题。 Python3.x 中是否有任何直接的方法，就像我们在 unix 中一样，我们可以根据行将 json 文件拆分为更小的文件。

例如前100000条记录进入第一个slit文件，然后剩下的进入后续的子json文件？

Answer 1

根据您的输入数据，如果其结构已知且一致，则将更难或更容易。

在我的示例中，想法是使用lazy generator 逐行读取文件，并在可以从输入构造有效对象时写入新文件。有点像手动解析。

在现实世界中，何时写入新文件的逻辑很大程度上取决于您的输入以及您想要实现的目标。

一些样本数据

[
    {
        "color": "red",
        "value": "#f00"
    },
    {
        "color": "green",
        "value": "#0f0"
    },
    {
        "color": "blue",
        "value": "#00f"
    },
    {
        "color": "cyan",
        "value": "#0ff"
    },
    {
        "color": "magenta",
        "value": "#f0f"
    },
    {
        "color": "yellow",
        "value": "#ff0"
    },
    {
        "color": "black",
        "value": "#000"
    }
]

# create a generator that yields each individual line
lines = (l for l in open('data.json'))

# o is used to accumulate some lines before
# writing to the files
o=''

# itemCount is used to count the number of valid json objects
itemCount=0

# read the file line by line to avoid memory issues
i=-1
while True:
  try:
    line = next(lines)
  except StopIteration:
    break
  i=i+1
  # ignore first square brackets
  if i == 0:
    continue
  # in this data I know every 5th lines a new object will begin
  # this logic depends on your input data
  if i%4==0:
    itemCount+=1
    # at this point I am able to create avalid json object
    # based on my knowledge of the input file structure
    validObject=o+line.replace("},\n", '}\n')
    o=''
    # now write each object to its own file
    with open(f'item-{itemCount}.json', 'w') as outfile:
      outfile.write(validObject)
  else:
    o+=line

这是一个带有工作示例的 repl：https://replit.com/@bluebrown/linebyline