如何处理大型excel文件中的内存错误

Question

#我的任务是读取这个文件文件创建一个数据透视表，所以在根据列名读取文件时我得到一个内存错误有没有办法处理它

import os
    import xlrd
    import pandas as pd
    import openpyxl as xl
    ##get current directory
    cwd= os.getcwd()
    print("This script is to update")
    print("Starting to populate the Data Validation File")
    filename_1 =[os.path.join(root, f) for root, _, files in os.walk(os.getcwd())
                           for f in files
                           if f.startswith('1') and f.endswith('.xlsx')]
    filename_1=filename_1[0]

    book_1 = xl.load_workbook(filename_1)
    ws = book_1["J"]
    print("Loaded the filename:")
    for filename in os.listdir(cwd):
        if filename.endswith('.xlsx') and filename.startswith(('Seeep_')):
            book = pd.ExcelFile(filename)
            for sheet in book.sheet_names:
   here i am i am getting the memory error how to handle this the file is really big**


    df = book.parse(sheet)
                if 'comp' in df.columns:
                    df.columns = df.columns.str.replace(' ', '')
                    df['comp1']=df['com']
                    print("For "+filename+"\n"+" Creating the Pivot Table")
                    book2=df[(df.Iy!='SE')].pivot_table(values='comp',index=['os','sv'], columns='comp', aggfunc='count',margins=True)
                    print(book2)
                    book3=df.pivot_table(values='compl',index=['Ring','os','sv'], columns='comp', aggfunc='count',margins=True)

Answer 1

我建议：

1. 将 Excel 工作表转换为单独的 CSV。例如。使用https://openpyxl.readthedocs.io/en/stable/optimized.html#read-only-mode 遍历行并使用Python 的CSV 编写API 将它们写入CSV https://docs.python.org/3/library/csv.html#csv.writer

2. 然后使用 Pandas 加载 CSV，确保只加载您真正需要的列，并设置列类型以使用更少的内存。 https://pythonspeed.com/articles/pandas-load-less-data/ 介绍了如何执行此操作。