如何使用 Python 以更快的方式将 csv 文件加载到 Oracle 表中

Question

我正在互联网上研究如何以更快的方式将 CSV 文件加载到 Oracle DB 表中

以下是我加载表的方式，该表作为文件中的 1000 条记录并很快被加载。

但如果有 50,000 条记录，则大约需要 50,000 条记录。 5 分钟加载。

有没有更快更有效的方式将 csv 文件加载到 Oracle 表中？

我的代码：

import cx_Oracle
import csv

myquery='insert into TABLE (COLUMNNAME1,COLUMNNAME2,COLUMNNAME3,COLUMNNAME4,COLUMNNAME5,COLUMNNAME6) values (:1, :2, :3 ,:4,:5,:6)'
separator='|'

oracleConnection='user/password@hostIP/database'

def fileLoading(oracleconnection, file_name,myquery,separator):
    try:
        con = cx_Oracle.connect(oracleConnection)
        print("Connection established")
        reader = csv.reader(open("demp.csv", "r"), header=None,delimiter=separator)
        columns = []
        for line in reader:
            columns.append(line)
            cur = con.cursor()
            for line in columns:
                print("Inserting record to table")
                insrt_stmt = myquery
                cur.execute(insrt_stmt, line)
                con.commit()
                cur.close()
                print("load completed")
    except Exception as er:
        print('ERRO:',er)

Answer 1

我不熟悉python，但我会这样尝试：

con = cx_Oracle.connect(oracleConnection)
print("Connection established")
reader = csv.reader(open("demp.csv", "r"), header=None,delimiter=separator)
cur = con.cursor()
insrt_stmt = myquery

for line in reader:
   columns = []
   for line in columns:
       columns.append(line)
       cur.execute(insrt_stmt, line)
con.commit()
cur.close()
print("load completed")

Answer 2

使用executeMany()，如documentation 所示。这比重复调用execute() 快​​得多：

import cx_Oracle
import csv

. . .

# Predefine the memory areas to match the table definition
cursor.setinputsizes(None, 25)

# Adjust the batch size to meet your memory and performance requirements
batch_size = 10000

with open('testsp.csv', 'r') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    sql = "insert into test (id,name) values (:1, :2)"
    data = []
    for line in csv_reader:
        data.append((line[0], line[1]))
        if len(data) % batch_size == 0:
            cursor.executemany(sql, data)
            data = []
    if data:
        cursor.executemany(sql, data)
    con.commit()