使用 Python 对大型文本文件进行多个正则表达式字符串替换

Question

我有一些非常大的文本文件，我想在其上执行多个基于正则表达式的字符串替换。目前我正在使用 Sublime 的类似功能来做这件事。但是，在大于 GB 的文件中，我的系统挂起。

我目前正在我的 sublime 中运行以下一些比赛

\\\n - 删除所有反斜杠后跟换行符。

\n - 删除所有换行符。

\=\\\" - 将所有 =\" 实例替换为 ="

在一种情况下，我还想对匹配进行分组并在替换文本中使用它。

我周围的一些专家建议为此编写一个快速的 python 脚本，性能不会成为问题。

以我有限的python知识，我尝试了以下方法：

import pandas as pd
import numpy as np

df = pd.read_csv('story_all.csv')

output = df.str.replace('\n', '')

output.to_csv(story_done.csv, sep='\n', encoding='utf-8')

但是，它不起作用。在我认为的某个地方，我可能做得过火了。

---

注意：文本文件是 CSV 的事实并不重要。我只需要执行一些字符串替换。 CSV 所需的新行在完成时会被保留。

---

得到的错误如下：

Traceback（最近一次调用最后一次）：文件“replace.py”，第 4 行，在 df = pd.read_csv('story_all.csv') 文件 "/Users/safwan/Envs/regex/lib/python2.7/site-packages/pandas/io/parsers.py", 第 709 行，在 parser_f 中 返回_read（filepath_or_buffer，kwds）文件“/Users/safwan/Envs/regex/lib/python2.7/site-packages/pandas/io/parsers.py”， 第 455 行，在 _read 数据 = parser.read(nrows) 文件“/Users/safwan/Envs/regex/lib/python2.7/site-packages/pandas/io/parsers.py”， 第 1069 行，已读 ret = self._engine.read(nrows) 文件“/Users/safwan/Envs/regex/lib/python2.7/site-packages/pandas/io/parsers.py”， 第 1839 行，已读 data = self._reader.read(nrows) 文件“pandas/_libs/parsers.pyx”，第 902 行，在 pandas._libs.parsers.TextReader.read 文件中 “pandas/_libs/parsers.pyx”，第 924 行，在 pandas._libs.parsers.TextReader._read_low_memory 文件 “pandas/_libs/parsers.pyx”，第 978 行，在 pandas._libs.parsers.TextReader._read_rows 文件 “pandas/_libs/parsers.pyx”，第 965 行，在 pandas._libs.parsers.TextReader._tokenize_rows 文件 “pandas/_libs/parsers.pyx”，第 2208 行，在 pandas._libs.parsers.raise_parser_error pandas.errors.ParserError： 标记数据时出错。 C 错误：预计第 8058 行中有 19 个字段，见 65

CSV 文件值示例：

id,title,name_in_english,type,water_directory_term,org_work_area_term,org_type_term,defined_state,org_location_taluka_term,org_location_state_term,org_location_village_term,org_name_term,ha_free_term,org_location_dist_term,fax,samprak_bekti,email,phoneno,website/blog,postal_address,sangathan_ke_bare_main,rajya_state,taluka_sahar,jilla_district,kisi_prakar_kaa_sangathan,name,ID,created,status
"883","some title","","org","lorem","ipsum","lorem","","","very large body field","","","","","admin","1","1230273749","1"
"884","some title","","org","lorem","ipsum","lorem","","","very large body field","","","","","admin","1","1230273749","1"
"885","some title","","org","lorem","ipsum","lorem","","","very large body field","","","","","admin","1","1230273749","1"
"886","some title","","org","lorem","ipsum","lorem","","","very large body field","","","","","admin","1","1230273749","1"

Answer 1

如果我理解正确，您可以执行以下操作。
这似乎适用于您共享的数据样本

import pandas as pd

df = pd.read_csv('story_all.csv', sep=',')

# Chars to replace
chars = [
    '\n',
]

output = df.replace(chars, '', regex=True)
output.to_csv('story_done.csv', sep=',', encoding='utf-8', index=False)

Answer 2

我终于能够在没有 pandas 帮助的情况下完成所需的任务。虽然该方法将整个文件读取到内存中，但它对于我的 MacBook Pro 上高达 1-1.5 GB 的文件效果很好。它符合我的目的。我找到了这个here 的基本代码。

# import the modules that we need. (re is for regex)
import os, re

# set the working directory for a shortcut
os.chdir('/Users/username/Code/python/regex')

# open the source file and read it
# fh = file('org.csv', 'r')
fh = file('story_all.csv', 'r')
thetext = fh.read()
fh.close()

# create the pattern object. Note the "r". In case you're unfamiliar with Python
# this is to set the string as raw so we don't have to escape our escape characters

#match all newline followed by backslash.
p1 = re.compile(r'\n\\')
# p2 = re.compile(r'\n')
#match all newline except the one followed by digits in quotes.
p2 = re.compile(r'\n+(?!\"\d+\")')
p3 = re.compile(r'\\N')
p4 = re.compile(r'\=\\\"')




# do the replace
result = p1.sub("", thetext)
result = p2.sub("", result)
result = p3.sub("", result)
result = p4.sub('="', result)

# write the file
f_out = file('done.csv', 'w')
f_out.write(result)
f_out.close()

对接近 1 GB 的文件使用时大约需要 30-40 秒。