我可以通过并行加速python文件方法`read()`吗？

Question

我有很多文件（300~500）要读取，我想加快这项任务。

理想化是：

from multiprocessing import Pool
import os
import _io

filelist = map(open,os.listdir())
if __name__ == '__main__':
    with Pool() as pool:
        a = pool.map(_io.TextIOWrapper.read,filelist)

当然，我得到了一个错误：

TypeError: cannot serialize '_io.TextIOWrapper' object

问题是：我可以通过并行来加速 I/O 过程吗？如果是，怎么做？

更新结论：

现在我找到了并行处理的方法并测试了我的代码：

我使用了 22 个项目，总计 63.2 MB

from multiprocessing import Pool
import os
import _io

def my_read(file_name):
    with open(file_name) as f:
        return f.read()

def mul():
    with Pool() as pool:
        a = pool.map(my_read, os.listdir())

def single():
    a = []
    for i in os.listdir():
        with open(i) as f:
            r = f.read()
        a.append(r)

if __name__ == '__main__':
    mul()
    # single()

遗憾的是，single() 花费 0.4 秒，而 mul() 花费 0.8 秒。

---

更新 1：

有人说这是一个IO密集型任务，所以我无法通过并行来改进它。 不过，我可以在Python doc找到这些词：

However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

---

完整代码在这里：

我的目的是将Epub转给txt。

我已经并行化了char2text，现在我想加速readall：

import zipfile
from multiprocessing import Pool

import bs4


def char2text(i):
    soup = bs4.BeautifulSoup(i)
    chapter = soup.body.getText().splitlines()
    chapter = "\n".join(chapter).strip() + "\n\n"
    return chapter


class Epub(zipfile.ZipFile):
    def __init__(self, file, mode='r', compression=0, allowZip64=False):
        zipfile.ZipFile.__init__(self, file, mode, compression, allowZip64)
        if mode == 'r':
            self.opf = self.read('OEBPS/content.opf').decode()
            opf_soup = bs4.BeautifulSoup(self.opf)
            self.author = opf_soup.find(name='dc:creator').getText()
            self.title = opf_soup.find(name='dc:title').getText()
            try:
                self.description = opf_soup.find(name='dc:description').getText()
            except:
                self.description = ''
            try:
                self.chrpattern = opf_soup.find(name='dc:chrpattern').getText()
            except:
                self.chrpattern = ''
            self.cover = self.read('OEBPS/images/cover.jpg')
        elif mode == 'w':
            pass

    def get_text(self):
        self.tempread = ""
        charlist = self.readall(self.namelist())
        with Pool() as pool:
            txtlist = pool.map(char2text, charlist)
        self.tempread = "".join(txtlist)
        return self.tempread

    def readall(self, namelist):
        charlist = []
        for i in namelist:
            if i.startswith('OEBPS/') and i.endswith('.xhtml'):
                r = self.read(i).decode()
                charlist.append(r)
        return charlist

    def epub2txt(self):
        tempread = self.get_text()
        with open(self.title + '.txt', 'w', encoding='utf8') as f:
            f.write(tempread)


if __name__ == "__main__":
    e = Epub("assz.epub")
    import cProfile
    cProfile.run("e.epub2txt()")

Answer 1

您是否尝试过类似的方法：

from multiprocessing import Pool
import os
import _io

def my_read(file_name):
    with open(file_name) as f:
        return _io.TextIOWrapper.read(f)


if __name__ == '__main__':
    with Pool() as pool:
        a = pool.map(my_read, os.listdir('some_dir'))

在子进程中打开/关闭文件对我来说听起来更合乎逻辑，并且字符串很容易序列化。

为您的 readall 方法尝试：

def readall(self, namelist):
    filter_func = lambda i: i.startswith('OEBPS/') and i.endswith('.xhtml')
    read_fun= lambda i:  self.read(i).decode()

    with Pool() as pool:
        a = pool.map(read_fun, filter(filter_func, namelist)) 
    return a