Python 3：如何从多个进程写入同一个文件而不弄乱它？

Question

我有一个可以随时启动或停止的程序。该程序用于从网页下载数据。首先，用户将在.csv 文件中定义一堆网页，然后保存.csv 文件，然后启动程序。该程序将读取.csv 文件并将其转换为作业列表。接下来，作业被分成 5 个独立的 downloader 函数，这些函数并行工作，但下载时间可能不同。

downloader（共有 5 个）完成下载网页后，我需要它打开 .csv 文件并删除链接。这样，随着时间的推移，.csv 文件会越来越小。问题是有时两个download 函数会尝试同时更新.csv 文件并导致程序崩溃。我该如何处理？

Answer 1

如果这是您的project from yesterday 的延续，则您的下载列表已经在内存中 - 只需在其进程完成下载时从加载列表中删除条目，然后仅在输入文件上写下整个列表退出“下载​​器”。没有理由不断写下更改。

如果您想知道（例如从外部进程）某个 url 何时被下载，即使您的“下载器”正在运行，请在每次进程返回下载成功时在 downloaded.dat 中写入一个新行。

当然，在这两种情况下，都从主进程/线程中写入，这样您就不必担心互斥锁。

更新 - 以下是使用与昨天相同的代码库的附加文件的方法：

def init_downloader(params):  # our downloader initializator
    downloader = Downloader(**params[0])  # instantiate our downloader
    downloader.run(params[1])  # run our downloader
    return params  # job finished, return the same params for identification

if __name__ == "__main__":  # important protection for cross-platform use

    downloader_params = [  # Downloaders will be initialized using these params
        {"port_number": 7751},
        {"port_number": 7851},
        {"port_number": 7951}
    ]
    downloader_cycle = cycle(downloader_params)  # use a cycle for round-robin distribution

    with open("downloaded_links.dat", "a+") as diff_file:  # open your diff file
        diff_file.seek(0)  # rewind the diff file to the beginning to capture all lines
        diff_links = {row.strip() for row in diff_file}  # load downloaded links into a set
        with open("input_links.dat", "r+") as input_file:  # open your input file
            available_links = []
            download_jobs = []  # store our downloader parameters + a link here
            # read our file line by line and filter out downloaded links
            for row in input_file:  # loop through our file
                link = row.strip()  # remove the extra whitespace to get the link
                if link not in diff_links:  # make sure link is not already downloaded
                    available_links.append(row)
                    download_jobs.append([next(downloader_cycle), link])
            input_file.seek(0)  # rewind our input file
            input_file.truncate()  # clear out the input file
            input_file.writelines(available_links)  # store back the available links
            diff_file.seek(0)  # rewind the diff file
            diff_file.truncate()  # blank out the diff file now that the input is updated
        # and now let's get to business...
        if download_jobs:
            download_pool = Pool(processes=5)  # make our pool use 5 processes
            # run asynchronously so we can capture results as soon as they ar available
            for response in download_pool.imap_unordered(init_downloader, download_jobs):
                # since it returns the same parameters, the second item is a link
                # add the link to our `diff` file so it doesn't get downloaded again
                diff_file.write(response[1] + "\n")
        else:
            print("Nothing left to download...")

正如我在评论中所写的那样，整个想法是在下载链接时使用文件来存储下载的链接，然后在下次运行时过滤掉下载的链接并更新输入文件。这样即使你强行杀死它，它也总是会从中断的地方继续（除了部分下载）。

Answer 2

使用多处理库中的“锁”对文件进行序列化操作。

您需要将锁传递给每个进程。每个进程都应该在打开文件之前“获取”锁，并在关闭文件后“释放”锁。

https://docs.python.org/2/library/multiprocessing.html

Answer 3

查看python中的锁定文件。锁定文件将使下一个进程等到文件解锁才能修改它。锁定文件是特定于平台的，因此您必须使用适用于您所在操作系统的任何方法。如果您需要弄清楚操作系统，请使用这样的 switch 语句。

import os

def my_lock(f):
    if os.name == "posix":
        # Unix or OS X specific locking here
    elif os.name == "nt":
        # Windows specific locking here
    else:
        print "Unknown operating system, lock unavailable"

然后我会查看this article 并弄清楚你想如何实现你的锁。