递归比较两个目录以确保它们具有相同的文件和子目录

Question

据我观察filecmp.dircmp 是递归的，但不足以满足我的需求，至少在 py2.我想比较两个目录及其所有包含的文件。这是否存在，或者我需要构建（例如使用os.walk）。我更喜欢预先构建的，其他人已经完成了单元测试:)

如果有帮助的话，实际的“比较”可能是草率的（例如，忽略权限）。

我想要一些布尔值，report_full_closure 是打印报告。它也只会出现在常见的子目录中。 AFIAC，如果他们在左侧或右侧目录中有任何内容仅，则它们是不同的目录。我改用os.walk 构建它。

Answer 1

根据@Mateusz Kobos 当前接受的答案，事实证明第二个filecmp.cmpfiles 和shallow=False 不是必需的，因此我们将其删除。可以从第一个dircmp 获得dirs_cmp.diff_files。一个常见的误解（我们也犯过！）是dir_cmp 只是肤浅的，不比较文件内容！事实证明这不是真的！ shallow=True 的意思只是为了节省时间，实际上并不认为两个最后修改时间不同的文件是不同的。如果两个文件的最后修改时间不同，则进入读取每个文件的内容并比较它们的内容。如果内容相同，即使最后修改日期不同，也是匹配的！我们在这里添加了详细的打印以增加清晰度。如果您想将st_modtime 中的差异视为不匹配，请参阅其他地方 (filecmp.cmp() ignoring differing os.stat() signatures?)。我们还改为使用更新的 pathlib 而不是 os 库。

import filecmp
from pathlib import Path

def compare_directories_recursive(dir1:Path, dir2:Path,verbose=True):
"""
Compares two directories recursively. 
First, file counts in each directory are compared. 
Second, files are assumed to be equal if their names, size and last modified date are equal (aka shallow=True in python terms)
If last modified date is different, then the contents are compared by reading each file. 
Caveat: if the contents are equal and last modified is NOT equal, files are still considered equal! 
This caveat is the default python filecmp behavior as unintuitive as it may seem.

@param dir1: First directory path
@param dir2: Second directory path
"""

dirs_cmp = filecmp.dircmp(str(dir1), str(dir2))
if len(dirs_cmp.left_only)>0:
    if verbose:
        print(f"Should not be any more files in original than in destination left_only: {dirs_cmp.left_only}")
    return False
if len(dirs_cmp.right_only)>0:
    if verbose:
        print(f"Should not be any more files in destination than in original right_only: {dirs_cmp.right_only}")
    return False
if len(dirs_cmp.funny_files)>0:
    if verbose:
        print(f"There should not be any funny files between original and destination. These file(s) are funny {dirs_cmp.funny_files}")
    return False
if len(dirs_cmp.diff_files)>0:
    if verbose:
        print(f"There should not be any different files between original and destination. These file(s) are different {dirs_cmp.diff_files}")
    return False

for common_dir in dirs_cmp.common_dirs:
    new_dir1 = Path(dir1).joinpath(common_dir)
    new_dir2 = Path(dir2).joinpath(common_dir)
    if not compare_directories_recursive(new_dir1, new_dir2):
        return False
return True

Answer 2

致任何寻找简单库的人：

https://github.com/mitar/python-deep-dircmp

DeepDirCmp 基本上是 filecmp.dircmp 的子类，并显示与 diff -qr dir1 dir2 相同的输出。

用法：

from deep_dircmp import DeepDirCmp

cmp = DeepDirCmp(dir1, dir2)
if len(cmp.get_diff_files_recursive()) == 0:
    print("Dirs match")
else:
    print("Dirs don't match")

Answer 3

这个递归函数似乎对我有用：

def has_differences(dcmp):
    differences = dcmp.left_only + dcmp.right_only + dcmp.diff_files
    if differences:
        return True
    return any([has_differences(subdcmp) for subdcmp in dcmp.subdirs.values()])

假设我没有忽略任何事情，如果你想知道目录是否相同，你可以直接否定结果：

from filecmp import dircmp

comparison = dircmp("dir1", "dir2")
same = not has_differences(comparison)

Answer 4

这是一个带有递归函数的简单解决方案：

import filecmp

def same_folders(dcmp):
    if dcmp.diff_files or dcmp.left_only or dcmp.right_only:
        return False
    for sub_dcmp in dcmp.subdirs.values():
        if not same_folders(sub_dcmp):
            return False
    return True

same_folders(filecmp.dircmp('/tmp/archive1', '/tmp/archive2'))

Answer 5

既然你想要的是 True 或 False 结果，如果你安装了diff：

def are_dir_trees_equal(dir1, dir2):
    process = Popen(["diff", "-r", dir1, dir2], stdout=PIPE)
    exit_code = process.wait()
    return not exit_code

Answer 6

这将检查文件是否位于相同的位置以及它们的内容是否相同。它不会正确验证空子文件夹。

import filecmp
import glob
import os

path_1 = '.'
path_2 = '.'

def folders_equal(f1, f2):
    file_pairs = list(zip(
        [x for x in glob.iglob(os.path.join(f1, '**'), recursive=True) if os.path.isfile(x)],
        [x for x in glob.iglob(os.path.join(f2, '**'), recursive=True) if os.path.isfile(x)]
    ))

    locations_equal = any([os.path.relpath(x, f1) == os.path.relpath(y, f2) for x, y in file_pairs])
    files_equal = all([filecmp.cmp(*x) for x in file_pairs]) 

    return locations_equal and files_equal

folders_equal(path_1, path_2)

Answer 7

基于python issue 12932 和filecmp documentation，您可以使用以下示例：

import os
import filecmp

# force content compare instead of os.stat attributes only comparison
filecmp.cmpfiles.__defaults__ = (False,)

def _is_same_helper(dircmp):
    assert not dircmp.funny_files
    if dircmp.left_only or dircmp.right_only or dircmp.diff_files or dircmp.funny_files:
        return False
    for sub_dircmp in dircmp.subdirs.values():
       if not _is_same_helper(sub_dircmp):
           return False
    return True

def is_same(dir1, dir2):
    """
    Recursively compare two directories
    :param dir1: path to first directory 
    :param dir2: path to second directory
    :return: True in case directories are the same, False otherwise
    """
    if not os.path.isdir(dir1) or not os.path.isdir(dir2):
        return False
    dircmp = filecmp.dircmp(dir1, dir2)
    return _is_same_helper(dircmp)

Answer 8

filecmp.dircmp 是要走的路。但它不会比较在两个比较目录中找到的具有相同路径的文件的内容。相反，filecmp.dircmp 只查看文件属性。由于dircmp 是一个类，您可以使用dircmp 子类修复它并覆盖其比较文件的phase3 函数，以确保比较内容而不是仅比较os.stat 属性。

import filecmp

class dircmp(filecmp.dircmp):
    """
    Compare the content of dir1 and dir2. In contrast with filecmp.dircmp, this
    subclass compares the content of files with the same path.
    """
    def phase3(self):
        """
        Find out differences between common files.
        Ensure we are using content comparison with shallow=False.
        """
        fcomp = filecmp.cmpfiles(self.left, self.right, self.common_files,
                                 shallow=False)
        self.same_files, self.diff_files, self.funny_files = fcomp

然后你可以用它来返回一个布尔值：

import os.path

def is_same(dir1, dir2):
    """
    Compare two directory trees content.
    Return False if they differ, True is they are the same.
    """
    compared = dircmp(dir1, dir2)
    if (compared.left_only or compared.right_only or compared.diff_files 
        or compared.funny_files):
        return False
    for subdir in compared.common_dirs:
        if not is_same(os.path.join(dir1, subdir), os.path.join(dir2, subdir)):
            return False
    return True

如果您想重用此代码 sn-p，特此专用于公共领域或您选择的知识共享 CC0（除了 SO 提供的默认许可 CC-BY-SA）。

Answer 9

比较dir1和dir2的布局的另一种解决方案，忽略文件的内容

在此处查看要点：https://gist.github.com/4164344

编辑：这是代码，以防万一由于某种原因丢失了要点：

import os

def compare_dir_layout(dir1, dir2):
    def _compare_dir_layout(dir1, dir2):
        for (dirpath, dirnames, filenames) in os.walk(dir1):
            for filename in filenames:
                relative_path = dirpath.replace(dir1, "")
                if os.path.exists( dir2 + relative_path + '\\' +  filename) == False:
                    print relative_path, filename
        return

    print 'files in "' + dir1 + '" but not in "' + dir2 +'"'
    _compare_dir_layout(dir1, dir2)
    print 'files in "' + dir2 + '" but not in "' + dir1 +'"'
    _compare_dir_layout(dir2, dir1)


compare_dir_layout('xxx', 'yyy')

Answer 10

def same(dir1, dir2):
"""Returns True if recursively identical, False otherwise

"""
    c = filecmp.dircmp(dir1, dir2)
    if c.left_only or c.right_only or c.diff_files or c.funny_files:
        return False
    else:
        safe_so_far = True
        for i in c.common_dirs:
            same_so_far = same_so_far and same(os.path.join(frompath, i), os.path.join(topath, i))
            if not same_so_far:
                break
        return same_so_far

Answer 11

这里有一个使用filecmp 模块的比较函数的替代实现。它使用递归而不是os.walk，所以它更简单一些。但是，它不会简单地通过使用common_dirs 和subdirs 属性来递归，因为在这种情况下，我们将隐式使用文件比较的默认“浅”实现，这可能不是您想要的。在下面的实现中，当比较同名文件时，我们总是只比较它们的内容。

import filecmp
import os.path

def are_dir_trees_equal(dir1, dir2):
    """
    Compare two directories recursively. Files in each directory are
    assumed to be equal if their names and contents are equal.

    @param dir1: First directory path
    @param dir2: Second directory path

    @return: True if the directory trees are the same and 
        there were no errors while accessing the directories or files, 
        False otherwise.
   """

    dirs_cmp = filecmp.dircmp(dir1, dir2)
    if len(dirs_cmp.left_only)>0 or len(dirs_cmp.right_only)>0 or \
        len(dirs_cmp.funny_files)>0:
        return False
    (_, mismatch, errors) =  filecmp.cmpfiles(
        dir1, dir2, dirs_cmp.common_files, shallow=False)
    if len(mismatch)>0 or len(errors)>0:
        return False
    for common_dir in dirs_cmp.common_dirs:
        new_dir1 = os.path.join(dir1, common_dir)
        new_dir2 = os.path.join(dir2, common_dir)
        if not are_dir_trees_equal(new_dir1, new_dir2):
            return False
    return True

Answer 12

这是我的解决方案：gist

def dirs_same_enough(dir1,dir2,report=False):
    ''' use os.walk and filecmp.cmpfiles to
    determine if two dirs are 'same enough'.

    Args:
        dir1, dir2:  two directory paths
        report:  if True, print the filecmp.dircmp(dir1,dir2).report_full_closure()
                 before returning

    Returns:
        bool

    '''
    # os walk:  root, list(dirs), list(files)
    # those lists won't have consistent ordering,
    # os.walk also has no guaranteed ordering, so have to sort.
    walk1 = sorted(list(os.walk(dir1)))
    walk2 = sorted(list(os.walk(dir2)))

    def report_and_exit(report,bool_):
        if report:
            filecmp.dircmp(dir1,dir2).report_full_closure()
            return bool_
        else:
            return bool_

    if len(walk1) != len(walk2):
        return false_or_report(report)

    for (p1,d1,fl1),(p2,d2,fl2) in zip(walk1,walk2):
        d1,fl1, d2, fl2 = set(d1),set(fl1),set(d2),set(fl2)
        if d1 != d2 or fl1 != fl2:
            return report_and_exit(report,False)
        for f in fl1:
            same,diff,weird = filecmp.cmpfiles(p1,p2,fl1,shallow=False)
            if diff or weird:
                return report_and_exit(report,False)

    return report_and_exit(report,True)

Answer 13

dircmp 可以递归：参见report_full_closure。

据我所知dircmp 不提供目录比较功能。不过，自己编写会很容易；在dircmp 上使用left_only 和right_only 检查目录中的文件是否相同，然后递归subdirs 属性。

Answer 14

report_full_closure() 方法是递归的：

comparison = filecmp.dircmp('/directory1', '/directory2')
comparison.report_full_closure()

编辑：在OP的编辑之后，我想说最好只使用filecmp中的其他功能。我认为os.walk 是不必要的；最好简单地通过 common_dirs 等生成的列表进行递归，尽管在某些情况下（大型目录树），如果实施不当，这可能会导致 Max Recursion Depth 错误。