Python subprocess.popen 和 rdiff-backup

Question

我想在 python 中创建一个 rdiff-backup 包装程序，用于将 windows 机器备份到 linux 服务器。

我想在包装程序中处理 rdiff-backup 的输出。但是当使用 subprocess 模块执行 rdiff-backup 并将 stdout 和 stderr 输送到包装器时，stdout 总是以管道的结尾结束。

在不使用管道传输标准输出的情况下使用子进程时，标准错误会在控制台中以正确的顺序打印。

当我在本地使用 rdiff-backup 而不是使用 ssh 管道时，我还注意到，stdout 和 stderr 的顺序是正确的。 Rdiff-backup 还使用 subprocess.popen 打开 ssh 会话并将数据通过管道传输到服务器。我怀疑由于某种原因 stdout get 被阻止，直到 ssh 会话关闭。

这是我的代码，这是真实程序使用线程监听标准输出的简化版本：

import sys
import subprocess

class Rdiffbackup(object):
    def __init__(self):
        #self.io_q = Queue()
        self.exe = 'F:\\workspace\\pysubprocess\\bin\\rdiff-backup\\rdiff-backup.exe'

        self.verbosity = '-v5'

        self.ssh_exe = './bin/openssh/bin/ssh'
        self.quiet = '-q'
        self.compression = '-C'
        self.port = '-p 5555'
        self.key = '-i ./keys/rdiffbackup'
        self.options = '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
        self.remote_schema = self.ssh_exe +  ' ' + self.quiet + ' ' + self.compression + ' ' + self.port + ' ' + self.key + ' ' + self.options + ' %s rdiff-backup --server'  

    def start(self,source,dest):
        with subprocess.Popen([self.exe,self.verbosity,'--remote-schema',self.remote_schema,source,dest],stdout=subprocess.PIPE,stderr=subprocess.STDOUT) as self.proc:     
            for line in self.proc.stdout:
                sys.stdout.write(line.decode("utf-8"))      

if __name__ == '__main__':
    rdb = Rdiffbackup()
    source = "C:/Users/vdrmrt/Desktop/data"    
    dest = "vdrmrt@hostname::backup"
    rdb.start(source,dest)

输出：

-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

正确的输出：

Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
  Access control lists                         Off
  Extended attributes                          Off
  Windows access control lists                 On
  Case sensitivity                             Off
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
  Ownership changing                           Off
  Hard linking                                 On
  fsync() directories                          On
  Directory inc permissions                    On
  High-bit permissions                         On
  Symlink permissions                          Off
  Extended filenames                           On
  Windows reserved filenames                   Off
  Access control lists                         On
  Extended attributes                          On
  Windows access control lists                 Off
  Case sensitivity                             On
  Escape DOS devices                           Off
  Escape trailing spaces                       Off
  Mac OS X style resource forks                Off
  Mac OS X Finder information                  Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup

Answer 1

我终于能够解决问题了。
Rdiff-backup 缓冲标准输出和标准错误，当它通过管道传输到另一个程序时。 解决方案是使用额外的 py2exe 选项重建 rdiff-backup：'unbuffered': True
在使用 Windows 7 构建时，我还必须添加一个额外的选项以使其正常工作。
Py2exe - win32api.pyc ImportError DLL load failed

这是来自 rdiff-backup 构建的最终 setup.py 的 sn-p。

if '--single-file' in sys.argv[1:]:
            sys.argv.remove('--single-file')
            extra_options.update({
                'options': {'py2exe': {'bundle_files': 1,
                                       'unbuffered': True,
                                       'dll_excludes': [ "mswsock.dll", "powrprof.dll" ]}},
                'zipfile': None
            })

Answer 2

尝试为stdout 和stderr 使用单独的管道。先阅读stdout 行，然后阅读stderr 中的行。