NodeJS ssh2 fastGet 解压挂起答案

【问题标题】：NodeJS ssh2 fastGet hangs on decompressionNodeJS ssh2 fastGet 解压挂起
【发布时间】：2016-04-30 07:34:20
【问题描述】：

使用 mscdex 的 nodejs ssh2 库，我有一个 sftp“获取”节点脚本，它将列出 ssh 服务器目录的内容，搜索特定文件，然后下载它们。当脚本到达调用“sftp.fastGet”的下载步骤时，提供给 fastGet 的回调永远不会被调用。

我在 ssh 连接选项中开启了“DEBUG”设置。似乎 sftp.fastGet 正在执行，但正在执行“解压缩”步骤。这是我得到的输出：

...
DEBUG: Parser: Verifying MAC
DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
DEBUG: Parser: Decompressing
// hangs here forever (or until timeout)...

更奇怪的是，好像有几个包过来，解压正确。这是（更详细的）调试：

info: Fetching xxx from /xxx/xxx/xxx to X:\xxx\xxx\xxx
debug: DEBUG[SFTP]: Outgoing: Writing OPEN
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG: Parser: IN_PACKET
debug: DEBUG: Parser: Decrypting
debug: DEBUG: Parser: pktLen:20,padLen:5,remainLen:16
debug: DEBUG: Parser: IN_PACKETDATA
debug: DEBUG: Parser: Decrypting
debug: DEBUG: Parser: HMAC size:20
debug: DEBUG: Parser: IN_PACKETDATAVERIFY
debug: DEBUG: Parser: Verifying MAC
debug: DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
debug: DEBUG: Parser: Decompressing
// doesn't seem to hang here
debug: DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Parser: Response: HANDLE
debug: DEBUG[SFTP]: Outgoing: Writing FSTAT
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG: Parser: IN_PACKETBEFORE (expecting 8)
debug: DEBUG: Parser: IN_PACKET
debug: DEBUG: Parser: Decrypting
debug: DEBUG: Parser: pktLen:28,padLen:10,remainLen:24
debug: DEBUG: Parser: IN_PACKETDATA
debug: DEBUG: Parser: Decrypting
debug: DEBUG: Parser: HMAC size:20
debug: DEBUG: Parser: IN_PACKETDATAVERIFY
debug: DEBUG: Parser: Verifying MAC
debug: DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
debug: DEBUG: Parser: Decompressing
// doesn't seem to hang here
debug: DEBUG: Parser: IN_PACKETDATAAFTER, packet: CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Parser: Response: ATTRS
debug: DEBUG: Parser: IN_PACKETBEFORE (expecting 8)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG[SFTP]: Outgoing: Writing READ
debug: DEBUG: Outgoing: Writing CHANNEL_DATA (0)
debug: DEBUG: Parser: IN_PACKET
debug: DEBUG: Parser: Decrypting
debug: DEBUG: Parser: pktLen:2388,padLen:6,remainLen:2384
debug: DEBUG: Parser: IN_PACKETDATA
debug: DEBUG: Parser: Decrypting
debug: DEBUG: Parser: HMAC size:20
debug: DEBUG: Parser: IN_PACKETDATAVERIFY
debug: DEBUG: Parser: Verifying MAC
debug: DEBUG: Parser: IN_PACKETDATAVERIFY (Valid HMAC)
debug: DEBUG: Parser: Decompressing
// hangs here forever...

我破解打开“[我的项目]\node_modules\ssh2-streams\lib\ssh.js”，发现“decompress.instance.flush”方法回调不是每次都被调用。

ssh.js line 544: ...
} else if (instate.status === IN_PACKETDATAAFTER) {
  if (decompress.instance) {
  if (!decomp) {
    debug('DEBUG: Parser: Decompressing');
    decompress.instance.write(instate.payload);
    // this function executes and calls the method below
    decompress.instance.flush(Z_PARTIAL_FLUSH, function(){
      // this callback function is called during the first two iterations,
      // but is not called the last time, when the process hangs
      instate.payload = decompress.instance.read();
      var nextSlice;
      if (i === chlen)
        nextSlice = EMPTY_BUFFER;
      else
        nextSlice = chunk.slice(i);
      self._transform(nextSlice, encoding, callback, true);
    });
    return;
  } else {
  ...

...当然，这是我正在运行的代码

var sshClient = require('ssh2').Client;

var client = new sshClient();

client.on('ready', ()=> {

  client.sftp( (sftpErr, sftp) => {

    sftp.readdir(remotepath, (dirErr, files) => {

      var validFiles = files.filter( (file) => {
        return file.filename.match(regex);
      });  

      async.eachSeries(validFiles, (ftpFile, cb) => {

        var remote = remotepath + ftpFile.filename;
        var local = path.join(localpath, ftpFile.filename);


        console.log('Fetching ' + ftpFile.filename + ' from ' + remote + ' to ' + local);

        sftp.fastGet(remote, local, (getErr) => {
          console.log('Fast Get Complete');
          // this is never called
        });

      });

    })

  });

});

client.connect({
  host: "xxx.xxx.xxx.xxx",
  port: 22,
  username: "someuser",
  password: "somepass",
  debug: console.log,
  algorithms: {
    key: [
      "diffie-hellman-group1-sha1",
    ],
    cipher: [
      "blowfish-cbc",
      "3des-cbc"
    ],
    compress: [
      "zlib"
    ],
    hmac: [
      "hmac-sha1",
      "hmac-md5"
    ]
  }
});

【问题讨论】：

标签： node.js ssh sftp freeze compression

【解决方案1】：

只有在从 zlib 流中读取所有（当前）未压缩数据后，才会调用 flush() 回调。当未压缩的数据长度超过 zlib 流的highWaterMark 时，这只是一个问题。现在应该从mscdex/ssh2-streams@8c0906c6b8 开始修复。

【讨论】：