【问题标题】:Paramiko SFTP connection drops while uploading large file上传大文件时 Paramiko SFTP 连接断开
【发布时间】:2016-02-24 03:31:21
【问题描述】:

我正在尝试将一个大文件 (1GB) 上传到 SFTP,但我不断收到以下变体:

File "/venv/lib/python2.7/site-packages/paramiko/file.py", line 339, in write
    self._write_all(data)
  File "/venv/lib/python2.7/site-packages/paramiko/file.py", line 456, in _write_all
    count = self._write(data)
  File "/venv/lib/python2.7/site-packages/paramiko/sftp_file.py", line 180, in _write
    t, msg = self.sftp._read_response(req)
  File "/venv/lib/python2.7/site-packages/paramiko/sftp_client.py", line 762, in _read_response
    raise SSHException('Server connection dropped: %s' % str(e))
SSHException: Server connection dropped: 

我注意到,如果我将 MAX_REQUEST_SIZE(在 sftp_file.py 中)更新为 1024 而不是 32768,它可以工作。这是否意味着我唯一的选择是使用MAX_REQUEST_SIZE = 1024 复制/粘贴自定义版本的 sftp_file.py?还有其他人有不会减慢上传速度的建议吗?

更新:最近几次我尝试更新MAX_REQUEST_SIZE时,它最终抛出了OperationalError: SSL SYSCALL error: EOF detected错误。作为参考,这是我目前正在做的:

    transport = paramiko.Transport((hostname, port))
    transport.connect(username, password)
    sftp = paramiko.SFTPClient.from_transport(transport)
    f = sftp.open(ftp_path, 'wb')
    f.write(file_obj.read())
    f.close()
    sftp.close()
    transport.close()

【问题讨论】:

    标签: python sftp paramiko


    【解决方案1】:

    IMO 很奇怪,您必须减小块大小。 SSH 的 RFC4253 声明:

    所有实现必须能够处理带有 32768 字节或更少的未压缩有效载荷长度和总数据包 大小为 35000 字节或更少(包括“packet_length”, 'padding_length'、'payload'、'random padding' 和 'mac')。

    除了使用自定义版本的 sftp_file.py 之外,您可以做的事情是:

    sftpclient = sshclient.open_sftp()
    with sftpclient.file(remote_file_path, mode='w') as rem_file:
        rem_file.MAX_REQUEST_SIZE = 1024
        rem_file.write(data)
    

    然后玩MAX_REQUEST_SIZE(即16384、24576、...)

    【讨论】:

      【解决方案2】:

      我在使用 get 之前添加了打开/关闭,但出于某种我不知道的原因,它适用于大文件。

          file = self._connection.open(remote_path, 'rb')
          file.close()
          self._connection.get(remote_path, local_path,callback=None)
      

      【讨论】:

        【解决方案3】:

        我收到错误-“ERROR_SFTP_PULL,AUTHENTICATION ERROR 服务器连接断开”: -正常的无密码 sftp (get) 正在从我的本地服务器到远程服务器,但是 SFTP PULL 通过 Python 脚本失败。 4天前运行正常。 - 下面的代码部分看起来很糟糕:

        1.  authReturnCode = "<class 'paramiko.ssh_exception.BadAuthenticationType'>", not sure please help
        2. **errorKey = "INFO_SFTP_PULL, "+ eachFiles + " successfully copyied from SFX for " + custName**
        
        Failure log:
        ===========
        2021-04-26 15:00:02,459 INFO::ipndProxyApp.py:787: Processing data for: eDCP
        2021-04-26 15:00:02,851 INFO::transport.py:1746: Connected (version 2.0, client SSHD)
        2021-04-26 15:00:03,860 INFO::transport.py:1746: Authentication (publickey) successful!
        2021-04-26 15:00:03,922 INFO::sftp.py:158: [chan 0] Opened sftp connection (server version 3)
        2021-04-26 15:15:15,376 INFO::sftp.py:158: [chan 0] sftp session closed.
        2021-04-26 15:15:15,491 ERROR::ipndProxyApp.py:119: ERROR_SFTP_PULL, AUTHENTICATION ERROR Server connection dropped:
        
        Normal log is:
        =============
        2021-04-22 15:00:02,138 INFO::ipndProxyApp.py:782: Processing data for: eDCP
        2021-04-22 15:00:02,256 INFO::transport.py:1746: Connected (version 2.0, client SSHD)
        2021-04-22 15:00:02,563 INFO::transport.py:1746: Authentication (publickey) successful!
        2021-04-22 15:00:02,586 INFO::sftp.py:158: [chan 0] Opened sftp connection (server version 3)
        2021-04-22 15:00:09,999 INFO::sftp.py:158: [chan 0] sftp session closed.
        2021-04-22 15:00:10,003 INFO::ipndProxyApp.py:122: INFO_SFTP_PULL, FILE_2021-04-21-18-00-35.txt successfully copied from SFX for eDCP
        
        Python Code:
        ===========
        import os, sys, logging, shutil
        import pysftp, mysql.connector
        import subprocess, csv, argparse
        from datetime import datetime, timedelta
        import time
        
        # Importing Configuration settings or other supporting scripts
        import settings
        import SQLqueries
        import fieldLengthVerification
        
        # Static Configuration settings, logging settings, global variables
        logging.basicConfig(filename='/var/log/ipnd.log', format='%(asctime)s %(levelname)s::%(filename)s:%(lineno)d: %(message)s', dateformat='%Y-%m-%d %I:%M:%S',level=logging.INFO)
        currentDateTime = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        currentDate = datetime.now().strftime("%Y-%m-%d")
        yesterdays_date = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
        twoDaysBack = (datetime.now() - timedelta(days=2)).strftime("%Y-%m-%d")
        cnopts = pysftp.CnOpts()
        cnopts.hostkeys = None
        
        
        # Handling Authentication errors for the SFT authentications only
        def errorHandling(errorFromProgram):
            """
            Function to handle authentication error for sftp only, it accepts exception errors and return string keyword.
        
            Parameters:
            errorFromProgram (class) : Error class generated by Exceptions
        
            Returns:
            str: str(errorFromProgram) or "Authentication Success" or "AUTHENTICATION ERROR " + str(errorFromProgram)
            """
        
            # This function to perform error handling and return string against the provided input  
            
            authReturnCode = "<class 'paramiko.ssh_exception.BadAuthenticationType'>"
        
            # Handling RHEL version of sftp (<class 'paramiko.ssh_exception.AuthenticationException'>)
            if str(errorFromProgram) == "Authentication failed.":
                errorKeyWord = str(errorFromProgram)
                return errorKeyWord
        
            # Handling SUSE version of sftp (<class 'paramiko.ssh_exception.BadAuthenticationType'>)
            elif str(type(errorFromProgram)) == authReturnCode:
                errorExcetption,a = errorFromProgram
                errorKeyWord = errorExcetption
                return errorKeyWord
        
            # Handling other situation then provious two
            elif (str(errorFromProgram) != "Authentication failed.") or (str(type(errorFromProgram)) != authReturnCode):    
                errorKeyWord = "AUTHENTICATION ERROR " + str(errorFromProgram)  
                return errorKeyWord
        
            # Handling other conditions if not handled in provious if and elif
            else:
                        infoKeyWord = "Authentication Success"
                return infoKeyWord
        
        
        # Fetch eDCP Data files from SFX sftp server
        
        def fetchFilesFromSftpLocation(custName, fileSrcHost, remoteFileLocation, fileSrcUser, fileSrcPassPhrase, dateToProcessFiles):
            """
            Function to fetch files from remove servers and store in location directory, return nothing and accept following parameters
        
            Parameters:
            custName (str): Customer Name.
            fileSrcHost (str): Hostname (FQDN) or IP address of the source server from where files would be fetched.
            fileLocation (str): Remote file location.
            fileSrcUser (str): User to connect remote server to fetch files.
            fileSrcPassPhrase (str): Password to connect remote server to fetch files.
            dateToProcessFiles (str): Date to fetch the files for   
            """
        
            # try to connect SFX server over sftp
            try:
                sftp = pysftp.Connection(fileSrcHost, username = fileSrcUser)
        
                # change to remote directory
                sftp.chdir(remoteFileLocation)
                listOfFilesAtSFX = sftp.listdir()
        
                for eachFiles in listOfFilesAtSFX:
                    
                    if eachFiles.startswith("IPNDUP" + custName.upper() +"_" + dateToProcessFiles):
                        
                        # Fetch files from SFTP
                        try:    
                            sftp.get(eachFiles)
                                                
                            **errorKey = "INFO_SFTP_PULL, "+ eachFiles + " successfully copyied from SFX for " + custName**
        
                        except Exception as e:
                            errorKey = "ERROR_SFTP_PULL, "+ errorHandling(e)
                            #print(errorKey)
        
                sftp.close()
        
            # Capture Exception
            except Exception as e:
                errorKey = errorHandling(e)
                errorKey = "ERROR_SFTP_PULL, "+ errorKey
        
        
                # Checking conditions to generate ERROR or INFO
                if len(errorKey) > 0:
                        listOfErrorKeys = errorKey.split('_')
                        if "ERROR" in listOfErrorKeys:
                                logging.error("%s" % errorKey)
                                sys.exit(1)
                        else:
                                logging.info("%s" % errorKey)
        
                else:
                        logging.error("%s" % "No Error keywords found")
        

        【讨论】:

          猜你喜欢
          • 2014-08-15
          • 2017-05-29
          • 2021-05-15
          • 2021-04-26
          • 2016-09-07
          • 2017-01-17
          • 2019-10-04
          • 1970-01-01
          • 1970-01-01
          相关资源
          最近更新 更多