【问题标题】:Uploads via ftplib in python taking signicantly longer than doing it manually在 python 中通过 ftplib 上传比手动上传花费的时间要长得多
【发布时间】:2014-01-06 18:35:45
【问题描述】:

所以,

由于某种原因,与通过 FTP 程序或网站本身上传相比,上传下面的一小段代码所需的时间要长得多。

文件达到 600kb,通过 ftp 程序/直接通过网络最多需要 1-3 秒,但通过 ftplib 需要 15-30 秒...慢了 15 倍以上

我上传的文件是单行 html/php 文件(缩小),有没有更快的上传方式...?

在我的实际代码中,我上传了 25 个以上的文件,所有文件的总大小最大为 1mb,是否有带有 .tmp 重命名的批量上传方法(防止加载部分更新的页面)对我来说会更好吗?

import ftplib

print "Uploading FULL.html"
## Login using the ftplib library and set the session as the variable ftp_session
ftp_session = ftplib.FTP('ftp.website.com','admin@website.com','Password1','',60)
## Open a file to upload
ftp_file = open('OUTPUT/DATA/FULL.html','r')
## Open a folder in the ftp server
ftp_session.cwd("/DATA/")
## Send/upload the file to the ftp server
ftp_session.storlines('STOR FULL.html.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("FULL.html.tmp", "FULL.html")

print "Uploading index.php"
## Open a file to upload
ftp_file = open('OUTPUT/index.php','r')
## Send/upload the file to the ftp server
ftp_session.storlines('STOR index.php.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("index.php.tmp", "index.php")

print "Uploading secondary.php"
## Open a file to upload
ftp_file = open('OUTPUT/secondary.php','r')
## Send/upload the file to the ftp server
ftp_session.storlines('STOR secondary.php.tmp', ftp_file)
## Close the ftp_file
ftp_file.close()
## Rename temp file
ftp_session.rename("secondary.php.tmp", "secondary.php")

## Quit out of the FTP session
ftp_session.quit()

上传输出示例....

Running...
Uploaded MAIN.html (284KB) in 9.677 seconds at 29.40 kbps
Uploaded GROUP1.html (12KB) in 0.660 seconds at 19.12 kbps
Uploaded GROUP10.html (9KB) in 1.137 seconds at 8.07 kbps
Uploaded GROUP11.html (14KB) in 1.014 seconds at 14.56 kbps
Uploaded GROUP12.html (17KB) in 0.913 seconds at 18.87 kbps
Uploaded GROUP13.html (12KB) in 0.675 seconds at 18.66 kbps
Uploaded GROUP14.html (13KB) in 0.738 seconds at 18.54 kbps
Uploaded GROUP15.html (16KB) in 0.971 seconds at 16.53 kbps
Uploaded GROUP16.html (14KB) in 1.755 seconds at 8.44 kbps
Uploaded GROUP17.html (13KB) in 1.504 seconds at 9.05 kbps
Uploaded GROUP18.html (14KB) in 0.725 seconds at 20.47 kbps
Uploaded GROUP19.html (14KB) in 1.691 seconds at 8.76 kbps
Uploaded GROUP2.html (10KB) in 0.541 seconds at 18.90 kbps
Uploaded GROUP20.html (13KB) in 0.799 seconds at 16.69 kbps
Uploaded GROUP21.html (14KB) in 0.713 seconds at 20.42 kbps
Uploaded GROUP22.html (14KB) in 0.776 seconds at 18.61 kbps
Uploaded GROUP23.html (14KB) in 0.592 seconds at 24.99 kbps
Uploaded GROUP24.html (13KB) in 0.734 seconds at 18.86 kbps
Uploaded GROUP25.html (20KB) in 0.845 seconds at 24.49 kbps
Uploaded GROUP26.html (18KB) in 0.637 seconds at 28.91 kbps
Uploaded GROUP27.html (11KB) in 0.781 seconds at 14.54 kbps
Uploaded GROUP28.html (16KB) in 0.798 seconds at 20.25 kbps
Uploaded GROUP29.html (12KB) in 0.542 seconds at 23.46 kbps
Uploaded GROUP3.html (16KB) in 0.585 seconds at 27.47 kbps
Uploaded GROUP4.html (14KB) in 0.622 seconds at 23.97 kbps
Uploaded GROUP5.html (14KB) in 0.716 seconds at 20.54 kbps
Uploaded GROUP6.html (18KB) in 1.109 seconds at 16.48 kbps
Uploaded GROUP7.html (11KB) in 0.654 seconds at 17.52 kbps
Uploaded GROUP8.html (9KB) in 0.560 seconds at 16.25 kbps
Uploaded GROUP9.html (12KB) in 0.582 seconds at 21.42 kbps
The whole routine took 72.468 seconds

【问题讨论】:

  • 尝试使用sniffer 转储ftplib 网络流量并对其进行检查和/或与手动案例进行比较。
  • @VasiliyFaronov 肯定是这样,但是使用嗅探器我找不到任何问题。我尝试了 wireshark、caspa、snoop 和其他一些,但没有任何东西告诉我出了什么问题。有没有一种简单的方法来做一些事情,比如压缩文件并上传 zip 并提取(idk 任何提取方式)
  • 我猜你速度变慢的原因是重命名。没有那个你测试了吗?

标签: python python-2.7 file-upload ftplib


【解决方案1】:

您的传输速度尽可能慢并且应用程序发送速度更快有几个原因:

  1. 您一次发送一行,ftp 应用程序将一次发送一个 8192 字节的块,
  2. 您正在以文本模式发送,应用程序可能正在以二进制压缩模式发送,

除非行尾有问题,请使用open(filename, "rb") 然后tp_session.storbinary('STOR index.php.tmp', ftp_file) 您也可以尝试发送命令'MODE COMPRESSED' 并检查回复,您可以调整块大小以找到最佳值。

我要设置的另一件事是:

import glob, os, ftplib
from time import time

sourcedir = '/where/your/files/are/locally'
destdir = '/where/to/put/them'
##    filelist = ['file2.ext', 'file2.ext', 'etc']
filelist = glob.glob(os.path.join(sourcedir, '*.php'))

ftp_session = ftplib.FTP('ftp.website.com','admin@website.com','password123','',60)
ftp_session.cwd(destdir)

for fname in filelist:
    filename = fname.replace(sourcedir+'\\', "")
    ftp_file = open(fname,'rb')
    result = ftp_session.storbinary('STOR %s.tmp' % filename, ftp_file)
    ftp_file.close()
    if 199 < int(result[0:3]) < 300: # Positive Completion
        ftp_session.rename('%s.tmp' % filename, filename)
    else:
        print "Upload of %s resulted in code %s" % (filename, result)
        break

您可以压缩或 tar.gz 文件并将其发送,但您需要安全的套接字连接和适当的权限才能单独解压。

【讨论】:

  • 在运行您的代码片段时,出现以下错误:if 199 &lt; int(result) &lt; 300: # Positive Completion ValueError: invalid literal for int() with base 10: '226-File successfully transferred\n226 13.494 seconds (measured here), 21.09 Kbytes per second'
  • 至于压缩,我在文档中找不到关于压缩的任何地方,至于在 storbinary 中发送,我在切断一些 html/php 文件时遇到了问题。
  • 对上述数字进行细微更改可能有助于解决错误 - 有关压缩的一些详细信息,请参阅en.wikipedia.org/wiki/Mode_Z_compression
  • 感谢您的细微编辑,我已编辑您的帖子以供审核,更改了一些内容,例如导入、字符串格式错误和其他一些更改。
  • 通过storlines 上传了 30 个文件,每个文件需要 60-70 秒,而 storbinary 需要 63-69 秒,在所有时间里,它们的平均上传时间是 66 秒,这是总共约 500kb。
【解决方案2】:

定义块大小以及 ftp 连接的 storbinary,因此您将获得比 FTP Filezilla 快 1.5-3.0 倍的连接速度:)

from ftplib import FTP

USER = "Your_user_id"
PASS = "Your_password"
PORT = 21
SERVER = 'ftp.billionuploads.com' #use FTP server name here

ftp = FTP()
ftp.connect(SERVER, PORT)
ftp.login(USER, PASS)

try:
    file = open(r'C:\Python27\1.jpg','rb')
    ftp.storbinary('STOR ' + '1.jpg', file,102400) #here we store file in 100kb blocksize
    ftp.quit()
    file.close()
    print "File transfered"
except:
    print "Error in File transfering"

【讨论】:

    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2017-06-29
    • 2020-09-07
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多