用于下载的 Python 多线程

Question

当时我需要下载 2 个不同的文件。

import urllib
urllib.urlretrieve('http://example.com/test1.zip', "C:\Test\test1.zip")
urllib.urlretrieve('http://example.com/test2.zip', "C:\Test\test2.zip")

我需要同时下载它们。 谢谢

Answer 1

您应该能够为此使用标准的 python 线程。您需要为每次下载创建一个代表单独线程的类，然后启动每个线程。

类似：

import urllib
from threading import Thread

class Downloader(Thread):
    def __init__(self, file_url, save_path):
        self.file_url = file_url
        self.save_path = save_path


    def run():
        urllib.urlretrieve(self.file_url, self.file_path)


Downloader('http://example.com/test1.zip', "C:\Test\test1.zip").start()
Downloader('http://example.com/test2.zip', "C:\Test\test2.zip").start()

把这个写在我的脑海里，所以不能保证有效。关键是继承Thread类并重写run()函数，然后启动线程。

关于 Dave_750 的回答，线程在这种情况下实际上是有效的，因为下载操作是 I/O，而不是解释 Python 代码，所以 GIL 并不是真正的问题。例如，我使用线程来发送多封电子邮件。

Answer 2

使用 multiprocessing.pool。我经常为此目的使用它，并且效果很好。线程不会给你带来太多好处，因为 Gil 仍然一次只允许一个工作

Answer 3

#import libraries for multithreaded applications
from multiprocessing import Pool

#import libraries for including requests.urlretrieve() function
from urllib import request

#import datetime libraries for date/ time operations
import datetime as dt

#import OS libraries for file and directory operations
import os

#import all functions from grib_downloader.py file - for testing purposes

#function to download a file from FTP link (ftp path) and save it at file_path location.
def download_func(args):
    ftp_path, file_path = args #args is a list with two values which unpacks into the variables ftp_path and file_path respectively
    request.urlretrieve(ftp_path, file_path) #python inbuilt function, part of urllib.requests library. needs an ftp/http url and a path/filename to save the downloaded file

    return ' '.join([ftp_path, ' file saved at location ', file_path])#function returns the url and the location where the file is saved as a string

#defines the main function that implements multitasking routines
def main():
#defines variables that are used for defining the download path for multiple files. in my case, I was downloading a bunch of weather GRIB2 files from an FTP site.

    parameters = ['CB', 'ICE', 'CAT', 'CB_0.25', 'ICE_0.25', 'TURB_0.25'];
    forecast_hour = "1200";
    t_folder = ['T+06', 'T+09', 'T+12', 'T+15', 'T+18', 'T+21', 'T+24', 'T+27', 'T+30', 'T+33', 'T+36'];
    
#create two blank lists to store the urls of the files to be downloaded and the path/filenames where they need to be saved in the computer
    iterable = [] # list to store the urls
    args = [] # list to store the download filenames
    new_iterable = []
#for loop to create a list of urls that comprise all files that need to be downloaded and filenames with which they need to be saved by using the listvariable.append() function
    for param in parameters:
        for folder in t_folder:
            iterable.append('ftp://sa1indsds01:kocac0fo@sadisftp.metoffice.gov.uk/GRIB2/COMPRESSED/EGRR/' + param + '/'
                            + folder + '/' + folder + '_' + forecast_hour)
            args.append('D:\\WINDTEMP\\'+ param + '\\' + folder +'_'+ forecast_hour)
    print (iterable)
    print (args)
#for loop to create a new list which is a 2D array comprising the iterable[] and args[] lists as made above. the range is 66 as there were a total of 66 files that i wanted to download. this can be changed as per ones requirement
    for i in range(0, 66):
        new_iterable.append((iterable[i], args[i]))
    print(new_iterable)# prints to check if the new list is made as required
#multitasking magic happens after this. using the imap_unordered(func_name, iterable_arguments) feature of python multiprocessing library
    n_core = 8
    p = Pool(n_core) #declare object p as instance of class Pool()
    for r in p.imap_unordered(download_func, new_iterable):# do not change this part of the code as it is where the multitasking occurs
        print(f"{r}. download successful")
#call the main() function
if __name__ == '__main__':
    main()