循环中的python多线程答案

【问题标题】：python multithreading in loops循环中的python多线程
【发布时间】：2017-02-19 00:58:04
【问题描述】：

我想我对线程的工作原理有一个非常基本的了解，但由于我不太了解，所以我无法弄清楚这一点。我想要一个大约 10 个线程的池限制，但棘手的部分是我不知道如何让它逐行读取。

proxies = {
    'http': 'http://123.10.210.213:9999',
    'https': 'http://123.10.210.213:9999'
}


def create_proxy_lst(txt):
    print("""
########################################
#       WORKING    |      NOT WORKING  #
########################################
        """)
    proxy_list = []
    with open(txt) as f:
        for line in f:
            proxy_list.append(line.strip('\n'))
    return proxy_list


def check_proxy(website="https://google.com/"):
    working = 0
    not_working = 0
    total = 0
    lst = create_proxy_lst("uncheckedproxys.txt")
    for proxy in lst:
        try:
            proxies["https"] = "http://" + proxy
            proxies["http"] = "http://" + proxy
            r = requests.get(website, timeout=1, proxies=proxies)
            if r.status_code == 200:
                print("%s" % proxy)
                working += 1
                total += 1
                os.system("title Working: " + str(working) + "\t     Not working " + str(not_working) + "     ✔" + "    Total:   " + str(total) + "/" + str(len(lst)))
        except Exception:
            print("\t\t    %s" % proxy)
            not_working += 1
            total += 1
            os.system("title Working: " + str(working) + "\t     Not working " + str(not_working) + "     ✖" + "    Total:   " + str(total) + "/" + str(len(lst)))

【问题讨论】：

标签： python multithreading for-loop proxy python-requests

【解决方案1】：

将你的代理放入一个Queue.Queue()，然后启动10个线程从队列中读取代理。

在你的情况下：

from Queue import Queue
from threading import Thread

def worker(proxy_queue):
    while not proxy_queue.empty():
        proxy = proxy_queue.get()

        working = 0
        not_working = 0
        total = 0

        try:
            proxies["https"] = "http://" + proxy
            proxies["http"] = "http://" + proxy
            r = requests.get(website, timeout=1, proxies=proxies)
            if r.status_code == 200:
                print("%s" % proxy)
                working += 1
                total += 1
                os.system("title Working: " + str(working) + "\t     Not working " + str(not_working) + "     ✔" + "    Total:   " + str(total) + "/" + str(len(lst)))
        except Exception:
            print("\t\t    %s" % proxy)
            not_working += 1
            total += 1
            os.system("title Working: " + str(working) + "\t     Not working " + str(not_working) + "     ✖" + "    Total:   " + str(total) + "/" + str(len(lst)))

if __name__ == '__main__':
    # Build a queue
    proxy_queue = Queue()

    # Put these proxies into the queue
    with open("uncheckedproxys.txt") as f:
        for line in f:
            proxy_queue.put(line.strip())

    # Create thread pool
    thread_pool = [Thread(target=worker, args=proxy_queue) for i in range(10)]

    # Start threads
    for thread in thread_pool:
        thread.start()

【讨论】：