【发布时间】:2018-02-19 22:40:20
【问题描述】:
我在写一个多线程的网络爬虫,每个线程的例程函数基本上是一个死循环,并且有一个共享字典来存储那些已经访问过的url,我使用线程锁来同步.
有时我可能想用 ctrl+c 中断它,我在线程运行函数中添加一个 try catch 来捕捉键盘中断,然后我可能想做一些最终任务,例如将数据转储到数据库或 pkl 文件。
问题是每次我打断它,就是进不去catch进程,有时卡在那儿,有时还继续运行。
那么在多线程编程中处理异常/中断的最佳实践是什么?
我的代码如下:
from bs4 import BeautifulSoup
import requests
import threading
import queue
import pickle
import os
from concurrent.futures import ThreadPoolExecutor
worker_num = 8
q = queue.Queue()
lock = threading.Lock()
if os.path.exists('./checked.pkl'):
with open('./checked.pkl', 'rb') as f:
checked = pickle.load(f)
else:
checked = set()
def get_links(url):
# do sth....
def run():
print(threading.current_thread())
try:
while True:
next_url = q.get()
links = get_links(next_url)
lock.acquire()
for link in links:
if link not in checked:
q.put(link)
print(len(checked))
lock.release()
except Exception as e:
print(e)
print('interrupt')
lock.acquire()
with open('./checked.pkl', 'wb') as f:
pickle.dump(checked, f)
lock.release()
if __name__ == '__main__':
q.put(start_url)
with ThreadPoolExecutor(worker_num) as executor:
for _ in range(worker_num):
executor.submit(run)
【问题讨论】:
标签: python multithreading python-3.x