如何在python中多线程发布请求答案

【问题标题】：How to multithread post request in python如何在python中多线程发布请求
【发布时间】：2021-08-06 09:02:12
【问题描述】：

我正在尝试使用 python 多线程发布请求以将哈希发送到malwarebazaar。第一次我正在从文件中读取哈希值，然后我试图对其进行多线程处理，但我被卡住了我的代码没有使用所有的哈希，哪里出错了？

import re
import json
import sys
import requests
from time import time
from concurrent.futures import ThreadPoolExecutor, as_completed
f=open('hashes.txt')
lines= f.readlines()
url='https://mb-api.abuse.ch/api/v1'
array=[]
for line in lines:
    line=line.replace('\n','')
    data={'query':'get_info','hash':line}
    array.append(data)
#print(array)

#start=time()
print(url)
def function(url):
        html = requests.post(url, data=array,stream=True)
        print(html.json())
        return html

start = time()

processes = []
with ThreadPoolExecutor(max_workers=10) as executor:
    for url in url:
     processes.append(executor.submit(function, url))

#for task in as_completed(processes):
#   print(task.result())

print(f'Time taken: {time() - start}')
#c = Counter(array)
#print(c)

【问题讨论】：

你还没有解释发生了什么问题
function 中的异常不在此代码中处理，因为从不轮询生成的期货。取消注释包含 task.result() 的循环以查看问题所在。
你怎么用stream=True？如果在发出请求时将流设置为 True，则除非您消耗所有数据或调用 Response.close，否则 Requests 无法将连接释放回池中。这可能导致连接效率低下。如果您发现自己在使用 stream=True 时部分读取了请求正文（或根本不读取它们），您应该在 with 语句中发出请求以确保它始终关闭。示例：with requests.get(url, data=array, stream=True) as r:
@roganjosh 它从文件print(url) def function(url): html = requests.post(url, data=x,stream=True) print(html.json()) return html中获取随机哈希值
@user56700 我从我的请求中得到了答案，但它需要随机散列 print(url) def function(url): html = requests.post(url, data=x,stream=True) print(html.json()) return html 和 processes = [] with ThreadPoolExecutor(max_workers=10) as executor: for x in array: processes.append(executor.submit(function, url))

标签： python multithreading python-multithreading

【解决方案1】：

这就是我的工作方式：

测试数据（hashes.txt）

bd4db5d00ba4633516169666635be7dee18a1916585a32def38498c0062b48a7
92b2ba8088561d9b67f64ee433b0f9a82599ad194672ab15f689ded2ed4d3c51
80a606be75ab17bfd2096e6f1efbb0df1968e72b4337cff0a0fc3016b088e794
b164e9f4c24d002fe0d8975972e569b3873d0d8ae3e6075f548498e261874b42
f676742212a35929267bfd3750a0bbd5609de0cc2ad43955331d2b3f27af6e8f

代码：

import re, sys, requests, time, concurrent.futures
url='https://mb-api.abuse.ch/api/v1'
array=[]
with open("hashes.txt") as f:
    for line in f:
        array.append({'query':'get_info','hash':line.rstrip("\n")})

def function(payload):
    with requests.post(url, data=payload,stream=True) as response:
        html = response.json()
        return(html)

start = time.time()

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    processes = {executor.submit(function, query) for query in array}
    for result in concurrent.futures.as_completed(processes):
        print(result.result()['query_status'])

print(f'Time taken: {time.time() - start}')

结果：

hash_not_found
hash_not_found
hash_not_found
hash_not_found
ok
Time taken: 1.113807201385498

【讨论】：

非常感谢！现在我明白我的错误了

【解决方案2】：

您可以使用 greenlets 发出并行请求。

from gevent.pool import Group

print (starttime)
group = Group() #start parallel fn calls
group.spawn(DoRequest,param1,param2)
group.spawn(DoRequest,param1,param2)
group.spawn(DoRequest,param1,param2)
group.join() #wait for all to finish
print (endtime)

def DoRequest(param1,param2):
    #make request and get response

【讨论】：