Python ASYNCIO ContentTypeError: 0, message='尝试使用意外的 mimetype 解码 JSON

Question

ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: application/octet-stream', url=URL('https://api-reports-prod-usamazon.s3.amazonaws.com/atvpdr-a2vzay-report-data-7aaf8bfa-5cb5-4c76-b430-01d68cd7024b.json.gz?X

当我尝试通过 Python - asyncio 方法获取位于 S3 Bucket 中的 gzip 文件时，我遇到了错误。

同步代码[工作中]

report = requests.get(location, headers=headers)
data = json.loads(gzip.decompress(report.content))

异步代码 [不工作]

async def get_data(session, url):
    async with session.get(url,headers=headers) as resp:
        data = await resp.json()
        return data 
  
async def main(req_url):
    async with aiohttp.ClientSession() as session:
        tasks = []
        url = req_url
        tasks.append(asyncio.ensure_future(get_data(session, url)))
        data = await asyncio.gather(*tasks)

start_time1 = time.time()
nest_asyncio.apply()
keyword_list = asyncio.run(main(location))
print("--- %s seconds ---" % (time.time() - start_time1))

提前致谢。

试过

async def get_data(session, url):
    async with session.get(url,headers=headers) as resp:
        data = json.loads(gzip.decompress(resp.content))
        return data 

哪个会报错

Traceback (most recent call last):

  File "<ipython-input-397-2f3527a7a82e>", line 20, in <module>
    keyword_list = asyncio.run(main(location))

  File "C:\Users\anaconda3\lib\site-packages\nest_asyncio.py", line 32, in run
    return loop.run_until_complete(future)

  File "C:\Users\anaconda3\lib\site-packages\nest_asyncio.py", line 70, in run_until_complete
    return f.result()

  File "C:\Users\anaconda3\lib\asyncio\futures.py", line 178, in result
    raise self._exception

  File "C:\Users\anaconda3\lib\asyncio\tasks.py", line 280, in __step
    result = coro.send(None)

  File "<ipython-input-397-2f3527a7a82e>", line 15, in main
    data = await asyncio.gather(*tasks)

  File "C:\Users\anaconda3\lib\asyncio\tasks.py", line 349, in __wakeup
    future.result()

  File "C:\Users\anaconda3\lib\asyncio\tasks.py", line 280, in __step
    result = coro.send(None)

  File "<ipython-input-397-2f3527a7a82e>", line 3, in get_data
    data = json.loads(gzip.decompress(resp.content))

  File "C:\Users\anaconda3\lib\gzip.py", line 547, in decompress
    with GzipFile(fileobj=io.BytesIO(data)) as f:

TypeError: a bytes-like object is required, not 'StreamReader'

'''

Answer 1

如果你使用同步代码

report = requests.get(location, headers=headers)
data = json.loads(gzip.decompress(report.content))

那么你应该在异步代码中做类似的事情

挖掘后发现需要await resp.read()而不是resp.content

async with session.get(url,headers=headers) as resp:
    data = json.loads(gzip.decompress(await resp.read()))
    return data 

---

您在main() 中忘记了return data

---

我无法访问包含 JSON 数据的 gzip 文件，因此我在 https://httpbin.org/get 的 JSON 上对其进行了测试

import asyncio
import aiohttp
import time

# --- functions ---

async def get_data(session, url):
    async with session.get(url, headers=headers) as resp:
        #return await resp.json()
        #return json.loads(gzip.decompress(await resp.read()))
        return await resp.read()

async def main(url):
    async with aiohttp.ClientSession() as session:
        tasks = asyncio.ensure_future(get_data(session, url))
        data = await asyncio.gather(tasks)
    return data
    
# --- main ---

headers = {}

location = 'https://httpbin.org/get'

start_time = time.time()

keyword_list = asyncio.run(main(location))
print(keyword_list)

end_time = time.time()

diff_time = end_time - start_time

print("---", diff_time, "seconds ---")