二:异步I/O模块的了解(同协程:线程在I/O请求上的优化)
(一)asyncio模块的使用
1.使用简单实例
import asyncio,time
async def func1(num):
print(num,"before func1.....")
await asyncio.sleep(num)
print(num,"after func1......")
tasks = [
asyncio.ensure_future(func1(1)),
asyncio.ensure_future(func1(3)),
]
begin = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(*tasks))
loop.close()
end = time.time()
print(end-begin)
1 before func1.....
3 before func1.....
1 after func1......
3 after func1......
3.003171682357788
(二).异步获取网站资源(不支持requests模块),使用asyncio模块的socket请求完成
不支持requests请求的原因:
异步的实现,需要自动切换协程,需要用到await或者yield,
而requests模块不提供切换协程的条件,所以当第一条协程执行到请求部分就停下来了,等服务器响应,
不会切到其他的协程去执行。
requests会阻塞asyncio循环,所以会出现收不到服务器响应的情况。可以使用aiohttp替代requests解决这个问题。
使用asyncio实现requests请求获取资源
import asyncio,time
async def func1(host,url='/'):
print(host,url)
reader, writer = await asyncio.open_connection(host,80)
request_header_content = '''GET %s HTTP/1.0\r\nHost: %s\r\n\r\n'''%(url,host)
request_header_content = bytes(request_header_content,encoding="utf-8")
writer.write(request_header_content)
await writer.drain()
text = await reader.read()
print(host,url,text)
writer.close()
tasks = [
asyncio.ensure_future(func1('www.cnblogs.com','/ssyfj/')),
asyncio.ensure_future(func1('www.baidu.com')),
asyncio.ensure_future(func1('www.ckook.com','/forum.php')),
]
begin = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(*tasks))
loop.close()
end = time.time()
print(end-begin)
![]()
www.cnblogs.com /ssyfj/
www.baidu.com /
www.ckook.com /forum.php
www.baidu.com / b'HTTP/1.0 200 OK\r\nAccept-Ranges: bytes\r\nCache-Control: no-cache\r\nContent-Length: 14615\r\nContent-Type: text/html\r\nDate: Sun, 24 Jun 2018 10:57:33 GMT\r\nLast-Modified: Mon, 11 Jun 2018 11:19:00 GMT\r\nP3p: CP=" OTI DSP COR IVA OUR IND COM "\r\nPragma: no-cache\r\nServer: BWS/1.1\r\nSet-Cookie: BAIDUID=3C070A482979333DD407E21A39ECA46A:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com\r\nSet-Cookie: BIDUPSID=3C070A482979333DD407E21A39ECA46A; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com\r\nSet-Cookie: PSTM=1529837853; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com\r\nVary: Accept-Encoding\r\nX-Ua-Compatible: IE=Edge,chrome=1\r\n\r\n<!DOCTYPE html><!--STATUS OK-->\r\n<html>\r\n<head>\r\n\t<meta http-equiv="content-type" content="text/html;charset=utf-8">\r\n\t<meta http-equiv="X-UA-Compatible" content="IE=Edge">\r\n\t<link rel="dns-prefetch" href="//s1.bdstatic.com"/>\r\n\t<link rel="dns-prefetch" href="//t1.baidu.com"/>\r\n\t<link rel="dns-prefetch" href="//t2.baidu.com"/>\r\n\t<link rel="dns-prefetch" href="//t3.baidu.com"/>\r\n\t<link rel="dns-prefetch" href="//t10.baidu.com"/>\r\n\t<link rel="dns-prefetch" href="//t11.baidu.com"/>\r\n\t<link rel="dns-prefetch" href="//t12.baidu.com"/>\r\n\t<link rel="dns-prefetch" href="//b1.bdstatic.com"/>\r\n\t<title>\xe7\x99\xbe\xe5\xba\xa6\xe4\xb8\x80\xe4\xb8\x8b\xef\xbc\x8c\xe4\xbd\xa0\xe5\xb0\xb1\xe7\x9f\xa5\xe9\x81\x93</title>\r\n\t<link href="http://s1.bdstatic.com/r/www/cache/static/home/css/index.css" rel="stylesheet" type="text/css" />\r\n\t<!--[if lte IE 8]><style index="index" >#content{height:480px\\9}#m{top:260px\\9}</style><![endif]-->\r\n\t<!--[if IE 8]><style index="index" >#u1 a.mnav,#u1 a.mnav:visited{font-family:simsun}</style><![endif]-->\r\n\t<script>var hashMatch = document.location.href.match(/#+(.*wd=[^&].+)/);if (hashMatch && hashMatch[0] && hashMatch[1]) {document.location.replace("http://"+location.host+"/s?"+hashMatch[1]);}var ns_c = function(){};</script>\r\n\t<script>function h(obj){obj.style.behavior=\'url(#default#homepage)\';var a = obj.setHomePage(\'//www.baidu.com/\');}</script>\r\n\t<noscript><meta http-equiv="refresh" content="0; url=/baidu.html?from=noscript"/></noscript>\r\n\t<script>window._ASYNC_START=new Date().getTime();</script>\r\n</head>\r\n<body link="#0000cc"><div >
www.cnblogs.com /ssyfj/ b'HTTP/1.1 200 OK\r\nDate: Sun, 24 Jun 2018 10:57:33 GMT\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 14922\r\nConnection: close\r\nVary: Accept-Encoding\r\nCache-Control: private, max-age=10\r\nExpires: Sun, 24 Jun 2018 10:57:43 GMT\r\nLast-Modified: Sun, 24 Jun 2018 10:57:33 GMT\r\nX-UA-Compatible: IE=10\r\n\r\n\r\n<!DOCTYPE html>\r\n<html lang="zh-cn">\r\n<head>\r\n<meta charset="utf-8"/>\r\n<meta name="viewport" content="width=device-width, initial-scale=1" />\r\n<title>\xe5\xb1\xb1\xe4\xb8\x8a\xe6\x9c\x89\xe9\xa3\x8e\xe6\x99\xaf - \xe5\x8d\x9a\xe5\xae\xa2\xe5\x9b\xad</title>\r\n<link type="text/css" rel="stylesheet" href="/bundles/blog-common.css?v=-hy83QNg62d4qYibixJzxMJkbf1P9fTBlqv7SK5zVL01"/>\n<link >
www.ckook.com /forum.php b'HTTP/1.1 301 Moved Permanently\r\nDate: Sun, 24 Jun 2018 10:57:33 GMT\r\nServer: Apache/2.4.23 (Win32) OpenSSL/1.0.2j PHP/5.4.45\r\nX-Powered-By: PHP/5.4.45\r\nLocation: https://www.ckook.com/forum.php\r\nContent-Length: 133\r\nContent-Type: text/html\r\n\r\n<br />\n<b>Notice</b>: Undefined index: HTTP_USER_AGENT in <b>C:\\www\\web\\www\\source\\class\\class_core.php</b> on line <b>19</b><br />\n'
5.589319705963135
输出