1.工欲善其事必先利其器,fiddler安装
https://www.telerik.com/fiddler
2.安装exe(无脑下一步)
3.安装成功后配置fiddler(因为启动fiddler时链接不能登录)
(1)开启fiddler----->Tools----->Options----->Actions----->Export root,此时桌面就会出现fiddler证书。
(2)(Firefox)进入网页,点击选项,搜索证书,查看证书,导入证书(FiddlerRoot.cer)全部勾上。
(Google Chrome)点击设置,进入高级,管理证书,同上导入证书。
(3)App,首先要设置同一网络下,设置,手动代理IP,输入本机IP,端口:8888,同时找到系统安全,安装上面说到的证书。
至此,大家就可以用fiddler抓包了,上一段简单的代码(抓取wawayaya阅读第一页图片)
重中之重:如果要运行下列代码,记得信任证书,fiddler----->Tools----->Options----->Actions------>Trust root(不然图片会加载失败)
#-*- coding: UTF-8 -*-
from urllib.request import urlretrieve
import requests
import os
def book_imgs_download(books_url,header):
req = requests.get(url = books_url, headers = header).json()
a = req["retinfo"]
book_num = len(a[\'libList\'])
print(book_num)
print(\'一共有%d个种类\' % book_num)
book_images_path = \'books_images\'
for each_book in a[\'libList\']:
book_photo_url = each_book[\'imageEn\']
book_name = each_book[\'cname\'] + \'.jpg\'
filename = book_images_path + \'/\' + book_name
if book_images_path not in os.listdir():
os.makedirs(book_images_path)
urlretrieve(url = book_photo_url, filename = filename)
if __name__ == \'__main__\':
headers = {\'Accept-Charset\': \'UTF-8\',
\'Accept-Encoding\': \'gzip\',
\'User-Agent\': \'Dalvik/2.1.0 (Linux; U; Android 6.0.1; Redmi 4 MIUI/V8.5.3.0.MBECNED)\',
\'X-Requested-With\': \'XMLHttpRequest\',
\'Content-type\': \'application/x-www-form-urlencoded\',
\'Connection\': \'Keep-Alive\',
\'Host\': \'duba.wawayaya.com\'}
books_url = "http://api3-joyreader.wawayaya.com/api/server/book/getLibHomeList?deviceType=phone&client_lang=zh&appVer=3.8.6&userId=610378&uuid=android_fe087c86fc37533980072e3aa76c244e&platform=android&osVer=23&mac=02:00:00:00:00:00&token=9bdebc55f4f3465def7f3dabd0a0727c&sig=009ff5f1&countryCode=CN&appId=2209410&app_ver=3.8.6&imei=863968035553662×tamp=1513081106655"
# req = requests.get(url = heros_url, headers = headers).json()
# a = req["retinfo"]
# b = a["libList"]
# c = b[1]
# print(c)
book_imgs_download(books_url, headers)
注意:链接失效较快,大约几分钟就会失效,这时就要重新获取链接。而且要注意把headers的东西改成自己的(
\'User-Agent,
\'Host
)