有道翻译爬虫小案例

有道翻译主页http://fanyi.youdao.com/

打开开发者工具，选择newwork中的xhr，然后输入要翻译的文字，点击请求记录会发现post请求地址如图所示

查看请求参数并多输入几组翻译对象，会发现其中参数的一些信息，i是要翻译的对象，from-to为翻译语言选择，其余的每次都会变的参数为salt，sign，以及ts。

通过搜索关键字sign会发现其在fanyi.mim.js中生成，打开js文件继续搜索sign并逐条查看找到以下有用的信息

 

可以看到ts是发起请求时的时间戳，salt实在ts的基础上再加一位随机数，而sign是由md5对关键字、salt和常量组成的字符串的加密散列值，对其重构代码如下：

salt = str(int(time.time() * 10000))
ts = salt[: -1]
sign = md5(("fanyideskweb" + word + salt + "Nw(nmmbP%A-r6U3EUn]Aj").encode()).hexdigest()

这里我直接给salt赋值了14位时间戳并取其前十三位为ts

之后就可以快乐的用requests.post请求并取其结果了，记得要加请求头部，不然还是没结果。完整代码如下：

import requests
import time
from hashlib import md5
import sys


class YouDaoTranslate:
    def __init__(self, sentence):
        self.url = \'http://fanyi.youdao.com/translate_o?smartresult=dict&smartresult=rule\'
        self.sentence = sentence
        self.headers = {
            \'Accept\': \'application/json, text/javascript, */*; q=0.01\',
            \'Accept-Encoding\': \'gzip, deflate\',
            \'Accept-Language\': \'zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7\',
            \'Connection\': \'keep-alive\',
            "Content-Length": "251",
            \'Content-Type\': \'application/x-www-form-urlencoded; charset=UTF-8\',
            \'Cookie\': \'OUTFOX_SEARCH_USER_ID=-181135089@10.169.0.102; JSESSIONID=aaanFjxGXbOYaaclPZDjx; \'
                      \'OUTFOX_SEARCH_USER_ID_NCOO=1741802456.8918452; ___rl__test__cookies=1590721858150\',
            \'Host\': \'fanyi.youdao.com\',
            \'Origin\': \'http://fanyi.youdao.com\',
            \'Pragma\': \'no-cache\',
            \'Referer\': \'http://fanyi.youdao.com/\',
            \'User-Agent\': \'Mozilla/5.0 (Windows NT 6.1; Win64; x64) \'
                          \'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36\',
            \'X-Requested-With\': \'XMLHttpRequest\'
        }

    def translate(self):
        salt = self.get_salt()
        ts = salt[: -1]
        sign = self.get_sign(self.sentence, salt)
        data = {
            \'i\': self.sentence,
            \'from\': \'AUTO\',
            \'to\': \'AUTO\',
            \'smartresult\': \'dict\',
            \'client\': \'fanyideskweb\',
            \'salt\': salt,
            \'sign\': sign,
            \'ts\': ts,
            \'bv\': \'710f3e24cb0088b9d9ea448919deb3bb\',
            \'doctype\': \'json\',
            \'version\': 2.1,
            \'keyfrom\': \'fanyi.web\',
            \'action\': \'FY_BY_REALTlME\',
        }
        try:
            req = requests.post(url=self.url, data=data, headers=self.headers)
            if req.status_code == 200:
                resp = req.json()
                return resp[\'translateResult\'][0][0][\'tgt\']
        except:
            return

    @staticmethod
    def get_sign(word, salt):
        return md5(("fanyideskweb" + word + salt + "Nw(nmmbP%A-r6U3EUn]Aj").encode()).hexdigest()

    @staticmethod
    def get_salt():
        return str(int(time.time() * 10000))


if __name__ == \'__main__\':
    while True:
        sentence = input(\'请输入你要翻译的内容(输入q退出):\')
        if sentence == \'q\':
            sys.exit()
        else:
            y = YouDaoTranslate(sentence)
            result = y.translate()
            print(result)