【发布时间】：2021-09-16 22:14:39
【问题描述】：
我试图通过分析 Ajax 界面来抓取 QS 大学排名数据，因为 QS 全球大学排名页面是通过 Ajax 加载的。但是，我在编译时遇到了几个错误。在这种情况下;我得到一个 keyerror 类型：KeyError: 'url'。
url = 'https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3740566.txt?1624879808?v=1625562924528'
headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 "
"Safari/537.36"}
path = 'C:/Users/DELL/PycharmProjects/scrapeQS/qs-rank.txt'
def get_page(url):
try:
r = requests.get(url, headers=headers)
if r.status_code == 200:
return r.json()
except requests.ConnectionError as e:
print(e)
def parser_page(json):
if json:
items = json.get('data')
for i in range(len(items)):
item = items[i]
qsrank = {}
if "=" in item['rank_display']:
rk_str = str(item['rank_display']).split('=')[-1]
qsrank['rank_display'] = rk_str
else:
qsrank['rank_display'] = item['rank_display']
qsrank['title'] = item['title']
qsrank['region'] = item['region']
qsrank['score'] = item['score']
qsrank['url'] = item['url']
yield qsrank
def main():
json = get_page(url)
results = parser_page(json)
for result in results:
with open(path, 'a') as f:
f.write(result['rank_display'] + ' ' + result['title'] + ' ' + result['region'] + ' '
+ result['score'] + ' ' + 'https://www.topuniversities.com' + result['url'] + '\n')
f.close()
print(result)
if __name__ == '__main__':
print('Started parsing！')
with open(path, 'a') as f:
f.write('Ranking' + ' ' + 'University' + ' ' + 'Country' + ' ' + 'QS score' + ' ' + 'Link' + '\n')
f.close()
main()
print('Done')
【问题讨论】：
-
当你做
with open时不需要f.close() -
是的；我会更新它。谢谢
标签： javascript python ajax xml web-scraping