从多个页面抓取表格

Question

我想从this 站点抓取数据。该表扩展到多个页面，但是 url 没有类似 page=3 的内容，因此我无法遍历各个页面来获取数据。

有没有另一种方法可以像这样抓取数据而无需遍历各个页面？如何从 python 访问不同的页面以抓取数据？

Answer 1

使用@SeanBreckenridge 提供的直接 JSON 链接，您可以创建一个包含所有数据的 CSV 文件，如下所示：

import requests
import json
import csv

header = ['University', 'Department', 'Overall Pay', 'LW Ratio', 'Academic Year', 'Program Year', 'Comment', 'V1', 'V2', 'V3', 'V4']

with open('output.csv', 'w', newline='') as f_output:
    csv_output = csv.writer(f_output)
    csv_output.writerow(header)

    for index in range(30):
        url = 'http://www.phdstipends.com/data/{}'.format(index)
        print(url)
        r = requests.get(url)

        if len(r.text) < 20:
            break

        csv_output.writerows(json.loads(r.text)['data'])

什么开始：

University,Department,Overall Pay,LW Ratio,Academic Year,Program Year,Comment,V1,V2,V3,V4
University of California - Santa Barbara (UCSB),Chemical Engineering,"$31,125",1.27,2014-2015,1st,,"$31,250",,,$125
Pennsylvania State University (Penn State),Chemistry,"$20,500",1.11,2014-2015,2nd,,"$20,500",,,
University of California - San Diego (UCSD),Mathematics,"$20,500",0.87,2013-2014,4th,,,"$18,000","$2,500",