抓取以 JSON 格式提供的无限结果（“查看更多”）

Question

我的目标是刮this URL。

列表中的每个项目都链接到有关它的更多信息。我的目标是抓取所有 17000 个链接页面。仅显示 10 个结果，查看更多按钮发出请求，通过 JSON 向列表中添加 10 个结果。我试图通过更改 batchsize 来修改请求，该参数用于定义列表中的结果数量，但这不起作用。我也尝试使用此代码（来自tutorial），但无法使其适应我的特定任务：

import json
import scrapy


class SpidyQuotesSpider(scrapy.Spider):
    name = 'spidyquotes'
    quotes_base_url = 'http://spidyquotes.herokuapp.com/api/quotes?page=%s'
    start_urls = [quotes_base_url % 1]
    download_delay = 1.5

    def parse(self, response):
        data = json.loads(response.body)
        for item in data.get('quotes', []):
            yield {
                'text': item.get('text'),
                'author': item.get('author', {}).get('name'),
                'tags': item.get('tags'),
            }
        if data['has_next']:
            next_page = data['page'] + 1
            yield scrapy.Request(self.quotes_base_url % next_page)

我查看了 here、here 和 here 的示例。但是，经过 2 天的尝试，我仍然无法弄清楚如何解决这个问题，因为我希望抓取的网站上的 URL 请求与所有示例都不同，而且似乎它们使抓取变得更加困难......

点击查看更多的请求如下：

请求网址： https://www.1177.se/api/hjv/search?batchsize=10&caretype=&componentname&cs=false&location=&p=2&q=&s=name&sortorder=name&st=4af2ed43-1154-4363-ae6b-718f9b84d23a

点击查看更多时，p=参数会逐渐增加：

返回的 JSON 格式如下：

{"Heading":"17952 träffar på Alla mottagningar","Query":"","Region":null,"NextPage":3,"Page":2,"BatchSize":10,"BatchText":"Visa 10 直到","TotalHits":17952,"SortOrder":"name","Latitude":0.0,"Longitude":0.0,"Bounds":null,"SearchHits":[{"HsaId":"SE162321000255-O23228" ,"FriendlyUrl":"/hitta-vard/kontaktkort/A5-Psykoterapi-Katia-Karlsson-Carli-AB-Lund/","DisplayName":"A5 Psykoterapi Katia Karlsson Carli AB, Lund","地址":"Stortorget 1, 隆德","电话号码":"073-046 26 68","HasMvkServices":true,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":55.703161529482479,"经度":13.193039057187006},{"HsaId":"SE162321000255-O22542" ,"FriendlyUrl":"/hitta-vard/kontaktkort/A5Psykoterapi-Gunilla-Lundqvist-Lund/","DisplayName":"A5Psykoterapi - Gunilla Lundqvist, Lund","Address":"Stortorget 1 5:e vån, Lund","PhoneNumber":"070-624 13 97","HasMvkServices":true,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":55.703161529482479,"经度":13.193039057187006},{"HsaId":"SE2321000057-6SV4" ,"FriendlyUrl":"/hitta-vard/kontaktkort/A6-Ogonklinik-AB/","DisplayName":"A6 Ögonklinik AB","地址":"Batterigatan 9 NB, 延雪平","电话号码":"036-860 20 30","HasMvkServices":true,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":57.768032303027383,"经度":14.202798620555548},{"HsaId":"SE162321000024-0059892" ,"FriendlyUrl":"/hitta-vard/kontaktkort/AB-Evelina-Linder-KBT/","DisplayName":"AB Evelina Linder KBT","地址":"Drottninggatan 1A, 乌普萨拉","电话号码":"073-593 00 73","HasMvkServices":false,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":59.858328320441558,"经度":17.638292776307694},{"HsaId":"SE162321000024-0052597" ,"FriendlyUrl":"/hitta-vard/kontaktkort/AB-Forsberg-KBT-konsult/","DisplayName":"AB Forsberg KBT-konsult","地址":"Trädgårdsgatan 5A, 乌普萨拉","电话号码":"070-818 17 11","HasMvkServices":false,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":59.856845411620185,"经度":17.635819529969204},{"HsaId":"SE2321000016-C7H4" ,"FriendlyUrl":"/hitta-vard/kontaktkort/AB-Lyhord-Ostermalmstorg/","DisplayName":"AB Lyhörd - Östermalmstorg","地址":"Östermalmstorg 1、斯德哥尔摩","电话号码":"08-425 004 00","HasMvkServices":false,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":59.336237708592563,"经度":18.079317099784653},{"HsaId":"SE2321000016-BH0B" ,"FriendlyUrl":"/hitta-vard/kontaktkort/AB-Suavis-horsel-Solna-Business-park/","DisplayName":"AB Suavis hörsel, Solna Business park","地址":"Svetsarvägen 15,2 tr,SOLNA","电话号码":"010-207 11 77","HasMvkServices":false,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":59.35928477168008,"经度":17.980058512140353},{"HsaId":"SE2321000016-56DM" ,"FriendlyUrl":"/hitta-vard/kontaktkort/AB-Vackra-Tander-Annette-Goransson/","DisplayName":"AB Vackra Tänder Annette Göransson","地址":"Drottninggatan 71A,斯德哥尔摩","电话号码":"08-21 52 62","HasMvkServices":false,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":59.33592153903674,"经度":18.059258535271329},{"HsaId":"SE5564844115-106Q" ,"FriendlyUrl":"/hitta-vard/kontaktkort/AB-Vackra-Tander-Norrmalm/","DisplayName":"AB Vackra Tänder, Norrmalm","地址":"Drottninggatan 71 A, 3 tr,","电话号码":"08-21 52 62","HasMvkServices":false,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":59.33592396728109,"经度":18.059118082991937},{"HsaId":"SE2321000016-97P2" ,"FriendlyUrl":"/hitta-vard/kontaktkort/ABA-Ogonklinik-i-Alvik/","DisplayName":"ABA Ögonklinik i Alvik","地址":"Tranebergsplan 3,,BROMMA","电话号码":"08-124 440 10","HasMvkServices":true,"VaccinatesForFlu":false,"VaccinatesForHpv":false,"距离":0.0,"纬度":59.33516807973394,"经度":17.978288641135208}],"HasZeroHits":false}

如果有一些初始代码行可以让我继续前进，我将不胜感激。

Answer 1

此代码可能有效，也可能无效，但鉴于您面临的问题，这是我将采取的方法。您可以将 {} 插入开始 url 以使用格式。此外，当您遍历 data['quotes'] 时，您现在处理的是 JSON 对象而不是 Scrapy 选择器。所以不需要调用.get()。

import json
import scrapy


class SpidyQuotesSpider(scrapy.Spider):
    name = 'spidyquotes'
    start_urls = ['https://www.1177.se/api/hjv/search?batchsize=10&caretype=&componentname&cs=false&location=&p={}&q=&s=name&sortorder=name&st=4af2ed43-1154-4363-ae6b-718f9b84d23a']

    def start_requests(self):
        # You may also need to replicate the headers used in the requests made to this URL.
        yield scrapy.Request(self.start_urls[0].format('1'))

    def parse(self, response):
        data = json.loads(response.body)
        for item in data['quotes']:
            # remember you're no longer dealing with a scrapy selector but now a json object
            yield {
                'text': item['text'],
                'author': item['name'],
                'tags': item['tags'],
            }
        if data['has_next']:
            # convert to integer to do addition
            next_page = int(data['page']) + 1
            yield scrapy.Request(self.start_urls[0].format(next_page), callback=self.parse)

Answer 2

这应该可以解决问题：

Headerz = {
    'accept': 'text/html, */*; q=0.01',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9',
    'cache-control': 'no-cache',
    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'pragma': 'no-cache',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
}

class SpidyQuotesSpider(scrapy.Spider):
    name = 'spidyquotes'
    start_urls = ['https://www.1177.se/api/hjv/search?batchsize=10&caretype=&componentname&cs=false&location=&p={}&q=&s=name&sortorder=name&st=4af2ed43-1154-4363-ae6b-718f9b84d23a']

    def start_requests(self):
        # You may also need to replicate the headers used in the requests made to this URL.
        yield scrapy.Request(self.start_urls[0].format('1'), headers=Headerz)

    def parse(self, response):
        data = json.loads(response.body)
        # you have json data in data variable, do what you intent to do so
        try:
            # paginate
            if not data['NextPage'] is None:
                nextpage_number = data['NextPage']
                nexturl = self.start_urls[0].format( str(nextpage_number) )
                yield scrapy.Request(nexturl, headers=Headerz)
        except:
            pass

这里的诀窍是使用正确的标题！