发布请求不适用于scrapy，但适用于请求答案

【问题标题】：Post request not working with scrapy but works with requests发布请求不适用于scrapy，但适用于请求
【发布时间】：2021-12-02 16:08:24
【问题描述】：

请求代码：

listings_url = "https://www.biltorvet.dk/Api/Search/Page"
form_data = {
        "pageNumber": "1",
        "searchOrigin": "1",
        "searchValue": "22526899",
        "sort": ""
    }
response = requests.post(listings_url, json=form_data)

if response.status_code == 200:
    data = response.json()
    print(data)

抓取代码：

class BiltorvetScraperSpider(scrapy.Spider):
    name = 'biltorvet'
    listings_url = "https://www.biltorvet.dk/Api/Search/Page"
    form_data = {
        "pageNumber": "1",
        "searchOrigin": "1",
        "searchValue": "22526899",
        "sort": ""
    }

def start_requests(self):
    yield FormRequest(url=self.listings_url, callback=self.parse, body=json.dumps(self.form_data))

def parse(self, response):
    print(response.text)

scrapy 请求获得 400 分。我也尝试了标题，但结果相同。尝试将参数从body更改为json仍然没有影响。

【问题讨论】：

标签： python-3.x python-requests scrapy

【解决方案1】：

这应该可以达到目的：

import json
import scrapy
from scrapy.http.request import Request

class BiltorvetScraperSpider(scrapy.Spider):
    name = 'biltorvet'
    start_url = "https://www.biltorvet.dk/Api/Search/Page"
    
    form_data = {
        "pageNumber": "1",
        "searchOrigin": "1",
        "searchValue": "22526899",
        "sort": ""
    }
    
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36',
        'Content-Type': 'application/json; charset=UTF-8',
    }

    def start_requests(self):
        yield Request(
            self.start_url, 
            headers=self.headers,
            callback=self.parse, 
            method='POST',
            body=json.dumps(self.form_data)
        )

    def parse(self, response):
        print(response.json())

或者，您可以根据documentation如下尝试：

from scrapy.http import JsonRequest

def start_requests(self):
    yield JsonRequest(
        self.start_url, 
        headers=self.headers,
        callback=self.parse, 
        data=self.form_data
    )

def parse(self, response):
    print(response.json())

【讨论】：