尽管在响应中正确填写了表单，但 FormRequest 不返回任何结果

Question

我正试图在他们的官方分行查找网站 (https://www.halifax.co.uk/branchfinder/search.asp) 上从哈利法克斯银行分行获取营业时间，但我必须在 FormRequest.from_response() 调用中做错了什么，因为对 FormRequest 的响应似乎与原始响应保持不变。

传入“邮政编码”值“EH”，应该有一个结果（爱丁堡，王子街 131 号，EH2 4AH），正如您在网站上执行相同操作时看到的那样，但我没有得到任何结果当我使用 FormRequest 进行操作时

这是网站来源的相关部分：

<form action="" id="branch-finder-search-form" method="post">
    <div style="display:none;" class="notice error" aria-live="assertive"></div>
    <div class="field split-2 clearfix">
        <div class="split radio">
            <div class="field-radio">
                <input value="branch" name="searchType" checked id="r1" type="radio"><label for="r1"><span></span>Branch</label>
            </div>
            <div class="field-radio">
                <input value="cash" name="searchType" id="r2" type="radio"><label for="r2"><span></span>Cash Machine</label>
            </div>
        </div>
    </div>
    <div class="field split-3 clearfix">
        <div class="split">
            <label for="street">Street</label><input name="street" id="street" type="text" value="">
        </div>
        <div class="split">
            <label for="town">Town</label><input name="town" id="town" type="text" value="">
        </div>
        <div class="split last">
            <label for="postcode">Post Code</label><input name="postcode" id="postcode" type="text" value="">
        </div>
    </div>
    <div class="field clearfix">
        <div class="split btn-submit">
            <input id="search" name="search" alt="Search" type="submit" value="Search" class="button button-primary" title="search"><span class="a_hide">ext search</span></input>
        </div>
        <noscript>
            <input value="yes" name="javascriptoff" id="javascriptoff" type="hidden">
            <div class="split btn-submit-nonjs">
                <input name="nonjsSubmit" type="submit" alt="Search" value="Search" class="button button-primary" title="search"><span class="a_hide">ext search</span></input>
            </div>
            <div></div>
        </noscript>
        <div style="display:none" id="no-result">
            <p>No Branch Found as per your search criteria</p>
        </div>
        <div id="branch-finder-results-container">
            <hr>
        </div>
    </div>
</form>

这是我目前的代码：

from scrapy.crawler import CrawlerProcess
from scrapy.utils.response import open_in_browser
from scrapy.http import FormRequest
import scrapy


class HalifaxSpider(scrapy.Spider):
    name = "halifax"
    start_urls = [
        "https://www.halifax.co.uk/branchfinder/search.asp"
    ]

    def parse(self, response):
        print(response.text)
        yield FormRequest.from_response(response, formname='branch-finder-search-form', formdata={"postcode": "EH"}, callback=self.open_in_browser)

    def open_in_browser(self, response):
        print(response.text)
        open_in_browser(response)


crawler = CrawlerProcess()
crawler.crawl(HalifaxSpider)
crawler.start()

我也尝试过传入参数 clickdata={"name": "search"}，以防没有点击搜索按钮，但我得到了相同的结果。

总的来说，我对网络抓取非常陌生，所以我真的被困住了，不知道从哪里开始。我已经阅读了一些文档，但是关于填写各种不同类型的表格的信息或教程并不多，所以我很茫然。

有人可以帮忙吗？

Answer 1

查看以下获取所需数据的方法。标题在这里起着重要作用，因此请确保包含它（我已经添加了）。

import scrapy
import urllib
from scrapy.crawler import CrawlerProcess

class HalifaxSpider(scrapy.Spider):
    name = "halifax"
    url = "https://www.halifax.co.uk/asp_includes/branch-finder/branch-finder.asp?"

    def start_requests(self):
        formdata = {
            'street': '',
            'town': '',
            'postcode': 'EH',
            'searchType': 'branch'
        }
        req_url = f'{self.url}{urllib.parse.urlencode(formdata)}'
        yield scrapy.Request(req_url,callback=self.parse)

    def parse(self,response):
        print(response.css("h2#resultTitle1 ::text").getall())

if __name__ == "__main__":
    crawler = CrawlerProcess({'USER_AGENT':'Mozilla/5.0'})
    crawler.crawl(HalifaxSpider)
    crawler.start()