scrapy 填写 POST 表格

Question

我正在尝试使用 scrapy 填写 POST 表单，尝试预订火车票。

我认为 FormRequest 类可以做到这一点，但我无法处理 javascript 表单。 Scrapy 爬虫什么也不返回。

我使用的文件足以发送表单。

import scrapy

from scrapy.item import Item, Field
from scrapy.http import FormRequest
from scrapy.spider import BaseSpider

class SncfItem(Item):
 title = Field()
 link = Field()
 desc = Field()

class SncfSpider(scrapy.Spider):
name = "sncf"
allowed_domains = ["voyages-sncf.com"]
start_urls = (
    'http://www.voyages-sncf.com/billet-train',
)

def parse(self, response):

    yield FormRequest.from_response(response,
                                    formname='saisie',
                                    formdata={'ORIGIN_CITY': 'Gare de Lyon (Paris)',
                                              'DESTINATION_CITY': 'Lyon Part-Dieu',
                                              'OUTWARD_DATE': '03.06.2015'},
                                    callback=self.parse1)

def parse1(self, response):
    print response.status

如果我在使用 mySpider 时遗漏了一个步骤，谁能告诉我？

任何帮助将不胜感激。
谢谢

scrapy crawl sncf -o items.xml  -t xml
2015-06-01 23:13:54+0200 [sncf] INFO: Spider opened
2015-06-01 23:13:54+0200 [sncf] INFO: Crawled 0 pages (at 0 pages/min),       scraped 0 items (at 0 items/min)
2015-06-01 23:13:54+0200 [scrapy] DEBUG: Telnet console listening on   127.0.0.1:6024
2015-06-01 23:13:54+0200 [scrapy] DEBUG: Web service listening on 127.0.0.1:6081
2015-06-01 23:13:55+0200 [sncf] DEBUG: Crawled (200) <GET     http://www.voyages-sncf.com/billet-train> (referer: None)
2015-06-01 23:13:56+0200 [sncf] DEBUG: Redirecting (302) to <GET   http://www.voyages-sncf.com/billet-train> from <POST http://www.voyages-   sncf.com/vsc/train-ticket/>
2015-06-01 23:13:56+0200 [sncf] DEBUG: Crawled (200) <GET   http://www.voyages-sncf.com/billet-train> (referer: http://www.voyages-  sncf.com/billet-train)
200
2015-06-01 23:13:56+0200 [sncf] INFO: Closing spider (finished)

Answer 1

您收到重定向，因为日期格式无效。

我在 Scrapy shell 中重放了该请求，执行以下操作：

$ scrapy shell http://www.voyages-sncf.com/billet-train
.... a few log messages later, I get a shell with the response...
>>> # first I recreate the FormRequest per your code:
>>> fr = FormRequest.from_response(response,
                                  formname='saisie',
                                  formdata={'ORIGIN_CITY': 'Gare de Lyon (Paris)',
                                           'DESTINATION_CITY': 'Lyon Part-Dieu',
                                           'OUTWARD_DATE': '03.06.2015'})
>>> # checked the url and method:
>>> fr.url
'http://www.voyages-sncf.com/vsc/train-ticket/'
>>> fr.method
'POST'
>>> fetch(fr)  # execute the request
>>> view(response)  # opened the result in browser

查看结果，我看到日期验证错误消息，上面写着：“Le format de la date d'aller que vous avez saisi est wrong. Nous vous invitons à utiliser le calendrier。”