如何使用 Scrapy 过滤搜索结果答案

【问题标题】：How can I filter search results using Scrapy如何使用 Scrapy 过滤搜索结果
【发布时间】：2021-03-21 01:56:47
【问题描述】：

我是抓取新手，我正在尝试从该网站https://seffaflik.epias.com.tr/transparency/uretim/gerceklesen-uretim/gercek-zamanli-uretim.xhtml抓取数据

当我尝试在不应用过滤器的情况下获取数据时，一切正常。但我需要的数据应该是针对特定发电厂和日期的。我很难找到无法应用过滤器的原因。

from scrapy.http import FormRequest
from ..items import EpiasscrapingItem

class EpiasSpider(scrapy.Spider):
    name = 'epias'
    start_urls =[
        'https://seffaflik.epias.com.tr/transparency/uretim/gerceklesen-uretim/gercek-zamanli-uretim.xhtml'
    ]

    def parse(self, response):
      
        return FormRequest.from_response(response, formdata = {
            'j_idt205':'j_idt205',
            'j_idt205:date1_input' : '20.03.2021',
            'j_idt205:date2_input' : '20.03.2021',
            'j_idt205:powerPlant_input' : '2614',
 
        }, callback=self.start_scraping)

    def start_scraping(self,response):
        items = EpiasscrapingItem()

        table_epias = response.css('.ui-datatable-odd')

        for epias in table_epias:
            date = epias.css('.ui-widget-content .TexAlCenter:nth-child(1)').css('::text').extract()
            time = epias.css('.ui-widget-content .TexAlCenter:nth-child(2)').css('::text').extract()
            biogas = epias.css('.ui-widget-content .TexAlCenter:nth-child(15)').css('::text').extract()

            items['date'] = date 
            items['time'] = time 
            items['biogas'] = biogas 

            yield items```

【问题讨论】：

标签： python-3.x web-scraping scrapy

【解决方案1】：

您忘记在应该与发布请求一起发送的参数中包含javax.faces.ViewState 和其他一些字段。您现在可以更改date1_input、date2_input 和powerPlant_input 的值以获取相关内容。以下脚本应该可以工作：

class EpiasSpider(scrapy.Spider):
    name = 'epias'
    
    start_urls = [
        'https://seffaflik.epias.com.tr/transparency/uretim/gerceklesen-uretim/gercek-zamanli-uretim.xhtml'
    ]

    post_url = 'https://seffaflik.epias.com.tr/transparency/uretim/gerceklesen-uretim/gercek-zamanli-uretim.xhtml'

    def parse(self, response):
        payload = {
            'j_idt205': 'j_idt205',
            'j_idt205:date1_input': '11.02.2021',
            'j_idt205:date2_input': '20.03.2021',
            'j_idt205:powerPlant_focus': '',
            'j_idt205:powerPlant_input': '2336',
            'j_idt205:goster': '',
            'j_idt205:dt_rppDD': '24',
            'javax.faces.ViewState': response.css(".ContainerIndent input[name='javax.faces.ViewState']::attr(value)").get()
        }

        yield scrapy.FormRequest(self.post_url,formdata=payload,callback=self.parse_content)

    def parse_content(self,response):
        for epias in response.css('.ui-datatable-odd'):
            items = {}
            date = epias.css('tr.ui-widget-content > .TexAlCenter:nth-child(1)::text').get()
            time = epias.css('tr.ui-widget-content > .TexAlCenter:nth-child(2)::text').get()
            total = epias.css('tr.ui-widget-content > .TexAlCenter:nth-child(3)::text').get()

            items['date'] = date 
            items['time'] = time 
            items['total'] = total 
            yield items

【讨论】：