Scrapy，通过onclick抓取数据

Question

我想在这个链接中提取每篇论文的标题和pdf链接：https://iclr.cc/Conferences/2019/Schedule?type=Poster

我的代码在这里

class ICLRCrawler(Spider):
    name = "ICLRCrawler"
    allowed_domains = ["iclr.cc"]
    start_urls = ["https://iclr.cc/Conferences/2019/Schedule?type=Poster", ]

    def parse(self, response):
        papers = Selector(response).xpath('//*[@id="content"]/div/div[@class="paper"]')
        titles = Selector(response).xpath('//*[@id="maincard_704"]/div[3]')
        links = Selector(response).xpath('//*[@id="maincard_704"]/div[6]/a[2]')
        for title, link in zip(titles, links):
            item = PapercrawlerItem()
            item['title'] = title.xpath('text()').extract()[0]
            item['pdf'] = link.xpath('/@href').extract()[0]
            item['sup'] = ''
            yield item 

但是，要获得每篇论文的标题和链接似乎并不容易。在这里，如何更改代码以获取数据？

Answer 1

您可以使用更简单的方法：

def parse(self, response):

    for poster in response.xpath('//div[starts-with(@id, "maincard_")]'):
        item = PapercrawlerItem()
        item["title"] = poster.xpath('.//div[@class="maincardBody"]/text()[1]').get()
        item["pdf"] = poster.xpath('.//a[@title="PDF"]/@href').get()

        yield item

Answer 2

您必须将Extract()[0] 替换为get_attribute('href')