xpath 不使用 scrapy shell 提取内容答案

【问题标题】：xpath doesnot extract contents using scrapy shellxpath 不使用 scrapy shell 提取内容
【发布时间】：2020-08-06 09:31:49
【问题描述】：

我可以在网站上使用 XPath helper google chrome 扩展来获取网页内容，但如果我使用 sxrapy shell 应用，则路径相同。

使用 XPath chrome 扩展：

网页 = anywebsite.com

XPath = //section[@class='rslwrp']/div/ul/li//h2[@class='store-name']/span/a/span/text()

但是当我使用 scrapy shell 申请时，它返回 null 为什么？

scrapy shell "www.mywebsite.com"
In [15]: response.xpath("//section[@class='rslwrp']/div/ul/li//h2[@class='store-
    ...: name']/span/a/span")                                                   
Out[15]: []

【问题讨论】：

标签： python xpath scrapy

【解决方案1】：

似乎设置了用户代理，它可以工作：

scrapy shell "www.mywebsite" -s USER_AGENT="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/53.0"

结果：

In [1]: response
Out[1]: <200 https://www.mywebsite.com>

'rslwrp' 之后还需要一个空格，所以'rslwrp '

In [9]: response.xpath("//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span")
Out[9]:
[<Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Brand Cit...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Kuntal Ou...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Brands Cl...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">4g Fashio...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Paridhan ...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Harish Ga...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Maa Vaish...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Shoppers ...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Bachoomal...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Vishal Me...'>]

【讨论】：