wljdeMacBook-Pro:~ wlj$ scrapy shell http://doc.scrapy.org/en/latest/_static/selectors-sample1.html
>>> response.url
\'https://doc.scrapy.org/en/latest/_static/selectors-sample1.html\'
>>> response.text
页面远吗如下：
<html>
<head></head>
<body>
<base href="http://example.com/" />
<title>Example website</title>
<div id="images">
<a href="image1.html">Name: My image 1 <br /><img src="image1_thumb.jpg" /></a>
<a href="image2.html">Name: My image 2 <br /><img src="image2_thumb.jpg" /></a>
<a href="image3.html">Name: My image 3 <br /><img src="image3_thumb.jpg" /></a>
<a href="image4.html">Name: My image 4 <br /><img src="image4_thumb.jpg" /></a>
<a href="image5.html">Name: My image 5 <br /><img src="image5_thumb.jpg" /></a>
</div>
</body>
</html>
XPath选择器
>>> result = response.selector.xpath(\'.//a\')
>>> result.xpath(\'./img\')
输出结果：
[<Selector xpath=\'./img\' data=\'<img src="image1_thumb.jpg">\'>,
<Selector xpath=\'./img\' data=\'<img src="image2_thumb.jpg">\'>,
<Selector xpath=\'./img\' data=\'<img src="image3_thumb.jpg">\'>,
<Selector xpath=\'./img\' data=\'<img src="image4_thumb.jpg">\'>,
<Selector xpath=\'./img\' data=\'<img src="image5_thumb.jpg">\'>]
>>> result[0]
<Selector xpath=\'.//a\' data=\'<a href="image1.html">Name: My image 1 <\'>
>>> response.xpath(\'//a[@href="image1.html"]/text()\').extract()
[\'Name: My image 1 \']
很多情况下，我们想要匹配第一个元素内容，通过加一个索引来获取。
>>> response.xpath(\'//a[@href="image1.html"]/text()\').extract()[0]
\'Name: My image 1 \'
>>>
这个写法明显是有风险的，一旦XPath有问题，那么extract()后的结果可能是一个空列表。如果再用索引来取，就导致数组越界。
另外一个方法可以专门提取单个元素，它叫作extract_first()。
>>> response.xpath(\'//a[@href="image1.html"]/text()\').extract_first()
\'Name: My image 1 \'
也可以为extract_first()方法设置一个默认值参数
>>> response.xpath(\'//a[@href="image1.html"]/text()\').extract_first(\'Defalt Image\')
\'Name: My image 1 \'
>>> response.xpath(\'//a[@href="image1"]/text()\').extract_first(\'Defalt Image\')
\'Defalt Image\'
>>> response.xpath(\'//a[@href="image1"]/text()\').extract_first()
>>>
如果XPath匹配不到任何元素，extract_first(\'Defalt Image\')传递了一个参数Defalt Image当默认值，返回值会使用这个参数，输出正是如此。
如果XPath匹配不到任何元素，xtract_first()会返回空，不会报错。extract()[0]则会报错。
CSS选择器
>>> response.css(\'a[href="image1.html"] img\').extract_first()
\'<img src="image1_thumb.jpg">\'
获取节点内部文本和属性
>>> response.css(\'a[href="image1.html"]::text\').extract_first()
\'Name: My image 1 \'
>>> response.css(\'a[href="image1.html"] img::attr(src)\').extract_first()
\'image1_thumb.jpg\'
CSS选择器和XPath选择器一样可以嵌套使用
>>> response.xpath(\'//a\').css(\'img\').xpath(\'@src\').extract()
[\'image1_thumb.jpg\', \'image2_thumb.jpg\', \'image3_thumb.jpg\', \'image4_thumb.jpg\', \'image5_thumb.jpg\']
>>> response.xpath(\'//a\').css(\'img\').xpath(\'@src\').extract_first()
\'image1_thumb.jpg\'
正交匹配
>>> response.xpath(\'//a/text()\').re(\'Name:\s(.*)\')
[\'My image 1 \', \'My image 2 \', \'My image 3 \', \'My image 4 \', \'My image 5 \']
>>> response.xpath(\'//a/text()\').re(\'(.*?):\s(.*)\')
[\'Name\', \'My image 1 \', \'Name\', \'My image 2 \', \'Name\', \'My image 3 \', \'Name\', \'My image 4 \', \'Name\', \'My image 5 \']
>>> response.xpath(\'//a/text()\').re(\'(.*?):\s(.*)\')
[\'Name\', \'My image 1 \', \'Name\', \'My image 2 \', \'Name\', \'My image 3 \', \'Name\', \'My image 4 \', \'Name\', \'My image 5 \']
>>> response.xpath(\'//a/text()\').re_first(\'(.*?):\s(.*)\')
\'Name\'
>>> response.xpath(\'//a/text()\').re_first(\'Name:\s(.*)\')
\'My image 1 \'
值得注意的是，response对象不能直接调用re()和re_first()方法。如果想对全文进行正则匹配，可以先调用xpath()方法再正则匹配。
>>> response.xpath(\'.\').re_first(\'Name:\s(.*)<br>\')
\'My image 1 \'