这是#site_comment_74240的HTML页面的sn-p
<div class="site_comment site_comment-even small normal-rank" id="site_comment_74240">
<div class="talkback-topic">
<a href="/comments/74240?counter=1&num=144" class="show-comment" data-ajax-url="/comments/74240.js?counter=1&num=144">144. מדיניות</a>
</div>
<div class="talkback-username">
<table><tr>
<td>קייזרמן פרדי </td>
<td>(01.11.2013)</td>
</tr></table>
</div>
“talkback-message”div 在您第一次获取它时不在 HTML 页面中，而是在您单击评论标题时通过一些 AJAX 查询异步获取，因此您必须获取它每条评论。
注释块，titles 在您的代码截图中，可以使用这样的 XPath 抓取：//div[starts-with(@id, "site_comment_"])，即所有具有以字符串“”site_comment_”开头的“id”属性的divs
您还可以将 CSS 选择器与 Selector.css() 一起使用。在您的情况下，您可以使用“id”方法（正如我在上面使用 XPath 所做的那样）来获取注释块，所以：
titles = sel.css("div[id^=site_comment_]")
或使用“site_comment”类而不使用其他不同的“site_comment-even”、“site_comment-odd”、“small”、“normal-rank”或“high-rank”：
titles = sel.css("div.site_comment")
然后，您将使用评论 div 中 ./div[@class="talkback-topic"]/a[@class="show-comment"]/@data-ajax-url 中的 URL 发出新的 Request。或者使用 CSS 选择器，div.talkback-topic > a.show-comment::attr(data-ajax-url)（顺便说一下，::attr(...) 不是标准的，而是使用伪元素函数对 CSS 选择器的 Scrapy 扩展）
你从 AJAX 调用中得到的是一些 Javascript 代码，你想抓取 old.after(...) 里面的内容
var old = $("#site_comment_72765");
old.attr('id', old.attr('id') + '_small');
old.hide();
old.after("\n<div class=\"site_comment site_comment-odd large high-rank\" id=\"site_comment_72765\">\n <div class=\"talkback-topic\">\n <a href=\"/comments/72765?counter=42&num=109\" class=\"show-comment\" data-ajax-url=\"/comments/72765.js?counter=42&num=109\">109. ביבי - האדם הנכון בראש ממשלת ישראל(לת)<\/a>\n <\/div>\n \n <div class=\"talkback-message\">\n \n <\/div>\n \n <div class=\"talkback-username\">\n <table><tr>\n <td>ישראל <\/td>\n <td>(11.03.2012)<\/td>\n <\/tr><\/table>\n <\/div>\n <div class=\"rank-controllers\">\n <table><tr>\n \n <td class=\"rabk-link\"><a href=\"#\" data-thumb=\"/comments/72765/thumb?type=up\"><img alt=\"\" src=\"/images/elements/thumbU.png?1376839523\" /><\/a><\/td>\n <td> | <\/td>\n <td class=\"rabk-link\"><a href=\"#\" data-thumb=\"/comments/72765/thumb?type=down\"><img alt=\"\" src=\"/images/elements/thumbD.png?1376839523\" /><\/a><\/td>\n \n <td> | <\/td>\n <td>11<\/td>\n \n <\/tr><\/table>\n <\/div>\n \n <div class=\"talkback-links\">\n <a href=\"/comments/new?add_to_root=true&html_id=site_comment_72765&sibling_id=72765\">תגובה חדשה<\/a>\n \n <a href=\"/comments/72765/comments/new?html_id=site_comment_72765\">הגיבו לתגובה<\/a>\n \n <a href=\"/i/offensive?comment_id=72765\" data-noajax=\"true\">דיווח תוכן פוגעני<\/a>\n <\/div>\n \n<\/div>");
var new_comment = $("#site_comment_72765");
这是您需要使用 Selector(text=this_ajax_html_data) 和 .//div[@class="talkback-message"]//text() XPath 或 div.talkback-message ::text CSS 选择器再次解析的 HTML 数据
这里有一个骷髅蜘蛛，可以帮助你实现这些想法：
from scrapy.spider import BaseSpider
from scrapy.selector import Selector
from scrapy.http import Request
from craigslist_sample.items import CraigslistSampleItem
import urlparse
import re
class MySpider(BaseSpider):
name = "craig"
allowed_domains = ["tbk.co.il"]
start_urls = ["http://www.tbk.co.il/tag/%D7%91%D7%A0%D7%99%D7%9E%D7%99%D7%9F_%D7%A0%D7%AA%D7%A0%D7%99%D7%94%D7%95/talkbacks"]
def parse(self, response):
sel = Selector(response)
comments = sel.css("div.site_comment")
for comment in comments:
item = CraigslistSampleItem()
# this probably has to be fixed
#item["title"] = comment.xpath("div[@class='talkback-message']text()").extract()
# issue an additional request to fetch the Javascript
# data containing the comment text
# and pass the incomplete item via meta dict
for url in comment.css('div.talkback-topic > a.show-comment::attr(data-ajax-url)').extract():
yield Request(url=urlparse.urljoin(response.url, url),
callback=self.parse_javascript_comment,
meta={"item": item})
break
# the line we are looking for begins with "old.after"
# and we want everythin inside the parentheses
_re_comment_html = re.compile(r'^old\.after\((?P<html>.+)\);$')
def parse_javascript_comment(self, response):
item = response.meta["item"]
# loop on Javascript content lines
for line in response.body.split("\n"):
matching = self._re_comment_html.search(line.strip())
if matching:
# what's inside the parentheses is a Javascript strings
# with escaped double-quotes
# a simple way to decode that into a Python string
# is to use eval()
# then there are these "<\/tag>" we want to remove
html = eval(matching.group("html")).replace(r"<\/", "</")
# once we have the HTML snippet, decode it using Selector()
decoded = Selector(text=html, type="html")
# and save the message text in the item
item["message"] = u''.join(decoded.css('div.talkback-message ::text').extract()).strip()
# and return it
return item
您可以使用scrapy runspider tbkspider.py 进行试用。