这里有什么错误？

Question

这些是我的代码，但它似乎是正确的，但它不起作用，请帮助

HEADER_XPATH = ['//h1[@class="story-body__h1"]//text()']    
AUTHOR_XPATH = ['//span[@class="byline__name"]//text()']   
PUBDATE_XPATH = ['//div/@data-datetime']  
WTAGS_XPATH = ['']   
CATEGORY_XPATH = ['//span[@rev="news|source""]//text()']    
TEXT = ['//div[@property="articleBody"]//p//text()']   
INTERLINKS = ['//div[@class="story-body__link"]//p//a/@href']  
DATE_FORMAT_STRING = '%Y-%m-%d'

class BBCSpider(Spider):
    name = "bbc"
    allowed_domains = ["bbc.com"]
    sitemap_urls = [
        'http://Www.bbc.com/news/sitemap/',
        'http://www.bbc.com/news/technology/',
        'http://www.bbc.com/news/science_and_environment/']

    def parse_page(self, response):
        items = []
        item = ContentItems()
        item['title'] = process_singular_item(self, response, HEADER_XPATH, single=True)
        item['resource'] = urlparse(response.url).hostname
        item['author'] = process_array_item(self, response, AUTHOR_XPATH, single=False)
        item['pubdate'] = process_date_item(self, response, PUBDATE_XPATH, DATE_FORMAT_STRING, single=True)
        item['tags'] = process_array_item(self, response, TAGS_XPATH, single=False)
        item['category'] = process_array_item(self, response, CATEGORY_XPATH, single=False)
        item['article_text'] = process_article_text(self, response, TEXT)
        item['external_links'] = process_external_links(self, response, INTERLINKS, single=False)
        item['link'] = response.url
        items.append(item)
        return items

Answer 1

你的蜘蛛只是结构很糟糕，因此它什么也没做。
scrapy.Spider 蜘蛛需要 start_urls 类属性，该属性应包含蜘蛛将用于开始抓取的 url 列表，所有这些 url 将回调到类方法 parse 这意味着它也是必需的。

你的蜘蛛有 sitemap_urls 类属性，它没有在任何地方使用，而且你的蜘蛛有 parse_page 类方法，也从未在任何地方使用过。
所以简而言之，你的蜘蛛应该看起来像这样：

class BBCSpider(Spider):
    name = "bbc"
    allowed_domains = ["bbc.com"]
    start_urls = [
        'http://Www.bbc.com/news/sitemap/',
        'http://www.bbc.com/news/technology/',
        'http://www.bbc.com/news/science_and_environment/']

    def parse(self, response):
        # This is a page with all of the articles
        article_urls = # find article urls in the pages
        for url in article_urls:
            yield Request(url, self.parse_page)

    def parse_page(self, response):
        # This is an article page
        items = []
        item = ContentItems()
        # populate item
        return item