Scrapy 在谷歌文档中找不到 div 标签答案

【问题标题】：Scrapy can't find div tag in google docsScrapy 在谷歌文档中找不到 div 标签
【发布时间】：2020-07-14 23:47:54
【问题描述】：

我正在尝试使用scrapy 从链接https://docs.google.com/document/d/1owRcDbh6vKuohojMhFD_2d4ivGM9J-nDziVRJR9Djck/edit?usp=sharing 获取文本“数据”，这是一个谷歌文档。我的代码是

import scrapy


class MitdataSpider(scrapy.Spider):
    name = 'mitdata'
    allowed_domains = ['www.docs.google.com']
    start_urls = ['https://docs.google.com/document/d/1owRcDbh6vKuohojMhFD_2d4ivGM9J-nDziVRJR9Djck/edit?usp=sharing']

    def parse(self, response):
        text = response.selector.xpath('//*[@id="kix-appview"]/div[7]/div/div[1]/div[1]/div/div/div/div[2]/div/div[2]/div[1]/div/div/div[1]/div/div/div[2]/div/table/tbody/tr[2]/td[2]/div/div/div/div/span[2]/span/span/text()[1]').extract()
        goodhtmlcode = response.selector.xpath('//*[@id="kix-appview"]').extract()
        badhhtmlcode = response.selector.xpath('//*[@id="kix-appview"]/div[7]').extract()
        print(text)
        print(goodhtmlcode)
        print(badhhtmlcode)

输出是

[]
['<div id="kix-appview" class="kix-appview"><div id="docs-findbar-id" class="docs-ui-unprintable"></div><div id="docs-spellcheckslidingdialog-id" class="docs-ui-unprintable"></div><div id="docs-checkupdialog-id" class="docs-ui-unprintable"></div><div id="docs-suggestiondialog-id" class="docs-ui-unprintable"></div><div class="kix-appview-editor-container"><div class="kix-appview-editor"></div></div><div id="kix-vertical-ruler" class="docs-ui-unprintable docs-vertical-ruler docs-ruler"><div class="docs-ruler-inner"></div></div></div>']
[]

我不明白为什么只是停止寻找元素。它能够找到直到//*[@id="docs-suggestiondialog-id"] 的所有内容，然后跳过//*[@id="docs-suggestiondialog-id"] 和//*[@id="kix-vertical-ruler"] 之间的任何内容。我错过了什么吗明显吗？

【问题讨论】：

您是否要抓取 Google 文档？我建议您将所有内容另存为 csv
是的，谢谢，我会尝试这可能会更容易，因为我正在查看的真正的 google 文档有许多带有文本的表格和单元格，但给了我完全相同的问题。

标签： python html web-scraping scrapy

【解决方案1】：

这是代码。

import scrapy
class MitdataSpider(scrapy.Spider):
    name = 'mitdata'
    allowed_domains = ['www.docs.google.com']
    start_urls = ['https://docs.google.com/document/d/1owRcDbh6vKuohojMhFD_2d4ivGM9J-nDziVRJR9Djck/edit?usp=sharing']

    def parse(self, response):
        text = response.selector.xpath('//*[@id="kix-appview"]/div[7]/div/div[1]/div[1]/div/div/div/div[2]/div/div[2]/div[1]/div/div/div[1]/div/div/div[2]/div/table/tbody/tr[2]/td[2]/div/div/div/div/span[2]/span/span/text()[1]').extract()
        goodhtmlcode = response.selector.xpath('//*[@id="kix-appview"]').extract()
        badhhtmlcode = response.selector.xpath('//*[@id="kix-appview"]/div[7]').extract()
        print(text)
        print(goodhtmlcode)
        print(badhhtmlcode)
        print()
        print(response.css('span::text').get())
        print()

这是数据。

[]
['<div id="kix-appview" class="kix-appview"><div id="docs-findbar-id" class="docs-ui-unprintable"></div><div id="docs-spellcheckslidingdialog-id" class="docs-ui-unprintable"></div><div id="docs-checkupdialog-id" class="docs-ui-unprintable"></div><div id="docs-suggestiondialog-id" class="docs-ui-unprintable"></div><div class="kix-appview-editor-container"><div class="kix-appview-editor"></div></div><div id="kix-vertical-ruler" class="docs-ui-unprintable docs-vertical-ruler docs-ruler"><div class="docs-ruler-inner"></div></div></div>']
[]

data

【讨论】：

您的代码找到了文档的标题，而不是表格单元格。选择标题与单元格中的相同是我的错。