页面上的 Python 抓取仍然包含像 \r \n \t 这样的字符答案

【问题标题】：Python scraping on page still contains chars like \r \n \t页面上的 Python 抓取仍然包含像 \r \n \t 这样的字符
【发布时间】：2014-02-01 05:04:23
【问题描述】：

我正在尝试使用 scrapy 0.20.2 在 http://www.dmoz.org/Computers/Programming/Languages/Python/Books 这个页面上进行抓取。

我可以做所有我需要的事情，比如获取信息和排序......

但是，我仍然在结果中得到 \r 和 \t 和 \n。例如这是一个 json {"desc": ["\r\n\t\t\t\r\n ", " \r\n\t\t\t\r\n - The primary goal of this book is to promote object-oriented design using Python and to illustrate the use of the emerging object-oriented design patterns.\r\nA secondary goal of the book is to present mathematical tools just in time. Analysis techniques and proofs are presented as needed and in the proper context.\r\n \r\n "], "link": ["http://www.brpreiss.com/books/opus7/html/book.html"], "title": ["Data Structures and Algorithms with Object-Oriented Design Patterns in Python"]},

数据是正确的，但我不想在结果中看到 \t 和 \r 和 \n。

我的蜘蛛是

from scrapy.spider import BaseSpider
from scrapy.selector import Selector

from dirbot.items import DmozItem

class DmozSpider(BaseSpider):
   name = "dmoz"
   allowed_domains = ["dmoz.org"]
   start_urls = [
       "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"
   ]

   def parse(self, response):
       sel = Selector(response)
       sites = sel.xpath('//ul[@class="directory-url"]/li')
       items = []
       for site in sites:
           item = DmozItem()
           item['title'] = site.xpath('a/text()').extract()
           item['link'] = site.xpath('a/@href').extract()
           item['desc'] = site.xpath('text()').extract()
           items.append(item)
       return items

【问题讨论】：

\r 和 \n 是行尾 (EOL) 字符，\t 是制表符。删除它们的最常见方法是使用 rstrip()
@emh 请提供示例，我应该在我的项目类中使用它吗？
@emh 当我尝试制作 site.xpath('a/text()').extract().rstrip() 时，我得到了一个空结果
你可以使用类似item['desc'] = map(unicode.strip, site.xpath('a/text()').extract())
是的，正如保罗所说，有几种方法可以做到这一点。使用 rstrip 你需要告诉 python 你想剥离什么。 .rstrip('\r\n\t') 之类的东西会告诉它剥离 EOL 和标签。这可能会有所帮助：stackoverflow.com/questions/10711116/…

标签： python regex scrapy

【解决方案1】：

我用过：

def parse(self, response):
    sel = Selector(response)
    sites = sel.xpath('//ul/li')
    items = []
    for site in sites:
        item = DmozItem()
        item['title'] = map(unicode.strip,site.xpath('a/text()').extract())
        item['link'] = map(unicode.strip, site.xpath('a/@href').extract())
        item['desc'] = map(unicode.strip, site.xpath('text()').extract())
        items.append(item)
    print "hello"
    return items

它有效。我不确定它是什么，但我仍在阅读 unicode.strip。我希望这会有所帮助

【讨论】：

【解决方案2】：

这是另一种方法（我使用了您的 JSON 数据）：

>>> data = {"desc": ["\r\n\t\t\t\r\n ", " \r\n\t\t\t\r\n - The primary goal of this book is to promote object-oriented design using Python and to illustrate the use of the emerging object-oriented design patterns.\r\nA secondary goal of the book is to present mathematical tools just in time. Analysis techniques and proofs are presented as needed and in the proper context.\r\n \r\n "], "link": ["http://www.brpreiss.com/books/opus7/html/book.html"], "title": ["Data Structures and Algorithms with Object-Oriented Design Patterns in Python"]}

>>> clean_data = ''.join(data['desc'])

>>> print clean_data.strip(' \r\n\t')

输出：

- The primary goal of this book is to promote object-oriented design using Python and to illustrate the use of the emerging object-oriented design patterns.
A secondary goal of the book is to present mathematical tools just in time. Analysis techniques and proofs are presented as needed and in the proper context.

代替：

['\r\n\t\t\t\r\n ', ' \r\n\t\t\t\r\n - The primary goal of this book is to promote object-oriented design using Python and to illustrate the use of the emerging object-oriented design patterns.\r\nA secondary goal of the book is to present mathematical tools just in time. Analysis techniques and proofs are presented as needed and in the proper context.\r\n \r\n ']

【讨论】：

【解决方案3】：

假设您想要 all \r、\n 和 \t 删除（不仅仅是边缘的东西），同时仍然保持 JSON 的形式，您可以尝试以下：

def normalize_whitespace(json):
    if isinstance(json, str):
        return ' '.join(json.split())

    if isinstance(json, dict):
        it = json.items() # iteritems in Python 2
    elif isinstance(json, list):
        it = enumerate(json)

    for k, v in it:
        json[k] = normalize_whitespace(v)

    return json

用法：

>>> normalize_whitespace({"desc": ["\r\n\t\t\t\r\n ", " \r\n\t\t\t\r\n - The primary goal of this book is to promote object-oriented design using Python and to illustrate the use of the emerging object-oriented design patterns.\r\nA secondary goal of the book is to present mathematical tools just in time. Analysis techniques and proofs are presented as needed and in the proper context.\r\n \r\n "], "link": ["http://www.brpreiss.com/books/opus7/html/book.html"], "title": ["Data Structures and Algorithms with Object-Oriented Design Patterns in Python"]})
{'title': ['Data Structures and Algorithms with Object-Oriented Design Patterns in Python'], 'link': ['http://www.brpreiss.com/books/opus7/html/book.html'], 'desc': ['', '- The primary goal of this book is to promote object-oriented design using Python and to illustrate the use of the emerging object-oriented design patterns. A secondary goal of the book is to present mathematical tools just in time. Analysis techniques and proofs are presented as needed and in the proper context.']}

正如https://stackoverflow.com/a/10711166/138772 所提醒的那样，split-join 方法可能比正则表达式替换更好，因为它结合了strip 功能和空白规范化。

【讨论】：