solr, 太阳黑子, 错误请求, 非法字符

Question

我正在将太阳黑子搜索引入我的项目。我只通过名称字段搜索就获得了 POC。当我引入描述字段并重新索引已售出时，我收到以下错误。

** Invoke sunspot:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute sunspot:reindex
Skipping progress bar: for progress reporting, add gem 'progress_bar' to your Gemfile
rake aborted!
RSolr::Error::Http: RSolr::Error::Http - 400 Bad Request
Error: {'responseHeader'=>{'status'=>400,'QTime'=>18},'error'=>{'msg'=>'Illegal character ((CTRL-CHAR, code 11))
 at [row,col {unknown-source}]: [42,1]','code'=>400}}

Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><add><doc><field name=\"id\">ItemsDesign 1322</field><field name=\"type\">ItemsDesign</field><field name=\"type\">ActiveRecord::Base</field><field name=\"class_name\">ItemsDesign</field><field name=\"name_text\">River City Clocks Musical Multi-Colored Quartz Cuckoo Clock</field><field name=\"description_text\">This colorful chalet style German quartz cuckoo clock accurately keeps time and plays 12 different melodies. Many colorful flowers are painted on the clock case and figures of a Saint Bernard and Alpine horn player are on each side of the clock dial. Two decorative pine cone weights are suspended beneath the clock case by two chains. The heart shaped pendulum continously swings back and forth.&#13;On every

我假设坏字符是 你可以在底部看到。那 散布在很多描述中。我什至不确定那是什么字符。

我可以做些什么来让 solr 忽略它或清理数据以便售出可以处理它。

谢谢

Answer 1

您需要在保存内容时去掉 UTF8 中的控制字符。 Solr 不会正确地重新索引并抛出此错误。
http://en.wikipedia.org/wiki/UTF-8#Codepage_layout

你可以这样使用：

name.gsub!(/\p{Cc}/, "") 

编辑： 如果您想全局覆盖它，我认为可以通过覆盖 AttributeExtractor 中的 value_for_methods 以及如果需要 BlockExtractor 来实现。 https://github.com/sunspot/sunspot/blob/master/sunspot/lib/sunspot/data_extractor.rb 我没有检查这个。 如果您设法添加一些全局补丁，请告诉我。 我最近有同样的问题。

Answer 2

将以下内容放入初始化程序以自动清除任何 UTF8 控制字符的太阳黑子调用：

# config/initializers/sunspot.rb
module Sunspot
  # 
  # DataExtractors present an internal API for the indexer to use to extract
  # field values from models for indexing. They must implement the #value_for
  # method, which takes an object and returns the value extracted from it.
  #
  module DataExtractor #:nodoc: all
    # 
    # AttributeExtractors extract data by simply calling a method on the block.
    #
    class AttributeExtractor
      def initialize(attribute_name)
        @attribute_name = attribute_name
      end

      def value_for(object)
        Filter.new( object.send(@attribute_name) ).value
      end
    end

    # 
    # BlockExtractors extract data by evaluating a block in the context of the
    # object instance, or if the block takes an argument, by passing the object
    # as the argument to the block. Either way, the return value of the block is
    # the value returned by the extractor.
    #
    class BlockExtractor
      def initialize(&block)
        @block = block
      end

      def value_for(object)
        Filter.new( Util.instance_eval_or_call(object, &@block) ).value
      end
    end

    # 
    # Constant data extractors simply return the same value for every object.
    #
    class Constant
      def initialize(value)
        @value = value
      end

      def value_for(object)
        Filter.new(@value).value
      end
    end

    # 
    # A Filter to allow easy value cleaning
    #
    class Filter
      def initialize(value)
        @value = value
      end
      def value
        strip_control_characters @value
      end
      def strip_control_characters(value)
        return value unless value.is_a? String

        value.chars.inject("") do |str, char|
          unless char.ascii_only? and (char.ord < 32 or char.ord == 127)
            str << char
          end
          str
        end

      end
    end

  end
end

来源（Sunspot Github 问题）：Sunspot Solr Reindexing failing due to illegal characters

Answer 3

我尝试了@thekingoftruth 提出的解决方案，但并没有解决问题。在他链接到的同一个 github 线程中找到了 Filter 类的替代版本，这解决了我的问题。

主要区别在于我通过 HABTM 关系使用嵌套模型。

这是我在模型中的搜索块：

 searchable do
    text :name, :description, :excerpt
    text :venue_name do
      venue.name if venue.present?
    end
    text :artist_name do
      artists.map { |a| a.name if a.present? } if artists.present?
    end
  end

这是对我有用的初始化程序： （在：config/initializers/sunspot.rb）

module Sunspot
  #
  # DataExtractors present an internal API for the indexer to use to extract
  # field values from models for indexing. They must implement the #value_for
  # method, which takes an object and returns the value extracted from it.
  #
  module DataExtractor #:nodoc: all
    #
    # AttributeExtractors extract data by simply calling a method on the block.
    #
    class AttributeExtractor
      def initialize(attribute_name)
        @attribute_name = attribute_name
      end

      def value_for(object)
        Filter.new( object.send(@attribute_name) ).value
      end
    end

    #
    # BlockExtractors extract data by evaluating a block in the context of the
    # object instance, or if the block takes an argument, by passing the object
    # as the argument to the block. Either way, the return value of the block is
    # the value returned by the extractor.
    #
    class BlockExtractor
      def initialize(&block)
        @block = block
      end

      def value_for(object)
        Filter.new( Util.instance_eval_or_call(object, &@block) ).value
      end
    end

    #
    # Constant data extractors simply return the same value for every object.
    #
    class Constant
      def initialize(value)
        @value = value
      end

      def value_for(object)
        Filter.new(@value).value
      end
    end

    #
    # A Filter to allow easy value cleaning
    #
    class Filter
      def initialize(value)
        @value = value
      end

      def value
        if @value.is_a? String
          strip_control_characters_from_string @value
        elsif @value.is_a? Array
          @value.map { |v| strip_control_characters_from_string v }
        elsif @value.is_a? Hash
          @value.inject({}) do |hash, (k, v)|
            hash.merge( strip_control_characters_from_string(k) => strip_control_characters_from_string(v) )
          end
        else
          @value
        end
      end

      def strip_control_characters_from_string(value)
        return value unless value.is_a? String

        value.chars.inject("") do |str, char|
          unless char.ascii_only? && (char.ord < 32 || char.ord == 127)
            str << char
          end
          str
        end

      end
    end

  end
end