ElasticSearch 逗号里面的数字答案

【问题标题】：ElasticSearch comma inside the numberElasticSearch 逗号里面的数字
【发布时间】：2017-07-09 20:57:33
【问题描述】：

我想删除数字中的逗号，例如“Warhammer 40,000: Dawn of War III”

如果您搜索“40000”，我希望它匹配。

但目前我的分词器给了我：

{
"tokens": [
{
"token": "warhammer",
"start_offset": 0,
"end_offset": 9,
"type": "word",
"position": 0
},
{
"token": "warhammer 40",
"start_offset": 0,
"end_offset": 12,
"type": "shingle",
"position": 0
},
{
"token": "40",
"start_offset": 10,
"end_offset": 12,
"type": "word",
"position": 1
},
{
"token": "000:",
"start_offset": 13,
"end_offset": 17,
"type": "word",
"position": 102
},
{
"token": "000: 000",
"start_offset": 13,
"end_offset": 16,
"type": "shingle",
"position": 102
},
{
"token": "000: 000 dawn",
"start_offset": 13,
"end_offset": 22,
"type": "shingle",
"position": 102
},
{
"token": "000: 000 dawn of",
"start_offset": 13,
"end_offset": 25,
"type": "shingle",
"position": 102
},
{
"token": "000: 000 dawn of war",
"start_offset": 13,
"end_offset": 29,
"type": "shingle",
"position": 102
},
{
"token": "000: 000 dawn of war 3",
"start_offset": 13,
"end_offset": 33,
"type": "shingle",
"position": 102
},
{
"token": "000",
"start_offset": 13,
"end_offset": 16,
"type": "word",
"position": 103
},
{
"token": "000 dawn",
"start_offset": 13,
"end_offset": 22,
"type": "shingle",
"position": 103
},
{
"token": "000 dawn of",
"start_offset": 13,
"end_offset": 25,
"type": "shingle",
"position": 103
},
{
"token": "000 dawn of war",
"start_offset": 13,
"end_offset": 29,
"type": "shingle",
"position": 103
},
{
"token": "000 dawn of war 3",
"start_offset": 13,
"end_offset": 33,
"type": "shingle",
"position": 103
},
{
"token": "dawn",
"start_offset": 18,
"end_offset": 22,
"type": "word",
"position": 104
},
{
"token": "dawn of",
"start_offset": 18,
"end_offset": 25,
"type": "shingle",
"position": 104
},
{
"token": "dawn of war",
"start_offset": 18,
"end_offset": 29,
"type": "shingle",
"position": 104
},
{
"token": "dawn of war 3",
"start_offset": 18,
"end_offset": 33,
"type": "shingle",
"position": 104
},
{
"token": "of war",
"start_offset": 23,
"end_offset": 29,
"type": "shingle",
"position": 105
},
{
"token": "of war 3",
"start_offset": 23,
"end_offset": 33,
"type": "shingle",
"position": 105
},
{
"token": "war",
"start_offset": 26,
"end_offset": 29,
"type": "word",
"position": 106
},
{
"token": "war 3",
"start_offset": 26,
"end_offset": 33,
"type": "shingle",
"position": 106
},
{
"token": "3",
"start_offset": 30,
"end_offset": 33,
"type": "SYNONYM",
"position": 107
}
]
}

这里的主要问题是“40”和“000”是不同的标记。我认为最好将它们视为单个令牌“40000”是否有可以合并两者的令牌过滤器？

编辑： 哦！我试过了：

      "analyzer": {
    "default": {
      "tokenizer": "keyword"
    }}

结果： http://localhost:9200/i/_analyze?text=Warhammer%2040,000:%20Dawn%20of%20War%20III 给我：

    {
"tokens": [
{
"token": "Warhammer 40",
"start_offset": 0,
"end_offset": 12,
"type": "word",
"position": 0
},
{
"token": "000: Dawn of War III",
"start_offset": 13,
"end_offset": 33,
"type": "word",
"position": 101
}
]
}

【问题讨论】：

标签： elasticsearch

【解决方案1】：

您可以使用字符过滤器合并带小数点的数字。在下面的 sn-p 中，名为“decimal_mark_filter”的字符过滤器将在标记化之前删除出现在数字之间的任何逗号。

{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "standard",
          "char_filter": [
            "decimal_mark_filter"
          ]
        }
      },
      "char_filter": {
        "decimal_mark_filter": {
          "type": "pattern_replace",
          "pattern": "(\\d+),(?=\\d)",
          "replacement": "$1"
        }
      }
    }
  }
}

分析器给出以下标记：

{
  "tokens": [
    {
      "token": "Warhammer",
      "start_offset": 0,
      "end_offset": 9,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "40000",
      "start_offset": 10,
      "end_offset": 16,
      "type": "<NUM>",
      "position": 1
    },
    {
      "token": "Dawn",
      "start_offset": 18,
      "end_offset": 22,
      "type": "<ALPHANUM>",
      "position": 2
    },
    {
      "token": "of",
      "start_offset": 23,
      "end_offset": 25,
      "type": "<ALPHANUM>",
      "position": 3
    },
    {
      "token": "War",
      "start_offset": 26,
      "end_offset": 29,
      "type": "<ALPHANUM>",
      "position": 4
    },
    {
      "token": "III",
      "start_offset": 30,
      "end_offset": 33,
      "type": "<ALPHANUM>",
      "position": 5
    }
  ]
}

这只是对the official Elasticsearch documentation上一个例子的修改

【讨论】：