将 Hashtags 和 Mentions 提取到单独的字段中

Question

我正在做一个 DIY 推文情绪分析器，我有一个类似这样的推文索引

"_source" : {
      "id" : 26930655,
      "status" : 1,
      "title" : "Here’s 5 underrated #BTC and realistic crypto accounts that everyone should follow:  @Quinnvestments , @JacobOracle , @jevauniedaye , @ginsbergonomics , @InspoCrypto",
      "hashtags" : null,
      "created_at" : 1622390229,
      "category" : null,
      "language" : 50
    },
    {
          "id" : 22521897,
          "status" : 1,
          "title" : "#bulls gonna overtake the #bears soon #ATH coming #ALTSEASON #BSCGem #eth #btc #memecoin #100xgems #satyasanatan ????????????????????""",
          "hashtags" : null,
          "created_at" : 1620045296,
          "category" : null,
          "language" : 50
    }

有映射是设置就像

"sentiment-en" : {
    "mappings" : {
      "properties" : {
        "category" : {
          "type" : "text"
        },
        "created_at" : {
          "type" : "integer"
        },
        
        "hashtags" : {
          "type" : "text"
        },
        "id" : {
          "type" : "long"
        },
        "language" : {
          "type" : "integer"
        },
        "status" : {
          "type" : "integer"
        },
        "title" : {
          "type" : "text",
          "fields" : {
            "raw" : {
              "type" : "keyword"
            },
            "raw_text" : {
              "type" : "text"
            },
            "stop" : {
              "type" : "text",
              "index_options" : "docs",
              "analyzer" : "stop_words_filter"
            },
            "syn" : {
              "type" : "text",
              "index_options" : "docs",
              "analyzer" : "synonyms_filter"
            }
          },
          "index_options" : "docs",
          "analyzer" : "all_ok_filter"
        }
      }
    }
  }
}




"settings" : {
      "index" : {
        "number_of_shards" : "10",
        "provided_name" : "sentiment-en",
        "creation_date" : "1627975717560",
        "analysis" : {
          "filter" : {
            "stop_words" : {
              "type" : "stop",
              "stopwords" : [ ]
            },
            "synonyms" : {
              "type" : "synonym",
              "synonyms" : [ ]
            }
          },
          "analyzer" : {
            "stop_words_filter" : {
              "filter" : [ "stop_words" ],
              "tokenizer" : "standard"
            },
            "synonyms_filter" : {
              "filter" : [ "synonyms" ],
              "tokenizer" : "standard"
            },
            "all_ok_filter" : {
              "filter" : [ "stop_words", "synonyms" ],
              "tokenizer" : "standard"
            }
          }
        },
        "number_of_replicas" : "0",
        "uuid" : "Q5yDYEXHSM-5kvyLGgsYYg",
        "version" : {
          "created" : "7090199"
        }
      }

现在的问题是我想在一个单独的字段中提取所有标签和提及。

我想要什么作为O/P

          "id" : 26930655,
          "status" : 1,
          "title" : "Here’s 5 underrated #BTC and realistic crypto accounts that everyone should follow:  @Quinnvestments , @JacobOracle , @jevauniedaye , @ginsbergonomics , @InspoCrypto",
          "hashtags" : BTC,
          "created_at" : 1622390229,
          "category" : null,
          "language" : 50
        },
        {
              "id" : 22521897,
              "status" : 1,
              "title" : "#bulls gonna overtake the #bears soon #ATH coming #ALTSEASON #BSCGem #eth #btc #memecoin #100xgems #satyasanatan ????????????????????""",
              "hashtags" : bulls,bears,ATH, ALTSEASON, BSCGem, eth , btc, memecoin, 100xGem, satyasanatan
              "created_at" : 1620045296,
              "category" : null,
              "language" : 50
        }

到目前为止我已经尝试过什么

1. 创建一个基于模式的标记器以仅读取 Hashtags 和提及，而没有其他用于字段标签和提及的标记在那里没有取得多大成功。

2. 尝试在没有任何分析器的情况下编写 n-gram 标记器也没有取得多大成功。

任何帮助将不胜感激，我愿意重新索引我的数据。提前谢谢！！！

Answer 1

您可以使用Logstash Twitter input plugin 来索引数据，并在blog 中提到的过滤器插件中的ruby 脚本下方配置。

if [message] {
  ruby {
    code => "event.set('hashtags', event.get('message').scan(/\#[a-z]*/i))"
  }
}

您可以将 Logtstash Elasticsearch Input 插件用于源索引，并在 Filter 插件和 Logtstash elasticsearch output plugin 中配置有关 ruby​​ 代码的目标索引。

input {
      elasticsearch {
        hosts => "localhost:9200"
        index => "current_twitter"
        query => '{ "query": { "query_string": { "query": "*" } } }'
        size => 500
        scroll => "5m"
      }
    }
filter{
    if [message] {
      ruby {
        code => "event.set('hashtags', event.get('message').scan(/\#[a-z]*/i))"
      }
    }
}   
output {
    elasticsearch {
        index => "new_twitter"
    }
}

另一种选择是将reingest API 与摄取管道一起使用，但摄取管道不支持 ruby​​ 代码。所以你需要将上面的 ruby​​ 代码转换为无痛脚本。