无法将数据加载到logstash答案

【问题标题】：Unable to load data into logstash无法将数据加载到logstash
【发布时间】：2017-03-31 18:47:09
【问题描述】：

我正在尝试按照以下链接上的指南进行操作：

http://www.viaboxx.de/code/easily-generate-live-heatmaps-for-geolocations-with-elk/#codesyntax_1

第一次对我来说效果很好，但是当我现在尝试时，它在我尝试加载 csv 数据的步骤中给出了以下错误。我执行的命令是：

cat test.csv | /opt/logstash/bin/logstash -f geostore.conf

我收到以下错误：

Settings: Default pipeline workers: 2
Pipeline main started
Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>, :level=>:warn}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

请帮忙！！！我花了几天时间试图弄清楚。

编辑添加geostore.conf：

 input { stdin {} } 
    filter { # Step 1, drop the csv header line 
       if [message] =~ /^#/ { 
          drop {} 
        } # Step 2, split latitude and longitude 
        csv { 
          separator => ',' 
          columns => [ 'lat', 'lon' ] } 
           # Step 3 # move lat and lon into location object # for defined geo_point type in ES 
        mutate { 
             rename => [ "lat", "[location][lat]", "lon", "[location][lon]" ] 
         } 
       } 
    output { 
       elasticsearch {
         hosts => 'localhost' 
         index => 'geostore' 
         document_type => "locality" 
         flush_size => 1000 
      } 
    }

我已经改变了我的输出部分：

output {
  elasticsearch {
    hosts => 'localhost'
    index => 'geostore'
    document_type  => "locality"
    flush_size => 1000
  }

到这里

output {
  elasticsearch {
    hosts => 'localhost'
    index => 'geostore'
    document_type  => "locality"
    flush_size => 1000
    stdout {}
  }

现在我收到了更详细的错误消息：

    fetched an invalid config {:config=>"input {\n    stdin {}\n}\nfilter {\n  #
 Step 1, drop the csv header line\n  if [message] =~ /^#/ {\n    drop {}\n  }\n 
\n  # Step 2, split latitude and longitude\n  csv {\n    separator => ','\n    
columns => [ 'lat', 'lon' ]\n  }\n \n  # Step 3\n  # move lat and lon into 
location object \n  # for defined geo_point type in ES\n  mutate {  \n    rename 
=> [ \"lat\", \"[location][lat]\", \"lon\", \"[location][lon]\" ]\n  
}\n}\noutput {\n  elasticsearch {\n    hosts => 'localhost'\n    index => 
'geostore'\n    document_type  => \"locality\"\n    flush_size => 1000\n    
stdout {}\n  }\n}\n\n", :reason=>"Expected one of #, => at line 29, column 12 
(byte 543) after output {\n  elasticsearch {\n    hosts => 'localhost'\n    
index => 'geostore'\n    document_type  => \"locality\"\n    flush_size => 
1000\n    stdout ", :level=>:error}

不明白为什么它第一次起作用。

Settings: Default pipeline workers: 2
Pipeline main started
Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>, :level=>:warn}
2017-03-30T13:46:31.171Z localhost.localdomain 53.97917361, -6.389038611
2017-03-30T13:46:31.171Z localhost.localdomain 54.00310028, -6.397707778
2017-03-30T13:46:31.172Z localhost.localdomain 53.99960056, -6.381966111
2017-03-30T13:46:31.172Z localhost.localdomain 54.00534917, -6.423718889
2017-03-30T13:46:31.172Z localhost.localdomain 51.92071667, -8.475726111
2017-03-30T13:46:31.172Z localhost.localdomain 51.82731222, -8.381912222
2017-03-30T13:46:31.173Z localhost.localdomain 51.81096639, -8.415731667
2017-03-30T13:46:31.173Z localhost.localdomain 54.28450222, -8.463775556
2017-03-30T13:46:31.173Z localhost.localdomain 54.27841, -8.495700278
2017-03-30T13:46:31.173Z localhost.localdomain 54.2681225, -8.462056944
2017-03-30T13:46:31.174Z localhost.localdomain 52.276167, -9.680497
2017-03-30T13:46:31.174Z localhost.localdomain 52.25660139, -9.703921389
2017-03-30T13:46:31.174Z localhost.localdomain 52.27031306, -9.723975556
2017-03-30T13:46:31.174Z localhost.localdomain 54.95663111, -7.714384167
2017-03-30T13:46:31.175Z localhost.localdomain 54.00133111, -7.352790833
2017-03-30T13:46:31.175Z localhost.localdomain 52.34264222, -6.4854175
2017-03-30T13:46:31.176Z localhost.localdomain 52.32439028, -6.464626111
2017-03-30T13:46:31.176Z localhost.localdomain 52.33008944, -6.487005
2017-03-30T13:46:31.176Z localhost.localdomain 53.70765861, -6.374657778
2017-03-30T13:46:31.177Z localhost.localdomain 53.72636306, -6.326768611
2017-03-30T13:46:31.177Z localhost.localdomain 53.71461361, -6.336066111
2017-03-30T13:46:31.177Z localhost.localdomain 51.55948417, -9.244535833
2017-03-30T13:46:31.177Z localhost.localdomain 53.52894667, -7.358543056
2017-03-30T13:46:31.177Z localhost.localdomain 53.51801167, -7.324215
2017-03-30T13:46:31.179Z localhost.localdomain 53.16202278, -6.795522222
2017-03-30T13:46:31.179Z localhost.localdomain 53.182702, -6.819299
2017-03-30T13:46:31.179Z localhost.localdomain 52.83053972, -8.991989444
2017-03-30T13:46:31.180Z localhost.localdomain 52.85651944, -8.965725833
2017-03-30T13:46:31.180Z localhost.localdomain 53.02885028, -7.300381667
2017-03-30T13:46:31.180Z localhost.localdomain
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

【问题讨论】：

您使用的是哪个 Logstash 版本？您能否也发布您的 geostore.conf 以确保它没有问题？最后，在输出部分，您能否添加stdout{codec => rubydebug}，因为我发现它通常有助于找到问题
感谢您回来。请参阅我的 geostore.conf。我的 logstash 版本是 2.3.4
它对我来说很好，你确定你的 csv 文件吗？正如我所提到的，尝试使用输出部分中的标准输出来获取有关该问题的更多详细信息。您也可以尝试直接输入一行而不使用您的 test.csv，例如 50.98474812, 7.47645034
抱歉忘记了。如何直接输入一行？能否请您给出命令语法？
您等待“主管道已启动”被写入，然后您在标准输入上写入，顺便说一下，您在放置标准输出时的错误是因为您将其包含在弹性搜索部分而不是输出部分。跨度>

标签： csv elasticsearch logstash

【解决方案1】：

Hopefull，这也会对其他人有所帮助。

我从命令行删除了模式：

curl -XDELETE 'localhost:9200/geostore?pretty';

然后去kibana也从那里删除它。如下所示再次重新加载模式并且它起作用了。

    curl -XPUT 'http://localhost:9200/geostore'

curl -XPUT 'http://localhost:9200/geostore/_mapping/locality' -d '
{
   "locality" : {
       "properties" : {
              "location" : {
                  "type" : "geo_point",
                  "geohash_prefix":     true,
                  "geohash_precision":  "1km"
              }
          }
       }
}'

cat test.csv | /opt/logstash/bin/logstash -f geostore.conf

这将需要几秒钟来启动 logstash、解析输入并将结果存储到 Elasticsearch。

现在我们在 Elasticsearch 中有了数据，让我们转到 Kibana 4。登录到 Kibana 后，您需要将索引添加到 Kibana。

转到：设置->索引->添加新->在索引名称字段中写入“geostore”。

添加索引后，您会看到索引文档中的所有字段，特别是您应该检查属性位置是否归类为geo_point。

整个过程在下面的链接中有详细描述。

http://www.viaboxx.de/code/easily-generate-live-heatmaps-for-geolocations-with-elk/#codesyntax_1

【讨论】：