使用 logstash 将 mongo 数据同步到弹性答案

【问题标题】：sync mongo data to elastic using logstash使用 logstash 将 mongo 数据同步到弹性
【发布时间】：2020-02-27 16:05:30
【问题描述】：

我想使用 mongodb 的 logstash-plugin 将我的 mongodb 数据（本地 mongodb）同步到弹性搜索（本地弹性）

我已经安装了logstash插件使用

bin/logstash-plugin install logstash-input-mongodb .

然后我在/usr/share/logstash 目录中创建了一个 mongodata.conf 文件。当我执行conf文件时，它显示 --> Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

我的配置文件是：

input{ 
 mongodb{
   uri => "mongodb://localhost:27017/reporterDB"
   placeholder_db_dir => "/opt/logstash-mongodb/"
   placeholder_db_name => "logstash_sqlite.db"
   collection => "iam_ms_test"
   batch_size => 5000
  }
}
filter{ 

 }
output {
    stdout { codec => rubydebug }
    elasticsearch {
            action => "index"
            hosts => "localhost:9200"
            user => elastic
    password => changeme
            index => "mongo_log"
            document_type => "document_type"
            document_id => "%{id}"     
    }
  }

我在 logstash-plain.log 文件中得到以下行

[2019-11-01T15:41:00,869][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-11-01T15:41:00,871][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>750, :thread=>"#<Thread:0x351f7fd1@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,068][INFO ][logstash.inputs.mongodb  ] Registering MongoDB input
[2019-11-01T15:41:01,116][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::MongoDB uri=>\"mongodb://localhost:27017/anchorReports\", placeholder_db_dir=>\"/opt/logstash-mongodb/\", placeholder_db_name=>\"logstash_sqlite.db\", collection=>\"hi_p5m\", batch_size=>5000, id=>\"ec7682e8c6c5676deca84d5072c5f7865120a107ffce81ce21caa878c6e4ed09\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_441f95b8-cc8a-4b9e-a45f-657ed2011e2b\", enable_metric=>true, charset=>\"UTF-8\">, since_table=>\"logstash_since\", since_column=>\"_id\", since_type=>\"id\", parse_method=>\"flatten\", isodate=>false, retry_delay=>3, generateId=>false, unpack_mongo_id=>false, message=>\"Default message...\", interval=>1>", :error=>"Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist", :thread=>"#<Thread:0x351f7fd1@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2019-11-01T15:41:01,869][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Sequel::DatabaseConnectionError: Java::JavaSql::SQLException: path to '/opt/logstash-mongodb/logstash_sqlite.db': '/opt/logstash-mongodb' does not exist>, :backtrace=>["org.sqlite.core.CoreConnection.open(org/sqlite/core/CoreConnection.java:190)", "org.sqlite.core.CoreConnection.<init>(org/sqlite/core/CoreConnection.java:74)", "org.sqlite.jdbc3.JDBC3Connection.<init>(org/sqlite/jdbc3/JDBC3Connection.java:24)", "org.sqlite.jdbc4.JDBC4Connection.<init>(org/sqlite/jdbc4/JDBC4Connection.java:23)", "org.sqlite.SQLiteConnection.<init>

"(org/sqlite/SQLiteConnection.java:45)", 

"org.sqlite.JDBC.createConnection(org/sqlite/JDBC.java:114)", 

"org.sqlite.JDBC.connect(org/sqlite/JDBC.java:88)"

我想要在 `index("mongo_log") 下的弹性搜索记录。我还想知道 placeholder_db_dir 和 placeholder_db_name 的用途，当我们使用 mongodb 作为输入数据库时，这些值应该是什么。

【问题讨论】：

标签： mongodb elasticsearch logstash logstash-configuration

【解决方案1】：

问题解决了！实际上目录 opt/logstash 没有被创建。所以我在opt下手动创建了logstash文件夹。之后我给了那个目录的写权限，这样当我们执行logstash的命令时，它就可以在这个文件夹中创建文件了。

【讨论】：