mongo-conector 未与 solr 连接 - 收集转储期间出现异常

Question

我正在用 solr 连接 MongoDB，

按照本文档进行集成： https://blog.toadworld.com/2017/02/03/indexing-mongodb-data-in-apache-solr

DB.Collection： solr.wlslog

D:\solr\bin 的路径>

mongo-connector --unique-key=id -n solr.wlslog -m localhost:27017 -t http://localhost:8983/solr/wlslog -d solr_doc_manager

我收到以下响应和错误：

2020-06-15 12:15:45,744 [ALWAYS] mongo_connector.connector:50 - Starting mongo-connector version: 3.1.1
2020-06-15 12:15:45,744 [ALWAYS] mongo_connector.connector:50 - Python version: 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)]
2020-06-15 12:15:45,745 [ALWAYS] mongo_connector.connector:50 - Platform: Windows-10-10.0.18362-SP0
2020-06-15 12:15:45,745 [ALWAYS] mongo_connector.connector:50 - pymongo version: 3.10.1
2020-06-15 12:15:45,755 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 4.2.2
2020-06-15 12:15:45,755 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.solr_doc_manager version: 0.1.0
2020-06-15 12:15:45,787 [CRITICAL] mongo_connector.oplog_manager:713 - Exception during collection dump
Traceback (most recent call last):
File "C:\Users\ancubate\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\mongo_connector\doc_managers\solr_doc_manager.py", line 292, in
batch = list(next(cleaned) for i in range(self.chunk_size))
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\ancubate\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\mongo_connector\oplog_manager.py", line 668, in do_dump
upsert_all(dm)
File "C:\Users\ancubate\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\mongo_connector\oplog_manager.py", line 651, in upsert_all
dm.bulk_upsert(docs_to_dump(from_coll), mapped_ns, long_ts)
File "C:\Users\ancubate\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\mongo_connector\util.py", line 33, in wrapped
return f(*args, **kwargs)
File "C:\Users\ancubate\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\mongo_connector\doc_managers\solr_doc_manager.py", line 292, in bulk_upsert
batch = list(next(cleaned) for i in range(self.chunk_size))
RuntimeError: generator raised StopIteration
2020-06-15 12:15:45,801 [ERROR] mongo_connector.oplog_manager:723 - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='rs0'), 'local'), 'oplog.rs')
2020-06-15 12:15:46,782 [ERROR] mongo_connector.connector:408 - MongoConnector: OplogThread <OplogThread(Thread-2, started 4936)> unexpectedly stopped! Shutting down

我在mongo-connector 的 GitHub 问题中进行了搜索，但没有得到任何解决方案：

Github-issue-870

Github-issue-898

Answer 1

问题终于解决了:)

我的系统操作系统是windows，我已经在C:\Program Files\MongoDB\（系统驱动器）中安装了mongodb，

在这个 mongo-connector 连接之前，我已经按照 blog 使用以下命令为 mongodb 启动了副本集：

mongod --port 27017 --dbpath ../data/db --replSet rs0

问题：

--dbpath ../data/db目录里面的问题，这个目录位于C:\Program Files\MongoDB\Server\4.2\data\db这个目录有所有权限但是父目录C:\Program Files没有所有权限，因为它的系统目录和受保护的目录。

实际问题是： （收集转储期间出现异常）

2020-06-15 12:15:45,787 [CRITICAL] mongo_connector.oplog_manager:713 - Exception during collection dump

解决方案：

我刚刚将我的--dbpath 更改为该目录位于系统受保护目录之外的另一个路径，如下所示：

mongod --port 27017 --dbpath C:/data/db --replSet rs0

之后我执行了以下连接命令，正如我在我的问题中发布的那样：

mongo-connector --unique-key=id -n solr.wlslog -m localhost:27017 -t http://localhost:8983/solr/wlslog -d solr_doc_manager

成功的mongo连接器日志结果：

2020-06-17 12:08:52,292 [ALWAYS] mongo_connector.connector:50 - Starting mongo-connector version: 3.1.1
2020-06-17 12:08:52,292 [ALWAYS] mongo_connector.connector:50 - Python version: 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)]
2020-06-17 12:08:52,293 [ALWAYS] mongo_connector.connector:50 - Platform: Windows-10-10.0.18362-SP0
2020-06-17 12:08:52,293 [ALWAYS] mongo_connector.connector:50 - pymongo version: 3.10.1
2020-06-17 12:08:52,310 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 4.2.2
2020-06-17 12:08:52,311 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.solr_doc_manager version: 0.1.0

希望这个答案对每个人都有帮助:)

Answer 2

就我而言，这并没有解决问题。 我使用的是python 3.8，所以对我来说这实际上是由于https://docs.python.org/3/whatsnew/3.7.html#changes-in-python-behavior

为 Python 3.7 中的所有代码启用 PEP 479，这意味着 协程中直接或间接引发的 StopIteration 异常 和生成器被转换为 RuntimeError 异常。 （由 Yury Selivanov 在 bpo-32670 中贡献。）

阅读 How yield catches StopIteration exception? 让我最初认为它与 yield doc 语句有关，但实际上问题是在第 292 行和后面的 solr_doc_manager.py 中调用 next() 的两行代码：

batch = list(next(cleaned) for i in range(self.chunk_size))

改为：

        batch = []
        for i in range(self.chunk_size):
            for x in cleaned:
                batch.append(x)