查询失败 [无法执行主查询] OutOfMemoryError: Java heap space

Question

嗨，我有弹性数据 80 M 的数据有 52 Go，我用这个查询在文本中搜索

$json_doc = '{
                 "query": {
                    "query_string": {
                        "query": "*' . $sujet . '*",
                        "lenient": true
                    }
                    },
     "size": 10000000,
     "from": 0,
     "fields": [
     "association_key","text"
                    ]
                }';

当我搜索基地中的一个单词时，我的数据看起来少于 20K，但当他超过 20K 时，我有这个错误

[2015-11-14 20:03:04,663][DEBUG][action.search.type       ] [Pixx] [myindex3][2], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][2]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:56)
    at org.apache.lucene.search.HitQueue.<init>(HitQueue.java:64)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:275)
    at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:37)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:135)
    at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:133)
    at org.apache.lucene.search.TopScoreDocCollector.create(TopScoreDocCollector.java:262)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:490)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more
[2015-11-14 20:03:11,764][DEBUG][action.search.type       ] [Pixx] [myindex3][0], node[x9gOi0WuQviXeE0eM6MsAg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@3989d22b] lastShard [true]
org.elasticsearch.search.query.QueryPhaseExecutionException: [myindex3][0]: query[filtered(_all:*startup _all:maroc*)->cache(_type:tweecoms3)],from[0],size[10000000]: Query Failed [Failed to execute main query]
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
    at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:286)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:297)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at org.apache.lucene.util.FixedBitSet.<init>(FixedBitSet.java:187)
    at org.elasticsearch.common.lucene.docset.DocIdSets.toCacheable(DocIdSets.java:81)
    at org.elasticsearch.index.cache.filter.weighted.WeightedFilterCache$FilterCacheFilterWrapper.getDocIdSet(WeightedFilterCache.java:177)
    at org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:46)
    at org.apache.lucene.search.FilteredQuery$1.bulkScorer(FilteredQuery.java:144)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
    at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157)
    ... 8 more

如何解决这个问题，我的专用服务器在 Debian 下仅包含 Intel Xeon W3520 c 4/8 t + 2.66 GHz RAM 32 GB ECC

Answer 1

更新：ES_HEAP_SIZE 不再是文档选项

Elastic Search 不会在未配置的情况下使用所有内存。

您需要配置和调整 Elastic Search 和 Debian 的内存设置。

您需要配置其中一项：

• /etc/elasticsearch/jvm.options：包含 JVM 选项的文件（文件位于 config/jvm.options（在 TAR 或 ZIP 发行版中，而不是 Debian 软件包中）
• ES_JAVA_OPTS：一个环境变量，带有额外的 JVM 设置，包括内存和 GC 调整（如果需要）（几乎是 jvm.options 中的内容，但在一行中，由空格分隔）

以下是 Elastic 截至 2020 年 2 月的最新文档的相关摘录：

以下是如何通过 jvm.options 设置堆大小的示例 文件：

-Xms2g ①
-Xmx2g ②

• ①设置最小堆大小为2g。
• ②设置最大堆大小为2g。

也可以通过环境变量设置堆大小。 这可以通过注释掉 Xms 和 Xmx 设置来完成 jvm.options 文件并通过 ES_JAVA_OPTS 设置这些值：

ES_JAVA_OPTS="-Xms2g -Xmx2g" ./bin/elasticsearch ①
ES_JAVA_OPTS="-Xms4000m -Xmx4000m" ./bin/elasticsearch ②

• ①设置 最小和最大堆大小为 2 GB。
• ② 设置最小和最大堆大小为 4000 MB。

在这里查看更多：

• Setting the heap size | Elasticsearch Reference [7.6] | Elastic
• Setting JVM options | Elasticsearch Reference [7.6] | Elastic

Answer 2

似乎您没有足够的空间用于 Java 堆。 由于您似乎拥有更多硬件方面的知识，您可以尝试增加它： Increase heap size in Java