如何提高弹性搜索性能

Question

我用python中的parallel_bulk函数写数据到elasticsearch，但是性能很低，我写10000条数据，耗时180s，我设置settings：

"settings": {
            "number_of_shards": 5,
            "number_of_replicas": 0,
            "refresh_interval": "30s",
            "index.translog.durability": "async",
            "index.translog.sync_interval": "30s"
       }

在 elasticsearch.yml 中，我设置了：

bootstrap.memory_lock: true
indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 96mb
# Search pool
thread_pool.search.size: 5
thread_pool.search.queue_size: 100

thread_pool.bulk.queue_size: 300
thread_pool.index.queue_size: 300

indices.fielddata.cache.size: 40%

discovery.zen.fd.ping_timeout: 120s
discovery.zen.fd.ping_retries: 6
discovery.zen.fd.ping_interval: 30s

但它并没有提高性能，我该怎么做？ 我在windows10上使用elasticsearch6.5.4，并且只有一个节点，我从Oracle产生数据到elasticsearch。

Answer 1

根据昨天帖子的代码，可以尝试创建oracle DB的es dump：

class CreateDump(object):
def __init__():
    self.output = r"/home/littlely/Scrivania/oracle_dump.json"
    self.index_name = "your_index_name"
    self.doc_type = "your_doc_type"
def _gen_data(self, index, doc_type, chunk_size):
    sql = """select * from tem_search_engine_1 where rownum <= 10000"""  
    self.cursor.execute(sql)
    col_name_list = [col[0].lower() for col in self.cursor.description]
    col_name_len = len(col_name_list)
    actions = []

    start = time.time()
    for row in self.cursor:
        source = {}
        tbl_id = ""
        for i in range(col_name_len):
            source.update({col_name_list[i]: str(row[i])})
            if col_name_list[i] == "tbl_id":
                tbl_id = row[i]
    self.writeOnFS(source, tbl_id)

def writeOnFS(source, tbl_id):
        with open(self.output, 'a') as f:
            prep = json.dumps({"index":{"_index" : self.index_name, "_type" : self.doc_type, "_id" : tbl_id}})
            data = json.dumps(source)
            print(data)
            f.write(prep + " \n")
            f.write(data + " \n") 

然后您将在self.output 路径中找到oracle 转储。所以你只需要批量处理你的 json 文件——二进制路径是 self.output 路径：

curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/<your_index_name>/<your_doc_type)/_bulk --data-binary @/home/littlely/Scrivania/oracle_dump.json

或者如果它太大，安装 GNU PARAllEl。在 Ubuntu 中：

sudo apt-get install parallel

然后：

cat /home/littlely/Scrivania/oracle_dump.json.json | parallel --pipe -L 2 -N 2000 -j3 'curl -H "Content-Type: application/x-ndjson" -s http://localhost:9200/<your_index_name>/_bulk --data-binary @- > /dev/null'

享受吧！