【问题标题】:Elasticsearch/Nest performance issueElasticsearch/Nest 性能问题
【发布时间】:2019-08-25 20:21:45
【问题描述】:

我注意到 NEST 库中 ISearchResponse.HitsMetadata.Total 属性的行为很奇怪。每当我异步删除文档并想立即从 Elasticsearch 检索剩余文档时,ISearchResponse 对象上可用的 HitsMetadata.Total 字段几乎永远不会正确更新。它通常表示删除操作之前的文档总数。当我暂停执行请求至少 700 毫秒时,行为恢复正常,好像 NEST(或者可能是 Elasticsearch 本身)需要更多时间来更新属性的状态。我是使用 NEST 和 Elasticsearch 的新手,所以我可能在这里做错了,或者我可能不完全理解库的工作原理,但我在这个问题上花了很多时间却无法得到周围。结果,我发送给客户端的分页元数据计算错误。我正在使用 NEST 6.6.0 和 Elasticsearch 6.6.2。

删除操作:

[HttpDelete("errors/{index}/{logeventId}")]
public async Task<IActionResult> DeleteErrorLog([FromRoute] string index, [FromRoute] string logeventId)
{
    if (string.IsNullOrEmpty(index))
    {
        return BadRequest();
    }

    if (string.IsNullOrEmpty(logeventId)) 
    {
        return BadRequest();
    }

    var getResponse = await _client.GetAsync<Logevent>(new GetRequest(index, typeof(Logevent), logeventId));

    if(!getResponse.Found)
    {
        return NotFound();
    }

    var deleteResponse = await _client.DeleteAsync(new DeleteRequest(index, typeof(Logevent), logeventId));

    if (!deleteResponse.IsValid)
    {
        throw new Exception($"Deleting document id {logeventId} failed");
    }

    return NoContent();

}

GET 操作:

[HttpGet("errors/{index}", Name = "GetErrors")]
public async Task<IActionResult> GetErrorLogs([FromRoute] string index, 
    [FromQuery]int pageNumber = 1, [FromQuery] int pageSize = 5)
{
    if (string.IsNullOrEmpty(index))
    {
        return BadRequest();
    }

    if(pageSize > MAX_PAGE_SIZE || pageSize < 1)
    {
        pageSize = 5;
    }

    if(pageNumber < 1)
    {
        pageNumber = 1;
    }

    var from = (pageNumber - 1) * pageSize;

    ISearchResponse<Logevent> searchResponse = await GetSearchResponse(index, from, pageSize);

    if (searchResponse.Hits.Count == 0)
    {
        return NotFound();
    }

    int totalPages = GetTotalPages(searchResponse, pageSize);

    var previousPageLink = pageNumber > 1 ? 
        CreateGetLogsForIndexResourceUri(ResourceUriType.PreviousPage, pageNumber, pageSize, "GetErrors") : null;

    var nextPageLink = pageNumber < totalPages ? 
        CreateGetLogsForIndexResourceUri(ResourceUriType.NextPage, pageNumber, pageSize, "GetErrors") : null;

    /* HERE, WHEN EXECUTED IMMMEDIATELY (UP TO 700 MILISSECONDS, THE 
       totalCount FIELD GETS MISCALCULATED AS IT RETURNS THE VALUE PRECEDING 
       THE DELETION OF A DOCUMENT 
    */
    var totalCount = searchResponse.HitsMetadata.Total;
    var count = searchResponse.Hits.Count;

    var paginationMetadata = new
    {
        totalCount = searchResponse.HitsMetadata.Total,
        totalPages,
        pageSize,
        currentPage = pageNumber,
        previousPageLink,
        nextPageLink
    };

    Response.Headers.Add("X-Pagination", Newtonsoft.Json.JsonConvert.SerializeObject(paginationMetadata));

    var logeventsDtos = Mapper.Map<IEnumerable<LogeventDto>>(searchResponse.Hits);

    return Ok(logeventsDtos);
}

GetSearchResponse 方法:

private async Task<ISearchResponse<Logevent>> GetSearchResponse(string index, int from, int pageSize)
{
    return await _client.SearchAsync<Logevent>(s =>
             s.Index(index).From(from).Size(pageSize).Query(q => q.MatchAll()));

} 

客户端启动服务器端操作的代码:

async deleteLogevent(item){
    this.deleteDialog = false;
    let logeventId = item.logeventId;
    let level = this.defaultSelected.name;
    let index = 'logstash'.concat('-', this.defaultSelected.value, '-', this.date);

    LogsService.deleteLogevent(level, index, logeventId).then(response => {
      if(response.status == 204){
        let logeventIndex = this.logs.findIndex(element => {return element.logeventId === item.logeventId});
        this.logs.splice(logeventIndex, 1);
        LogsService.getLogs(level, index, this.pageNumber).then(reloadResponse => {
          this.logs.splice(0);
          reloadResponse.data.forEach(element => {
          this.logs.push(element)
          });
          this.setPaginationMetadata(reloadResponse.headers["x-pagination"]);
        })
      }
    }).catch(error => {

    })

【问题讨论】:

    标签: elasticsearch nest


    【解决方案1】:

    这是 Elasticsearch 的正常和预期行为。在出现刷新间隔之前,索引、更新和删除等操作的更改不会反映在对搜索请求的响应中。 Mike McCandless' blog post on how Lucene handles deleted documents 现在已经有几年了,但仍然很重要。 Elasticsearch 关于Near Real-Time Search 部分的在线权威指南也是一个很好的资源。

    这是一个演示行为的示例

    private static void Main()
    {
        var defaultIndex = "refresh_example";
        var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
    
        var settings = new ConnectionSettings(pool)
            .DefaultIndex(defaultIndex)
            .DefaultTypeName("_doc");
    
        var client = new ElasticClient(settings);
    
        if (client.IndexExists(defaultIndex).Exists)
            client.DeleteIndex(defaultIndex);
    
        client.CreateIndex(defaultIndex, c => c
            .Mappings(m => m
                .Map<Document>(mm => mm
                    .AutoMap()
                )
            )
        );
    
        var indexResponse = client.IndexDocument(new Document 
        {
            Id = 1,
            Name = "foo"
        });
    
        // hit count is likely to be 0 here because no refresh interval has occurred
        var searchResponse = client.Search<Document>();
        Console.WriteLine($"search hit count after index no refresh: {searchResponse.Hits.Count}");
    
        // a get for the exact document will return it however.
        var getResponse = client.Get<Document>(1);
        Console.WriteLine($"get document with id 1, name is: {getResponse.Source.Name}");
    
        // use refresh API to refresh the index
        var refreshResponse = client.Refresh(defaultIndex);
    
        // now the hit count is 1
        searchResponse = client.Search<Document>();
        Console.WriteLine($"search hit count after refresh: {searchResponse.Hits.Count}");
    
        // index another document, and refresh at the same time
        indexResponse = client.Index(new Document
        {
            Id = 2,
            Name = "bar"
        }, i => i.Refresh(Refresh.WaitFor));
    
        // now the hit count is 2
        searchResponse = client.Search<Document>();
        Console.WriteLine($"search hit count after index with refresh: {searchResponse.Hits.Count}");
    
        // now delete document with id 1
        var deleteResponse = client.Delete<Document>(1);
        Console.WriteLine($"document with id 1 deleted");
    
        // hit count is still 2
        searchResponse = client.Search<Document>();
        Console.WriteLine($"search hit count before refresh: {searchResponse.Hits.Count}");
    
        // refresh
        refreshResponse = client.Refresh(defaultIndex);
    
        // hit count is 1
        searchResponse = client.Search<Document>();
        Console.WriteLine($"search hit count after refresh: {searchResponse.Hits.Count}");
    }
    
    public class Document 
    {
        public int Id { get; set; }
    
        public string Name { get;set; }
    }
    

    这是写入控制台的内容

    search hit count after index no refresh: 0
    get document with id 1, name is: foo
    search hit count after refresh: 1
    search hit count after index with refresh: 2
    document with id 1 deleted
    search hit count before refresh: 2
    search hit count after refresh: 1
    

    您可能会想,“为什么我不刷新每个操作?”。不这样做的原因是性能;当您调用刷新 API 或将刷新指定为操作的一部分时,会写入并打开一个新段,该段使用系统资源,需要提交到磁盘并可能稍后与其他段合并。不断调用 refresh 会创建 很多 段。然而,在测试中调用 refresh 来进行断言很有用。

    最好编写您的应用程序以通过删除和搜索来处理这种近实时性质。对于分页,这对于其他数据存储也是类似的场景。

    【讨论】:

    • 感谢拉斯的解释。
    猜你喜欢
    • 1970-01-01
    • 2014-03-11
    • 1970-01-01
    • 2014-10-21
    • 1970-01-01
    • 1970-01-01
    • 2020-10-27
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多