获取基本 Eland 示例的错误（从本地安装的 ELK docker 容器加载索引）答案

【问题标题】：Getting error for basic Eland example (loading index from a locally installed ELK docker container)获取基本 Eland 示例的错误（从本地安装的 ELK docker 容器加载索引）
【发布时间】：2022-12-06 21:17:55
【问题描述】：

我们基于this example在docker中安装了ELK。喜欢：

docker run -d --name elasticsearchdb --net es-stack-network -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:6.8.13

docker run -d --name kibana-es-ui --net es-stack-network -e "ELASTICSEARCH_URL=http://elasticsearchdb:9200" -p 5601:5601 kibana:6.8.13

然后我们使用基本的内置数据集设置 Elastic，包括默认提供的航班数据集。

然后我们尝试使用 Eland 将该数据拉入数据框，我认为我们正确地遵循了 documentation。

但是使用代码：

    import eland as ed
    index_name = 'flights'
    ed_df = ed.DataFrame('localhost:9200', index_name)

我们得到这个错误：

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\elastic_transport\client_utils.py:198, in url_to_node_config(url)
    192     raise ValueError(f"Could not parse URL {url!r}") from None
    194 if any(
    195     component in (None, "")
    196     for component in (parsed_url.scheme, parsed_url.host, parsed_url.port)
    197 ):
--> 198     raise ValueError(
    199         "URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')"
    200     )
    202 headers = {}
    203 if parsed_url.auth:

ValueError: URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')

所以当我们添加http://时，像这样：

    import eland as ed
    index_name = 'flights'
    ed_df = ed.DataFrame('http://localhost:9200', index_name)

我们得到这个错误：

File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\elastic_transport\_node\_http_urllib3.py:199, in Urllib3HttpNode.perform_request(self, method, target, body, headers, request_timeout)
    191         err = ConnectionError(str(e), errors=(e,))
    192     self._log_request(
    193         method=method,
    194         target=target,
   (...)
    197         exception=err,
    198     )
--> 199     raise err from None
    201 meta = ApiResponseMeta(
    202     node=self.config,
    203     duration=duration,
   (...)
    206     headers=response_headers,
    207 )
    208 self._log_request(
    209     method=method,
    210     target=target,
   (...)
    214     response=data,
    215 )

ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))))

所以我想，好吧，也许由于某种原因它默认在 HTTPS 上提供服务，也许不相关但在我看到的日志中：

05T17:17:04.734Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.18.0.3:9200, remoteAddress=/172.18.0.1:59788}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[21683dc12cff][transport_worker][T#14]","log.logger":"org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4HttpServerTransport","elasticsearch.cluster.uuid":"XuzqXMk_QgShA3L5HnfXgw","elasticsearch.node.id":"H1CsKboeTyaFFjk2-1nw2w","elasticsearch.node.name":"21683dc12cff","elasticsearch.cluster.name":"docker-cluster"}

所以我尝试用 https 替换 http 并得到这个错误：

TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)))

所以我查找这个错误并找到 this thread 说做类似的事情：

import ssl
from elasticsearch.connection import create_ssl_context

ssl_context = create_ssl_context(<use `cafile`, or `cadata` or `capath` to set your CA or CAs)
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE

es = Elasticsearch('localhost', ssl_context=context, timeout=60

但这没有帮助，因为 Eland 在内部处理 elasticsearch 实例化，我无法控制它。

这是一个非常基本的场景，所以我相信解决方案一定比这一切简单得多。我该怎么做才能完成这项工作？

【问题讨论】：

标签： elasticsearch elk eland

【解决方案1】：

对于仍在苦苦挣扎的人，以下内容对我使用 Docker/docker-compose 的本地 Elastic 集群有效：

Following this guide 你会在本地有 http_ca.crt 文件，使用命令：

docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .

您可以在创建 es_client 时使用 http_ca.crt 文件：

from elasticsearch import Elasticsearch
es_client = Elasticsearch("https://localhost:9200",
                          ca_certs="/path/to/http_ca.crt",
                          basic_auth=("[elastic username]",
                                      "[elastic password]"))

然后使用 es_client 连接到 eland：

import eland as ed
df = ed.DataFrame(es_client=es_client, es_index_pattern="[Your index]")
df.head()

【讨论】：