使用 Python 从 HDFS 读取文件时出现连接超时错误答案

【问题标题】：Connection timeout error while reading a file from HDFS using Python使用 Python 从 HDFS 读取文件时出现连接超时错误
【发布时间】：2018-02-02 02:49:13
【问题描述】：

我在 VM 中创建了一个单节点 HDFS（hadoop.master，IP：192.168.12.52）。文件etc/hadoop/core-site.xml的namenode配置如下：

<configuration>
 <property>
  <name>fs.defaultFS</name>
  <value>hdfs://master.hadoop:9000/</value>
 </property>
</configuration>

我想从本地物理桌面上的 HDFS 读取文件。为此，这是我的代码，我已将其保存在一个名为 hdfs_read.py 的文件中：

from hdfs import InsecureClient
client = InsecureClient('http://192.168.12.52:9000')
with client.read('/opt/hadoop/LICENSE.txt') as reader:
  features = reader.read()
  print(features)

现在当我运行它时，我收到以下超时错误：

$ python3 hdfs_read.py 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 137, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 81, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 162, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 146, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.HTTPConnection object at 0x7f2d88cef2b0>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 273, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.12.52', port=9000): Max retries exceeded with url: /webhdfs/v1/home/edhuser/testdata.txt?user.name=embs&offset=0&op=OPEN (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f2d88cef2b0>: Failed to establish a new connection: [Errno 113] No route to host',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hdfs_read_local.py", line 3, in <module>
    with client.read('/home/edhuser/testdata.txt') as reader:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 678, in read
    buffersize=buffer_size,
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 118, in api_handler
    raise err
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 107, in api_handler
    **self.kwargs
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 207, in _request
    **kwargs
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.12.52', port=9000): Max retries exceeded with url: /webhdfs/v1/home/edhuser/testdata.txt?user.name=embs&offset=0&op=OPEN (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f2d88cef2b0>: Failed to establish a new connection: [Errno 113] No route to host',))
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 137, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 81, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 162, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 146, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.HTTPConnection object at 0x7f2d88cef2b0>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 273, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.12.52', port=9000): Max retries exceeded with url: /webhdfs/v1/home/edhuser/testdata.txt?user.name=embs&offset=0&op=OPEN (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f2d88cef2b0>: Failed to establish a new connection: [Errno 113] No route to host',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hdfs_read.py", line 3, in <module>
    with client.read('/home/edhuser/testdata.txt') as reader:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 678, in read
    buffersize=buffer_size,
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 118, in api_handler
    raise err
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 107, in api_handler
    **self.kwargs
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 207, in _request
    **kwargs
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.12.52', port=9000): Max retries exceeded with url: /webhdfs/v1/home/edhuser/testdata.txt?user.name=embs&offset=0&op=OPEN (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f2d88cef2b0>: Failed to establish a new connection: [Errno 113] No route to host',))

如何解决此连接问题？我使用了错误的端口吗？我虽然在core-site.xml 中指定了namenode 正在使用的端口，但我在上面显示了该端口的9000。无论如何，我已经出于各种目的尝试了hadoop安装文档中提到的所有默认端口50070、8020、8048，但我仍然得到同样的错误。我应该使用client = InsecureClient('hdfs://192.168.12.52:9000')，或者client = InsecureClient('file:///192.168.12.52:9000')，或者类似的东西，而不是client = InsecureClient('http://192.168.12.52:9000')？我曾在不同的时间在别处看到过这些。

顺便说一下，我可以通过web访问HDFS，如下截图所示：

另外，即使它连接成功，我想我可能没有提供正确的文件路径 (/opt/hadoop/README.txt)。我给出了这个文件路径，因为这是我在hadoop安装目录中搜索文件和目录列表时看到的，即/opt/hadoop：

$ ls /opt/hadoop/
bin                 lib          read_from_hdfs.py  write_to_hdfs_2.py
connect_to_hdfs.py  libexec      README.txt         write_to_hdfs3.py
etc                 LICENSE.txt  sbin               write_to_hdfs.py
hdfs_read_write.py  logs         share
include             NOTICE.txt   test_storage

但我知道 HDFS 是独立的，也许我之前通过 hdfs dfs -get /test_storage/ ./ 将我的 HDFS 的内容复制到本地机器中，这就是它显示这些文件的原因。但是当我在namenode的路径中搜索文件时，它会返回一些难以辨认的文件：

$ls /opt/volume/namenode/current/
edits_0000000000000000001-0000000000000000002
edits_0000000000000000003-0000000000000000010
edits_0000000000000000011-0000000000000000012
edits_0000000000000000013-0000000000000000015
edits_0000000000000000016-0000000000000000023
edits_0000000000000000024-0000000000000000025
edits_0000000000000000026-0000000000000000032
edits_0000000000000000033-0000000000000000033
edits_0000000000000000034-0000000000000000035
edits_0000000000000000036-0000000000000000037
edits_0000000000000000038-0000000000000000039
edits_0000000000000000040-0000000000000000041
edits_0000000000000000042-0000000000000000043
edits_0000000000000000044-0000000000000000045
edits_0000000000000000046-0000000000000000047
edits_0000000000000000048-0000000000000000049
edits_0000000000000000050-0000000000000000051
edits_0000000000000000052-0000000000000000053
edits_0000000000000000054-0000000000000000055
edits_0000000000000000056-0000000000000000057
edits_0000000000000000058-0000000000000000059
edits_0000000000000000060-0000000000000000061
edits_0000000000000000062-0000000000000000063
edits_0000000000000000064-0000000000000000065
edits_0000000000000000066-0000000000000000067
edits_0000000000000000068-0000000000000000070
edits_0000000000000000071-0000000000000000072
edits_0000000000000000073-0000000000000000074
edits_0000000000000000075-0000000000000000076
edits_0000000000000000077-0000000000000000078
edits_inprogress_0000000000000000079
fsimage_0000000000000000076
fsimage_0000000000000000076.md5
fsimage_0000000000000000078
fsimage_0000000000000000078.md5
seen_txid
VERSION

那么，如果我指定了错误读取的文件路径，那么正确的文件路径是什么？

编辑：将端口更改为 50070（即client = InsecureClient('http://192.168.12.52:50070')）后，出现以下错误：

$ python3 hdfs_read_local.py 
Traceback (most recent call last):
  File "hdfs_read.py", line 3, in <module>
    with client.read('/opt/hadoop/LICENSE.txt') as reader:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 678, in read
    buffersize=buffer_size,
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 112, in api_handler
    raise err
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 107, in api_handler
    **self.kwargs
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 210, in _request
    _on_error(response)
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 50, in _on_error
    raise HdfsError(message, exception=exception)
hdfs.util.HdfsError: File /opt/hadoop/LICENSE.txt not found.

EDIT2：从/opt/hadoop/LICENSE.txt修改文件路径后到/test_storage/LICENSE.txt，这似乎是正确的HDFS路径，运行python脚本，我得到以下错误：

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 137, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 81, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 162, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 146, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.HTTPConnection object at 0x7f2e87867400>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 273, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='pr2.embs', port=50075): Max retries exceeded with url: /webhdfs/v1/test_storage/LICENSE.txt?op=OPEN&user.name=embs&namenoderpcaddress=192.168.12.52:9000&offset=0 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f2e87867400>: Failed to establish a new connection: [Errno 113] No route to host',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hdfs_read_local.py", line 3, in <module>
    with client.read('/test_storage/LICENSE.txt') as reader:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 678, in read
    buffersize=buffer_size,
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 118, in api_handler
    raise err
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 107, in api_handler
    **self.kwargs
  File "/home/embs/.local/lib/python3.6/site-packages/hdfs/client.py", line 207, in _request
    **kwargs
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 195, in resolve_redirects
    **adapter_kwargs
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='pr2.embs', port=50075): Max retries exceeded with url: /webhdfs/v1/test_storage/LICENSE.txt?op=OPEN&user.name=embs&namenoderpcaddress=192.168.12.52:9000&offset=0 (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f2e87867400>: Failed to establish a new connection: [Errno 113] No route to host',))

【问题讨论】：

9000 不是 HTTP 端口...你需要hdfs://
但是，错误中提到了WebHDFS，您需要单独启用它
@cricket_007 当我将http 更改为hdfs 时出现此错误：InvalidSchema: No connection adapters were found for 'hdfs://192.168.12.52:9000/webhdfs/v1/opt/hadoop/LICENSE.txt'
我没有意识到这个库正在使用 WebHDFS，所以是的，它需要是 HTTP。但在端口 50700
好的，那么这样做，我收到一个新错误，我已在问题详细信息中更新了该错误。接下来我可以尝试什么？是不是我在 python 代码中给出了错误的文件路径，就像我之前在问题细节中所怀疑的那样？

标签： python hadoop hdfs

【解决方案1】：

正如link 此处所述，此 python 库正在使用 webhdfs。如果要测试主机和文件路径是否正确，可以使用以下命令curl -i 'http://192.168.12.52:50070/webhdfs/v1/<PATH>?op=LISTSTATUS'。这将列出 hdfs 中的目录。如果你做对了，你可以在 python 中使用相同的“配置”。

from hdfs import InsecureClient
client = InsecureClient('http://192.168.12.52:50070')
with client.read('<hdfs_path>') as reader:
    features = reader.read()
    print(features)

【讨论】：

【解决方案2】：

http://192.168.12.52:9000

9000 是一个 RPC 端口。 50070 是默认的 HTTP WebHDFS 端口。

如果 WebHDFS 被禁用，或者 datanode 未公开端口 50075（datanode http 地址），您可能会收到No route to host，因为它已关闭，或者您更改了该属性

client.read('/opt/hadoop/LICENSE.txt')

您正在伪分布式模式下运行 HDFS，但您正在尝试读取本地文件。 /opt 默认情况下在 HDFS 中不存在，并且您只运行了本地 ls... 您应该使用 hadoop fs -ls /opt 查看您尝试打开的路径中确实存在哪些文件

但是当我在namenode的路径中搜索文件时，它返回了一些难以辨认的文件：

您的文件未存储在名称节点中...它们的元数据是

您的文件存储在 datanode 数据目录中，但作为块，而不是人类可读的内容

您可以运行此命令来获取所有块及其位置的列表

hdfs fsck /path/to/file.txt -files -blocks

【讨论】：

当我执行hdfs dfs -ls /test_storage/ 时，我在问题详细信息中的 HDFS Web 屏幕截图中准确地看到了这两个文件，没有别的。除了test_storage，我在 HDFS 中没有任何其他文件夹。就像我之前说的，我知道我输入的文件路径可能是错误的。但是我不知道在这个 python 代码中给出的正确 HDFS 文件路径是什么，以便可以从其他计算机读取 HDFS 文件，这是我需要帮助的地方。
“正确的路径”是 HDFS 中的任何东西......你所拥有的唯一原因是因为给定的路径不存在
好的，所以我将文件路径修改为/test_storage/LICENSE.txt，现在它给出了连接错误。在python脚本中，我将端口输入为50070，但在错误中它在某处显示端口9000，不知道为什么。我在问题详细信息中显示了 EDIT 2 中的确切错误。
是的，没错！我不知道他们从哪里来。我没有在任何 xml 配置文件中编写这些内容。端口 50075。在谷歌上搜索，似乎是 datanode 的端口，pr2.embs 是服务器的名称 192.168.12.52。但同样，我不知道 hadoop 的名字是从哪里来的，因为我没有在它的任何 xml 配置文件中输入它，我仔细检查了。
听起来您的 DNS 或 DHCP 服务器有一个缓存条目，或者您的 /etc/hosts 文件有冲突的条目

【解决方案3】：

网络配置可能有问题。暂时试试这个经过调整的代码：

from hdfs import InsecureClient
client = InsecureClient('http://0.0.0.0:50070')
with client.read('/test-storage/LICENSE.txt') as reader:
    features = reader.read()
    print(features)

了解IP地址0.0.0.0

【讨论】：

@Kristada673 如果上述答案对您有用，您能否发表评论？谢谢。

【解决方案4】：

您好，我遇到了类似的问题。看起来端口是正确的。我的情况是，我能够获取目录列表，但无法写入任何数据。问题出在我的 vpn 中，它阻塞了一些端口，并且读写使用不同的端口。

【讨论】：