使用 smart_open 从 Amazon S3 流式传输数据导致 TypeError答案

【问题标题】：Streaming data from Amazon S3 using smart_open causing TypeError使用 smart_open 从 Amazon S3 流式传输数据导致 TypeError
【发布时间】：2018-02-16 10:13:19
【问题描述】：

我正在尝试将位于 Amazon S3 中的大型文本文件中的数据流式传输到我的 AWS Lambda，我正在使用 smart_open 来完成此操作，这是我的测试代码 -

import smart_open

def stream_data():
    my_bucket = 'monkey-business-dev'
    my_key = 'incoming_monkey_data/banana/banana'
    uri = 's3://{}/{}'.format(my_bucket, my_key)
    total_lines = 0
    total_records = 0
    for line in smart_open.smart_open(uri):
        total_records += 1


if __name__ == '__main__':
    stream_data()

我正在使用 Python3x，但我遇到了这个异常 -

/usr/local/lib/python3.6/site-packages/odo/backends/pandas.py:94: FutureWarning: pandas.tslib is deprecated and will be removed in a future version.
You can access NaTType as type(pandas.NaT)
  @convert.register((pd.Timestamp, pd.Timedelta), (pd.tslib.NaTType, type(None)))
Traceback (most recent call last):
  File "/Users/xxxx/PycharmProjects/monkey_lambda/datastream_from_s3.py", line 16, in <module>
    stream_data()
  File "/Users/xxxx/PycharmProjects/monkey_lambda/datastream_from_s3.py", line 11, in stream_data
    for line in smart_open.smart_open(uri):
  File "/usr/local/lib/python3.6/site-packages/smart_open/smart_open_lib.py", line 163, in smart_open
    bucket = s3_connection.get_bucket(parsed_uri.bucket_id)
  File "/usr/local/lib/python3.6/site-packages/boto/s3/connection.py", line 509, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/usr/local/lib/python3.6/site-packages/boto/s3/connection.py", line 528, in head_bucket
    response = self.make_request('HEAD', bucket_name, headers=headers)
  File "/usr/local/lib/python3.6/site-packages/boto/s3/connection.py", line 671, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python3.6/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python3.6/site-packages/boto/connection.py", line 913, in _mexe
    self.is_secure)
  File "/usr/local/lib/python3.6/site-packages/boto/connection.py", line 705, in get_http_connection
    return self.new_http_connection(host, port, is_secure)
  File "/usr/local/lib/python3.6/site-packages/boto/connection.py", line 747, in new_http_connection
    connection = self.proxy_ssl(host, is_secure and 443 or 80)
  File "/usr/local/lib/python3.6/site-packages/boto/connection.py", line 796, in proxy_ssl
    sock.sendall("CONNECT %s HTTP/1.0\r\n" % host)
TypeError: a bytes-like object is required, not 'str'

我尝试将其转换为 utf-8，但随后出现以下错误 -

/usr/local/lib/python3.6/site-packages/odo/backends/pandas.py:94: FutureWarning: pandas.tslib is deprecated and will be removed in a future version.
You can access NaTType as type(pandas.NaT)
  @convert.register((pd.Timestamp, pd.Timedelta), (pd.tslib.NaTType, type(None)))
Traceback (most recent call last):
  File "/Users/xxxxx/PycharmProjects/monkey_lambda/datastream_from_s3.py", line 16, in <module>
    stream_data()
  File "/Users/xxxxx/PycharmProjects/monkey_lambda/datastream_from_s3.py", line 11, in stream_data
    for line in smart_open.smart_open(uri.encode()):
  File "/usr/local/lib/python3.6/site-packages/smart_open/smart_open_lib.py", line 208, in smart_open
    raise TypeError('don\'t know how to handle uri %s' % repr(uri))
TypeError: don't know how to handle uri b's3://monkey-business-dev/incoming_monkey_data/banana/banana'

Process finished with exit code 1

【问题讨论】：

标签： python python-3.x amazon-s3 aws-lambda

【解决方案1】：

我知道这有点晚了，您可能已经想通了，但是您正在做的事情有两个问题：您正在尝试使用 smart_open 读取目录。这是你应该做的：例如，如果您的文件名为“test.txt”：

my_bucket = 'monkey-business-dev'
my_key = 'incoming_monkey_data/banana/banana'
my_file = 'test.txt'
uri = 's3://{}/{}/{}'.format(my_bucket, my_key, my_file)
file = smart_open.smart_open(uri, encoding="utf-8")
content = file.read()
content_list = content.split("\n")
total_lines = len(content_list)
return total_lines

如果有帮助，请支持此答案。谢谢。

【讨论】：