使用 boto3 和 athena 在 python 中读取 alb 日志

Question

我想在我的 django 代码中访问 alb 日志，我正在尝试使用 athena 查询来获取我正在使用 boto3 版本 1.3.1 连接到 athena 客户端的日志，以使用 python 查询到 athena

这是我在 python 3.5 版中的代码

import time
import boto3

query = """
            SELECT *
            from alb_logs
            """
DATABASE = 'alb_logs'
output='s3://bucket/'

client = boto3.client('athena')
# Execution
response = client.start_query_execution(
    QueryString=query,
    QueryExecutionContext={
        'Database': DATABASE
    },
    ResultConfiguration={
        'OutputLocation': output,
    }
)
print(response)

但我不能告诉雅典娜它给出了 UnknownServiceError 这是我得到的错误

UnknownServiceError: Unknown service: 'athena'. Valid service names are: acm, apigateway, application-autoscaling, appstream, autoscaling, batch, budgets, cloudformation, cloudfront, cloudhsm, cloudsearch, cloudsearchdomain, cloudtrail, cloudwatch, codebuild, codecommit, codedeploy, codepipeline, cognito-identity, cognito-idp, cognito-sync, config, datapipeline, devicefarm, directconnect, discovery, dms, ds, dynamodb, dynamodbstreams, ec2, ecr, ecs, efs, elasticache, elasticbeanstalk, elastictranscoder, elb, elbv2, emr, es, events, firehose, gamelift, glacier, health, iam, importexport, inspector, iot, iot-data, kinesis, kinesisanalytics, kms, lambda, lightsail, logs, machinelearning, marketplacecommerceanalytics, meteringmarketplace, opsworks, opsworkscm, pinpoint, polly, rds, redshift, rekognition, route53, route53domains, s3, sdb, servicecatalog, ses, shield, sms, snowball, sns, sqs, ssm, stepfunctions, storagegateway, sts, support, swf, waf, waf-regional, workspaces, xray

这是错误的回溯：

---------------------------------------------------------------------------
UnknownServiceError                       Traceback (most recent call last)
<ipython-input-208-8eaf965f48e5> in <module>
     16 output='s3://sr-prod-alb-logs-mumbai/'
     17 
---> 18 client = boto3.client('athena')
     19 # Execution
     20 response = client.start_query_execution(

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/boto3/__init__.py in client(*args, **kwargs)
     77     See :py:meth:`boto3.session.Session.client`.
     78     """
---> 79     return _get_default_session().client(*args, **kwargs)
     80 
     81 def resource(*args, **kwargs):

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/boto3/session.py in client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    248             aws_access_key_id=aws_access_key_id,
    249             aws_secret_access_key=aws_secret_access_key,
--> 250             aws_session_token=aws_session_token, config=config)
    251 
    252     def resource(self, service_name, region_name=None, api_version=None,

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/session.py in create_client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    826             is_secure=use_ssl, endpoint_url=endpoint_url, verify=verify,
    827             credentials=credentials, scoped_config=self.get_scoped_config(),
--> 828             client_config=config, api_version=api_version)
    829         return client
    830 

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/client.py in create_client(self, service_name, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, api_version, client_config)
     60                       api_version=None,
     61                       client_config=None):
---> 62         service_model = self._load_service_model(service_name, api_version)
     63         cls = self._create_client_class(service_name, service_model)
     64         endpoint_bridge = ClientEndpointBridge(

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/client.py in _load_service_model(self, service_name, api_version)
     90     def _load_service_model(self, service_name, api_version=None):
     91         json_model = self._loader.load_service_model(service_name, 'service-2',
---> 92                                                      api_version=api_version)
     93         service_model = ServiceModel(json_model, service_name=service_name)
     94         self._register_retries(service_model)

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/loaders.py in _wrapper(self, *args, **kwargs)
    130         if key in self._cache:
    131             return self._cache[key]
--> 132         data = func(self, *args, **kwargs)
    133         self._cache[key] = data
    134         return data

~/.pyenv/versions/squadrun_python3.5/lib/python3.5/site-packages/botocore/loaders.py in load_service_model(self, service_name, type_name, api_version)
    376             raise UnknownServiceError(
    377                 service_name=service_name,
--> 378                 known_service_names=', '.join(sorted(known_services)))
    379         if api_version is None:
    380             api_version = self.determine_latest_version(

跟进问题： 如果我使用 gzip 文件读取 alb-logs 并使用文档中提供的正则表达式 (https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html) 将其制作成所需的格式，如果亚马逊将来在 alb 日志中添加更多列或者我的正则表达式将此代码是否仍然有效休息？ 还有其他更好的方法来读取 python 中的 alb-logs 吗？

Answer 1

我会说你的boto3 已经过时了。当前版本为1.26.16（2022 年 11 月）。

您使用的版本是 2016 年 7 月的版本，Athena 是 2016 年 11 月才发布的。