【发布时间】：2018-10-31 18:13:12
【问题描述】：
是否可以使用 AWS Glue 作业在 Redshift 中加载多个表？
这些是我遵循的步骤。
- 从 S3 爬取 json，数据已转换为数据目录表。
- 我创建了一个作业，它将在 redshift 中上传数据目录表，但它只限制我为每个作业上传一个表。在作业属性中（添加作业时），我选择的此作业运行选项是：AWS Glue 生成的建议脚本。
我不熟悉 python，而且我是 AWS Glue 的新手。但我有几个表需要上传。
这是一个示例脚本：
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
## @params: [TempDir, JOB_NAME]
args = getResolvedOptions(sys.argv, ['TempDir','JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
## @type: DataSource
## @args: [database = "sampledb", table_name = "abs", transformation_ctx = "datasource0"]
## @return: datasource0
## @inputs: []
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "sampledb", table_name = "abs", transformation_ctx = "datasource0")
## @type: ApplyMapping
## @args: [mapping = [("value", "int", "value", "int"), ("sex", "string", "sex", "string"), ("age", "string", "age", "string"), ("highest year of school completed", "string", "highest year of school completed", "string"), ("state", "string", "state", "string"), ("region type", "string", "region type", "string"), ("lga 2011", "string", "lga 2011", "string"), ("frequency", "string", "frequency", "string"), ("time", "string", "time", "string")], transformation_ctx = "applymapping1"]
## @return: applymapping1
## @inputs: [frame = datasource0]
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("value", "int", "value", "int"), ("sex", "string", "sex", "string"), ("age", "string", "age", "string"), ("highest year of school completed", "string", "highest year of school completed", "string"), ("state", "string", "state", "string"), ("region type", "string", "region type", "string"), ("lga 2011", "string", "lga 2011", "string"), ("frequency", "string", "frequency", "string"), ("time", "string", "time", "string")], transformation_ctx = "applymapping1")
## @type: ResolveChoice
## @args: [choice = "make_cols", transformation_ctx = "resolvechoice2"]
## @return: resolvechoice2
## @inputs: [frame = applymapping1]
resolvechoice2 = ResolveChoice.apply(frame = applymapping1, choice = "make_cols", transformation_ctx = "resolvechoice2")
## @type: DropNullFields
## @args: [transformation_ctx = "dropnullfields3"]
## @return: dropnullfields3
## @inputs: [frame = resolvechoice2]
dropnullfields3 = DropNullFields.apply(frame = resolvechoice2, transformation_ctx = "dropnullfields3")
## @type: DataSink
## @args: [catalog_connection = "redshift", connection_options = {"dbtable": "abs", "database": "dbmla"}, redshift_tmp_dir = TempDir, transformation_ctx = "datasink4"]
## @return: datasink4
## @inputs: [frame = dropnullfields3]
datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = dropnullfields3, catalog_connection = "redshift", connection_options = {"dbtable": "abs", "database": "dbmla"}, redshift_tmp_dir = args["TempDir"], transformation_ctx = "datasink4")
job.commit()
aws 胶水数据库：sampledb
aws 胶水中的表名：abs
红移数据库：dbmla
请提供有关如何上传它们的示例。谢谢！
【问题讨论】：
-
您是否尝试编辑生成的代码，根据表的数量添加更多数据源？并尝试做一份工作？您可以在运行作业之前修改生成的 python 脚本。我还没有尝试过，但希望它有效。
-
非常感谢尤瓦。你真的很有帮助。我也会尝试这种方式，并会让你知道输出。
-
但是，我正在考虑不在脚本中输入特定的表名。我的计划也是调出每个数据目录表。爬虫将以每小时的速度运行，因此传入的表可能包含新的数据集或只是旧表的更新。进行此设置后，将很难在脚本中输入已用胶水抓取的每个表名，因为该作业将不包含新表。我在想是否有办法让作业加载一组表，只需重写现有的，然后上传新的数据目录表。
-
我认为胶水生成的代码不可能开箱即用。您可能需要编写自己的自定义代码，使用 AWS Glue API 获取表列表，并循环遍历列表、执行转换等。我没有尝试过，但您可以尝试一下。请参阅此处了解更多信息：docs.aws.amazon.com/glue/latest/webapi/API_GetTables.html.
-
dbbest.com/blog/aws-glue-etl-service这里的解决方法和我上面说的一样，我刚看到，所以分享给大家参考