AWS DMS SQL Server 到 s3 parquet - 更改数据类型转换规则和“不支持 Parquet 类型：INT32 (UINT_8)”

Question

我们使用 AWS DMS 将 SQL Server 数据库作为 parquet 文件转储到 S3 中。想法是在镶木地板上运行一些带有火花的分析。完全加载完成后，将无法读取镶木地板，因为它们在架构中有 UINT 字段。 Spark 拒绝使用Parquet type not supported: INT32 (UINT_8) 阅读它们。我们使用转换规则来覆盖UINT 列的数据类型。但看起来它们没有被 DMS 引擎拾取。为什么？

有许多规则，如“将单位转换为 int”，见下文（注意 UINT1 是 1 字节无符号 DMS DataTypes）：

{
  "rule-type": "transformation",
  "rule-id": "7",
  "rule-name": "uintToInt",
  "rule-action": "change-data-type",
  "rule-target": "column",
  "object-locator": {
    "schema-name": "%",
    "table-name": "%",
    "column-name": "%",
    "data-type": "uint1"
  },
  "data-type": {
    "type": "int4"
  }
}

S3 DataFormat=parquet;ParquetVersion=parquet_2_0 和 DMS 引擎版本是 3.3.2

但是仍然使用 uint 获取 parquet 模式。见下文：

id: int32
name: string
value: string
status: uint8

尝试使用 spark 阅读这样的镶木地板给了我

org.apache.spark.sql.AnalysisException: Parquet type not supported: INT32 (UINT_8);
    at org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter.typeNotSupported$1(ParquetSchemaConverter.scala:100)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetToSparkSchemaConverter.convertPrimitiveField(ParquetSchemaConverter.scala:136)

为什么没有触发 DMS 转换规则？

Answer 1

在 DMS 上将数据直接从 UINT 转换为 INT 可解决此问题。 您的映射规则应如下所示：

{
"rules": [
    ...
    {
        "rule-type": "transformation",
        "rule-id": "2",
        "rule-name": "unit1-to-int1",
        "rule-action": "change-data-type",
        "rule-target": "column",
        "object-locator": {
            "schema-name": "schema",
            "table-name": "%",
            "column-name": "%",
            "data-type": "uint1"
        },
        "data-type": {
            "type": "int1"
        }
    },
    {
        "rule-type": "transformation",
        "rule-id": "3",
        "rule-name": "unit2-to-int2",
        "rule-action": "change-data-type",
        "rule-target": "column",
        "object-locator": {
            "schema-name": "schema",
            "table-name": "%",
            "column-name": "%",
            "data-type": "uint2"
        },
        "data-type": {
            "type": "int2"
        }
    },
    {
        "rule-type": "transformation",
        "rule-id": "4",
        "rule-name": "unit4-to-int4",
        "rule-action": "change-data-type",
        "rule-target": "column",
        "object-locator": {
            "schema-name": "schema",
            "table-name": "%",
            "column-name": "%",
            "data-type": "uint4"
        },
        "data-type": {
            "type": "int4"
        }
    },
    {
        "rule-type": "transformation",
        "rule-id": "5",
        "rule-name": "unit8-to-int8",
        "rule-action": "change-data-type",
        "rule-target": "column",
        "object-locator": {
            "schema-name": "schema",
            "table-name": "%",
            "column-name": "%",
            "data-type": "uint8"
        },
        "data-type": {
            "type": "int8"
        }
    }
]}

文档：https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html#CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Transformations

Answer 2

我能够在镶木地板文件上进行转换的唯一方法是指定要转换的确切列。比如：

{
   "rules": [
   ...
   {
    "rule-type": "transformation",
    "rule-id": "2",
    "rule-name": "unit1-to-int1",
    "rule-action": "change-data-type",
    "rule-target": "column",
    "object-locator": {
        "schema-name": "acessa",
        "table-name": "<table_name>",
        "column-name": "<column_name>"
    },
    "data-type": {
        "type": "int1"
    }
   }
  ]
}

在对象定位器中使用通配符 % 作为列名不起作用

Answer 3

如果您需要选择特定列并将所有 UINTS 转换为 INTS，则无法专门命名要转换的列。如果您进行通配符数据类型转换，它将包括覆盖您的列选择并包括所有列。