使用复制和选择将 Parquet 数据加载到表中时出现数据类型错误

Question

我正在尝试将镶木地板数据从 AWS S3 阶段移动到 Snowflake 中的表中，并且不断收到数据类型错误。具体来说，无论我如何调整列，此错误都会不断弹出

无法识别数值''

挑战在于，我在 90 多的列中移动，当我收到错误时，Snowflake 并没有告诉我哪一列或哪一行坏了，所以很难找到和修复。我也不能使用 validate 函数来解决问题，因为 VALIDATE 和 VALIDATE_PIPE_LOAD 不支持 COPY with transform。当我在 select 语句中隔离每个单独的列时，所有输出都很好 - 所以我什至无法找到问题单元格本身！

有没有办法让 Snowflake 忽略错误并将其移入或将列全部重铸为其他东西而不识别大海捞针单元中的针？我尝试过的一些没有解决办法的事情包括：

• 完全删除我的复制语句中的 cast :: 并尝试将所有内容作为 VARIANT 移动 - 这给了我 Failed to cast variant value "" to FIXED 错误
• 将所有内容更改为 VARCHAR
• 将文件格式更改为 csv
• 在 TRY_TO_NUMBER/DECIMAL() 中包装数字和小数
• 我也尝试了这里列出的所有建议，但无济于事 - "Numeric value '' is not recognized" - what column?

我的代码如下：

# create the file format for parquet files

CREATE FILE FORMAT MYPARQUETFORMAT
  TYPE = PARQUET
  COMPRESSION = snappy;

# create my table 

create or replace table mytable (
 ADM VARCHAR,
ADMDATER DATE default null,
admit_dateR DATE default null,
ADMTYPE NUMBER(38,0) default null,
BEGDATER VARCHAR,
BILL DECIMAL default null,
CLMCNT1 VARCHAR,
CLMCNT2 VARCHAR,
CLMCNT3 VARCHAR,
DAYS NUMBER(38,0) default null,
DISPSTAT VARCHAR,
DRG VARCHAR,
DX1 VARCHAR,
DX2 VARCHAR,
DX3 VARCHAR,
DX4 VARCHAR,
DX5 VARCHAR,
DX6 VARCHAR,
DX7 VARCHAR,
DX8 VARCHAR,
DX9 VARCHAR,
DX10 VARCHAR,
DX11 VARCHAR,
DX12 VARCHAR,
DX13 VARCHAR,
DX14 VARCHAR,
DX15 VARCHAR,
DX16 VARCHAR,
DX17 VARCHAR,
DX18 VARCHAR,
DX19 VARCHAR,
DX20 VARCHAR,
DX21 VARCHAR,
DX22 VARCHAR,
DX23 VARCHAR,
DX24 VARCHAR,
DX25 VARCHAR,
INSTTYPE VARCHAR,
Intake_BENCAT VARCHAR,
MDC NUMBER(38,0) default null,
PAID DECIMAL,
POA2 VARCHAR,
POA3 VARCHAR,
POA4 VARCHAR,
POA5 VARCHAR,
POA6 VARCHAR,
POA7 VARCHAR,
POA8 VARCHAR,
POA9 VARCHAR,
POA10 VARCHAR,
POA11 VARCHAR,
POA12 VARCHAR,
POA13 VARCHAR,
POA14 VARCHAR,
POA15 VARCHAR,
POA16 VARCHAR,
POA17 VARCHAR,
POA18 VARCHAR,
POA19 VARCHAR,
POA20 VARCHAR,
POA21 VARCHAR,
POA22 VARCHAR,
POA23 VARCHAR,
POA24 VARCHAR,
POA25 VARCHAR,
RandomID INT,
RWP NUMBER(38,0) default null,
TOTDAYS NUMBER(38,0) default null,
PROC2 VARCHAR,
PROC3 VARCHAR,
PROC4 VARCHAR,
PROC5 VARCHAR,
PROC6 VARCHAR,
PROC7 VARCHAR,
PROC8 VARCHAR,
PROC9 VARCHAR,
PROC10 VARCHAR,
PROC11 VARCHAR,
PROC12 VARCHAR,
PROC13 VARCHAR,
PROC14 VARCHAR,
PROC15 VARCHAR,
PROC16 VARCHAR,
PROC17 VARCHAR,
PROC18 VARCHAR,
PROC19 VARCHAR,
PROC20 VARCHAR,
PROC21 VARCHAR,
PROC22 VARCHAR,
PROC23 VARCHAR,
PROC24 VARCHAR,
PROC25 VARCHAR
);

# Copy data using copy and select statements

COPY INTO TEDI
FROM(SELECT
$1:ADM::VARCHAR,
$1:ADMDATER::DATE,
$1:admit_dateR::DATE,
$1:ADMTYPE::NUMBER(38,0),
$1:BEGDATER::VARCHAR,
$1:BILL::DECIMAL,
$1:CLMCNT1::VARCHAR,
$1:CLMCNT2::VARCHAR,
$1:CLMCNT3::VARCHAR,
$1:DAYS::NUMBER(38,0),
$1:DISPSTAT::VARCHAR,
$1:DRG::VARCHAR,
$1:DX1::VARCHAR,
$1:DX2::VARCHAR,
$1:DX3::VARCHAR,
$1:DX4::VARCHAR,
$1:DX5::VARCHAR,
$1:DX6::VARCHAR,
$1:DX7::VARCHAR,
$1:DX8::VARCHAR,
$1:DX9::VARCHAR,
$1:DX10::VARCHAR,
$1:DX11::VARCHAR,
$1:DX12::VARCHAR,
$1:DX13::VARCHAR,
$1:DX14::VARCHAR,
$1:DX15::VARCHAR,
$1:DX16::VARCHAR,
$1:DX17::VARCHAR,
$1:DX18::VARCHAR,
$1:DX19::VARCHAR,
$1:DX20::VARCHAR,
$1:DX21::VARCHAR,
$1:DX22::VARCHAR,
$1:DX23::VARCHAR,
$1:DX24::VARCHAR,
$1:DX25::VARCHAR,
$1:INSTTYPE::VARCHAR,
$1:Intake_BENCAT::VARCHAR,
$1:MDC::NUMBER(38,0),
$1:PAID::DECIMAL,
$1:POA10::VARCHAR,
$1:POA11::VARCHAR,
$1:POA12::VARCHAR,
$1:POA13::VARCHAR,
$1:POA14::VARCHAR,
$1:POA15::VARCHAR,
$1:POA16::VARCHAR,
$1:POA17::VARCHAR,
$1:POA18::VARCHAR,
$1:POA19::VARCHAR,
$1:POA2::VARCHAR,
$1:POA20::VARCHAR,
$1:POA21::VARCHAR,
$1:POA22::VARCHAR,
$1:POA23::VARCHAR,
$1:POA24::VARCHAR,
$1:POA25::VARCHAR,
$1:POA3::DATE,
$1:POA4::DATE,
$1:POA5::DATE,
$1:POA6::DATE,
$1:POA7::DATE,
$1:POA8::DATE,
$1:POA9::DATE,
$1:RandomID::INT,
$1:RWP::NUMBER(38,0),
$1:TOTDAYS::NUMBER(38,0),
$1:PROC2::VARCHAR,
$1:PROC3::VARCHAR,
$1:PROC4::VARCHAR,
$1:PROC5::VARCHAR,
$1:PROC6::VARCHAR,
$1:PROC7::VARCHAR,
$1:PROC8::VARCHAR,
$1:PROC9::VARCHAR,
$1:PROC10::VARCHAR,
$1:PROC11::VARCHAR,
$1:PROC12::VARCHAR,
$1:PROC13::VARCHAR,
$1:PROC14::VARCHAR,
$1:PROC15::VARCHAR,
$1:PROC16::VARCHAR,
$1:PROC17::VARCHAR,
$1:PROC18::VARCHAR,
$1:PROC19::VARCHAR,
$1:PROC20::VARCHAR,
$1:PROC21::VARCHAR,
$1:PROC22::VARCHAR,
$1:PROC23::VARCHAR,
$1:PROC24::VARCHAR,
$1:PROC25::VARCHAR
FROM @s3_stage/tedi.parquet
(file_format => MYPARQUETFORMAT));

Answer 1

试试下面的语法。我先将 parquet 数据转换为 varchar，然后再应用 try_to_number。

select metadata$filename as file_name
      ,$1:date_column::VARCHAR as date_column --format 20210203 
      ,$1:address1::VARCHAR as address1 --alpha numberic
      ,TRY_TO_NUMBER($1:address1::VARCHAR) as address2 --output null for alpha numeric rows. Produce result only for numeric rows. It should convert '' to NULL.
      ,TO_NUMBER($1:date_column::VARCHAR) as add_date2 
from @public.stage_name/directory_path/file_name.parquet (file_format => public.parque_format) t;