通过 spark dataframe 将文件加载到 delta 表中答案

【问题标题】：Loading files through spark dataframe into delta table通过 spark dataframe 将文件加载到 delta 表中
【发布时间】：2022-12-04 11:01:00
【问题描述】：

I have a folder in azure zen2 VAS In this folder i have files Vas_1.csv Vas_2.csv Vas_3.csv

I have to load these files using pyspark dataframe in table VAS in delta with two additional column derived in runtime load date and file_name.

Once the data is loaded in VAS table . Next day Few more files VAS_4.csv and VAS_5.csv comes into zen2 folder and now I have to load these two files in the same table VAS *Note the folder VAS in zen2 has 5 files now so for the second time load i have to skip the previously loaded file

标签： apache-spark pyspark databricks azure-data-lake

【解决方案1】：

【解决方案2】：