PySpark 删除所有特殊字符的所有列名中的特殊字符 - 错误无法解析给定列

Question

我正在尝试从 csv 创建的数据框中的名称中删除特殊字符。有 100 列名称很长。我已经尝试了多种方法，至少在其中一列上都返回错误？

df = spark.read.format("com.databricks.spark.csv") \
  .option("mode", "DROPMALFORMED") \
  .option("header", "true") \
  .option("inferschema", "true") \
  .option("delimiter", ",").load(getArgument('sourceCSVpath') + getArgument('sourceCSV'))

df = df.select([F.col(col).alias(re.sub("[^0-9a-zA-Z$]+","",col)) for col in df.columns])

temp_df1 = df.select([F.col(col).alias(col.replace('- ', '')) for col in df.columns])

错误 无法解析给定输入列的“Organization - No. Of Employees - Employee Figures Date”

Domestic Ultimate Employee Information Scope Code,NACE Revision 2 Description - Priority 4,NACE Revision 2 Description - Priority 5,NACE Revision 2 Description - Priority 6,Organization - No. Of Employees - Employee Figures Date,Number of Employees Scope Text,Organization Founded Date,NACE Revision 2 Description - Priority 1
9067,,,,,Headquarters Only (Employs Here),1997,Hospital activities
9067,,,,,Headquarters Only (Employs Here),1997,Hospital activities
9067,,,,,Headquarters Only (Employs Here),1997,Hospital activities
9067,,,,,Headquarters Only (Employs Here),1997,Hospital activities
9067,,,,,Headquarters Only (Employs Here),1997,Hospital activities

Answer 1

要从列名中删除点（或任何其他不需要的字符），您可以使用DataFrame.toDF：

temp_df1 = df.toDF(*[re.sub("\.", "", col) for col in df.columns])

toDF 只考虑列的顺序。不使用在 select 语句中可能导致问题的列的原始名称。