将 SQL-Python 代码转换为 Pyspark-SQL 的问题

Question

我需要转换为 PySpark 的 python 代码，但我是 PySpark(SQL) 的新手。我正在尝试将我的 SQL 代码转换为 PySpark-sql 但出现语法错误。有人可以在这里为我提供 PySpark 的帮助吗？以下是我在 Python(Jupyter) 代码中成功执行的 Python-SQL 代码。

SQL_query="""
(WITH VCTE_Promotions as (SELECT v.Shortname, v.Employee_ID_ALT, v.Job_Level, 
                         v.Management_Level, CAST(sysdatetime() AS date) AS PIT_Date, v.Employee_Status_Alt as Employee_Status, 
                         v.Work_Location_Region, v.Work_Location_Country_Desc, v.HML, 
                         [DM_GlobalStaff].[dbo].[V_Worker_PIT].Is_Manager
FROM           [DM_GlobalStaff].[dbo].[V_Worker_CUR] as v 
LEFT OUTER JOIN
[DM_GlobalStaff].[dbo].[V_Worker_PIT] ON v.Management_Level = [DM_GlobalStaff].[dbo].[V_Worker_PIT].Management_Level),
VCTE_Promotion_v2_Eval as (
SELECT        Employee_ID_ALT,
                             ( SELECT max([pit_date]) AS prior_data 
                               FROM [DM_GlobalStaff].[dbo].[V_Worker_PIT] AS t
                               WHERE (employee_id_alt = a.Employee_ID_ALT) AND (PIT_Date < a.PIT_Date) AND (Is_Manager <> a.Is_Manager) OR
                                      (employee_id_alt = a.Employee_ID_ALT) AND (PIT_Date < a.PIT_Date) AND (Job_Level <> a.Job_Level)) AS prev_job_change_date, Is_Manager
FROM            VCTE_Promotions AS a)
SELECT  VCTE_Promotion_v2_Eval.Employee_ID_ALT, COALESCE (v_cur.Employee_Status_ALT, N'') AS Curr_Emp_Status, 
                         COALESCE (v_cur.Employee_Type, N'') AS Curr_Employee_Type, v_cur.Hire_Date_Alt AS Curr_Hire_Date, 
                         COALESCE (VCTE_Promotion_v2_Eval.Is_Manager, N'') AS Curr_Ismanager, 
                         CASE WHEN v_m.Job_Level < v_cur.Job_Level OR
                         (VCTE_Promotion_v2_Eval.Is_Manager = 1 AND v_m.Is_Manager = 0 AND v_m.Job_Level <= v_cur.Job_Level) 
                         THEN 'Promotion' WHEN v_m.Job_Level <> v_cur.Job_Level OR
                         VCTE_Promotion_v2_Eval.Is_Manager <> v_m.Is_Manager THEN 'Other' ELSE '' END AS Promotion, v_cur.Tenure, 
                         v_cur.Review_Rating_Current
FROM            VCTE_Promotion_v2_Eval INNER JOIN
                         [DM_GlobalStaff].[dbo].[V_Worker_CUR] as v_cur ON VCTE_Promotion_v2_Eval.Employee_ID_ALT = v_cur.Employee_ID_ALT LEFT OUTER JOIN
                         [DM_GlobalStaff].[dbo].[V_Worker_PIT] as v_m ON VCTE_Promotion_v2_Eval.prev_job_change_date = v_m.PIT_Date AND 
                         VCTE_Promotion_v2_Eval.Employee_ID_ALT = v_m.employee_id_alt
) as pr """

SQL_query= spark.read.jdbc(url=jdbcUrl, table=SQL_query, properties=connectionProperties)

SQL_query.count()

错误：-

Py4JJavaError                             Traceback (most recent call last)
<command-448309067524118> in <module>()
----> 1 xpromotions = spark.read.jdbc(url=jdbcUrl, table=xpromotions, properties=connectionProperties)

/databricks/spark/python/pyspark/sql/readwriter.py in jdbc(self, url, table, column, lowerBound, upperBound, numPartitions, predicates, properties)
    533             jpredicates = utils.toJArray(gateway, gateway.jvm.java.lang.String, predicates)
    534             return self._df(self._jreader.jdbc(url, table, jpredicates, jprop))
--> 535         return self._df(self._jreader.jdbc(url, table, jprop))
    536 
    537 

/databricks/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1158         answer = self.gateway_client.send_command(command)
   1159         return_value = get_return_value(
-> 1160             answer, self.gateway_client, self.target_id, self.name)
   1161 
   1162         for temp_arg in temp_args:

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/databricks/spark/python/lib/py4j-0.10.6-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    318                 raise Py4JJavaError(
    319                     "An error occurred while calling {0}{1}{2}.\n".
--> 320                     format(target_id, ".", name), value)
    321             else:
    322                 raise Py4JError(

Py4JJavaError: An error occurred while calling o351.jdbc.
: com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near the keyword 'WITH'.
    at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
    at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1535)
    at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:467)
    at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:409)
    at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7151)
    at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2478)
    at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:219)
    at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:199)
    at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:331)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:60)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:115)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:52)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:358)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:281)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:269)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:196)
    at org.apache.spark.sql.DataFrameReader.jdbc(DataFrameReader.scala:296)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
    at py4j.Gateway.invoke(Gateway.java:295)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:226)
    at java.lang.Thread.run(Thread.java:748)

Answer 1

您在一个查询中做了很多事情，但这是行不通的。这就是你想要做的：

1. 加入表 V_Worker_CUR 和 V_Worker_PIT 并通过以下方式引用它 姓名VCTE_Promotions
2. 从上面带有一些过滤器的连接表中获取max(pit_date)，并将其称为另一个名为VCTE_Promotion_v2_Eval 的表
3. 将上面的表格VCTE_Promotion_v2_Eval 与V_Worker_CUR 和V_Worker_PIT 连接起来，并将其称为pr

这不是 spark 的工作方式。建议如下：

1. 在 spark 中加载单个表并将它们存储为数据帧，即 V_Worker_CUR 和 V_Worker_PIT
2. 现在要么将它们加载为临时表，要么直接处理数据帧，然后加入它们并执行其他任务。

这里的重点是不要编写复杂的查询来加载数据，而是加载数据，然后使用 Spark 使用 Spark 支持的任何语言 API 来操作它。