如何解决java spark中的“线程“main”java.lang.Error中的异常：未解决的编译问题：SparkSession无法解析为类型”

Question

我在 Azure Windows 虚拟机中安装了 “VScode、jdk 8、python 3.8 和 databricks-connect==8.1.*”。之后，我创建了一个 databricks 集群并使用 cmd配置了 Databricks-connect。设置完所有路径变量后，我按照 Java 代码执行：

'''

import java.util.ArrayList;
import java.util.List;
import java.sql.Date;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.types.*;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.RowFactory;
import org.apache.spark.sql.Dataset;

public class test_{
    public static void main(String[] args) throws Exception {
        SparkSession spark = SparkSession.builder().appName("Temps Demo").config("spark.master", "local").getOrCreate();


        // Create a Spark DataFrame consisting of high and low temperatures
        // by airport code and date.
        StructType schema = new StructType(new StructField[] {
            new StructField("AirportCode", DataTypes.StringType, false, Metadata.empty()),
            new StructField("Date", DataTypes.DateType, false, Metadata.empty()),
            new StructField("TempHighF", DataTypes.IntegerType, false, Metadata.empty()),
            new StructField("TempLowF", DataTypes.IntegerType, false, Metadata.empty()),
        });
}}'''

================================================ ========== 在 VScode 终端内，我得到低于输出 '''

PS C:\Users\gaurav>  & 'c:\Users\gaurav\.vscode\extensions\vscjava.vscode-java-debug-0.35.0\scripts\launcher.bat' 'C:\Program Files\AdoptOpenJDK\jdk-11.0.11.9-hotspot\bin\java.exe' '-Dfile.encoding=UTF-8' '-cp' 'C:\Users\gaurav\AppData\Local\Temp\vscodesws_8933a\jdt_ws\jdt.ls-java-project\bin' 'App' 
    Exception in thread "main" java.lang.Error: Unresolved compilation problems: 
            SparkSession cannot be resolved to a type
            SparkSession cannot be resolved
            StructType cannot be resolved to a type
            StructType cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
    PS C:\Users\gaurav>  & 'c:\Users\gaurav\.vscode\extensions\vscjava.vscode-java-debug-0.35.0\scripts\launcher.bat' 'C:\Program Files\AdoptOpenJDK\jdk-11.0.11.9-hotspot\bin\java.exe' '-Dfile.encoding=UTF-8' '-cp' 'C:\Users\gaurav\AppData\Local\Temp\vscodesws_8933a\jdt_ws\jdt.ls-java-project\bin' 'test_'
    Exception in thread "main" java.lang.Error: Unresolved compilation problems: 
            SparkSession cannot be resolved to a type
            SparkSession cannot be resolved
            StructType cannot be resolved to a type
            StructType cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
            StructField cannot be resolved to a type
            StructField cannot be resolved to a type
            DataTypes cannot be resolved to a variable
            Metadata cannot be resolved
    
            at test_.main(testjava.java:22)

'''

当我在配置 databricks 连接后在 cmd 中运行“databricks-connect 测试”时。我得到以下输出... [1]：https://i.stack.imgur.com/p5WG6.png

Answer 1

org.apache.spark.sql.types.StructType 可以在 spark-catalyst 中实现，将 spark-catalyst.jar 和 spark-sql.jar 添加到 Referenced Libraries。

此外，java 扩展需要 jdk11 或最新 版本来支持。

满足以上条件，我的项目中没有显示StructField cannot be resolved to a type。