如何从 snowpark scala maven 应用程序构建可执行 jar 并从命令行运行

Question

我能够为 spark 应用程序和 java 应用程序构建一个 jar，但同样不适用于 snowpark 应用程序。我想知道，如何从 snowpark scala 应用程序构建可执行 jar 并从命令行运行。我可以构建 jar 但无法执行表单命令行

下面是我的错误

Exception in thread "main" net.snowflake.client.jdbc.SnowflakeSQLException: User Error Report: 
Java Stack Trace:
java.lang.RuntimeException: java.lang.ClassNotFoundException: us.company.snowpark.etl.HashProcessor
        at function_handler_0//com.snowflake.snowpark.internal.JavaUtils$.doDeserializeAndCloseInputStream(JavaUtils.scala:351)
        at function_handler_0//com.snowflake.snowpark.internal.JavaUtils$.deserialize(JavaUtils.scala:335)
        at function_handler_0//com.snowflake.snowpark.internal.JavaUtils.deserialize(JavaUtils.scala)
        at function_handler_0//SnowUDF.<init>(InlineCode.java:12)
Caused by: java.lang.ClassNotFoundException: us.company.snowpark.etl.HashProcessor
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:398)
        at java.base/java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:745)
        at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1965)
        at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1851)
        at java.base/java.io.ObjectInputStream.readClass(ObjectInputStream.java:1814)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1639)
        at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2434)
        at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
        at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2166)
        at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1668)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:482)
        at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:440)
        at function_handler_0//com.snowflake.snowpark.internal.JavaUtils$.doDeserializeAndCloseInputStream(JavaUtils.scala:348)
        ... 3 more
 in function SNOWPARK_TEMP_FUNCTION_KTVPWBIRM0FSHTU with handler SnowUDF.compute
        at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowExceptionSub(SnowflakeUtil.java:127)
        at net.snowflake.client.jdbc.SnowflakeUtil.checkErrorAndThrowException(SnowflakeUtil.java:67)
        at net.snowflake.client.core.StmtUtil.pollForOutput(StmtUtil.java:442)
        at net.snowflake.client.core.StmtUtil.execute(StmtUtil.java:345)
        at net.snowflake.client.core.SFStatement.executeHelper(SFStatement.java:487)
        at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:198)
        at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:135)
        at net.snowflake.client.core.SFStatement.execute(SFStatement.java:781)
        at net.snowflake.client.core.SFStatement.execute(SFStatement.java:677)
        at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:238)
        at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:117)
        at com.snowflake.snowpark.internal.ServerConnection.$anonfun$runQueryGetResult$1(ServerConnection.scala:358)
        at com.snowflake.snowpark.internal.ServerConnection.withValidConnection(ServerConnection.scala:810)
        at com.snowflake.snowpark.internal.ServerConnection.runQueryGetResult(ServerConnection.scala:353)
        at com.snowflake.snowpark.internal.ServerConnection.runQuery(ServerConnection.scala:336)
        at com.snowflake.snowpark.Session.runQuery(Session.scala:781)
        at com.snowflake.snowpark.internal.UDXRegistrationHandler.createJavaUDF(UDXRegistrationHandler.scala:735)
        at com.snowflake.snowpark.internal.UDXRegistrationHandler.$anonfun$registerUDF$5(UDXRegistrationHandler.scala:117)
        at com.snowflake.snowpark.internal.UDXRegistrationHandler.retryAfterFixingClassPath(UDXRegistrationHandler.scala:54)
        at com.snowflake.snowpark.internal.UDXRegistrationHandler.$anonfun$registerUDF$4(UDXRegistrationHandler.scala:99)
        at com.snowflake.snowpark.internal.UDXRegistrationHandler.withUploadFailureCleanup(UDXRegistrationHandler.scala:169)
        at com.snowflake.snowpark.internal.UDXRegistrationHandler.registerUDF(UDXRegistrationHandler.scala:99)
        at com.snowflake.snowpark.UDFRegistration.register(UDFRegistration.scala:2368)
        at com.snowflake.snowpark.functions$.registerUdf(functions.scala:2998)
        at com.snowflake.snowpark.functions$.udf(functions.scala:3110)
        at us.company.snowpark.etl.HashProcessor.<init>(HashProcessor.scala:228)
        at us.company.snowpark.app.SnowparkAppDriver$.main(SnowparkAppDriver.scala:24)
        at us.company.snowpark.app.SnowparkAppDriver.main(SnowparkAppDriver.scala)

Answer 1

我在我的 pom 中添加了下面提到的构建并运行了“mvn clean install”。在项目的目标文件夹中生成可执行的 fat jar。关于maven jar插件的更多信息可以在https://maven.apache.org/plugins/maven-jar-plugin/找到

<build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <resources>
      <resource>
        <directory>src/main/resources</directory>
      </resource>
    </resources>
    <plugins>
      <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.2.2</version>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <args>
            <arg>-encoding</arg>
            <arg>${project.build.sourceEncoding}</arg>
          </args>
          <checkMultipleScalaVersions>false</checkMultipleScalaVersions>
        </configuration>
      </plugin>

      <plugin>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.2.1</version>
        <executions>
          <execution>
            <id>jar-with-dependencies</id>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer
                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>us.company.mainClass</mainClass>
                </transformer>
              </transformers>
              <shadedArtifactAttached>true</shadedArtifactAttached>
              <shadedClassifierName>FAT</shadedClassifierName>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.7</version>
        <configuration>
          <skipTests>true</skipTests>
        </configuration>
      </plugin>
    </plugins>
  </build>

Answer 2

只需按照 Snowpark 文档中的指南操作即可：

https://docs.snowflake.com/en/sql-reference/stored-procedures-scala.html#using-sbt-to-build-a-jar-file-with-dependencies