苏打水：不能使用 spark ml 管道的支持答案

【问题标题】：Sparkling water: Can't make use of the support of spark ml pipelines苏打水：不能使用 spark ml 管道的支持
【发布时间】：2017-02-11 19:20:50
【问题描述】：

根据 Sparkling water 的 this 博客，您现在可以使用 Spark ML 管道组件在最新版本中构建 DL 模型。我尝试在我的 build.sbt 中添加最新版本

"org.apache.spark" % "spark-mllib_2.10" % "2.0.0" % "provided",
"ai.h2o" % "sparkling-water-core_2.10" % "1.6.5" % "provided"

但运气不好，尝试导入 org.apache.spark.ml.h2o.H2OPipeline 不起作用。火花罐中似乎不存在 spark.ml 中的 h2o 包。尽管它似乎在上面的链接以及here 中都有效。我真的很想重用我的 spark-mllib 特征转换器来使用 h2o 创建一个 DL 模型，如博客中所示。

任何帮助表示赞赏！

谢谢。

【问题讨论】：

不确定是否是问题所在，但您使用的是 Spark 2.0 和苏打水 1.6.5，您应该使用最近发布的苏打水 2.0。
我怀疑是这样的。 mvnrepository.com/artifact/ai.h2o/sparkling-water-core_2.10 直到 1.6.8 才发布。
此外，我们正在谈论来自org.apache.spark.ml 的丢失包对吗？

标签： scala apache-spark apache-spark-mllib h2o sparkling-water

【解决方案1】：

1）请不要使用spar 2与sw 1.6.5 - 它不起作用。我们为Scala 2.11释放了SW2.0 https://mvnrepository.com/artifact/ai.h2o/sparkling-water-core_2.11

2）您只在您的构建中添加SW核心，您正在寻找的课程是在Sparkling-water-ml https://mvnrepository.com/artifact/ai.h2o/sparkling-water-ml_2.11

中

【讨论】：

谢谢！如果您还可以将我指出与ML管道支持的一些文档，这将是伟大的，这是对苏打水的支持。 span>
@ void不幸的是，我们在文件方面没有太大，因为我们仍在实施这个并考虑一个实验特征。目前，我们只支持H2O的GBM和Deeaplinning作为管道的一部分（这是一些示例代码github.com/h2oai/sparkling-water/blob/master/examples/pipelines/…）。我们对贡献非常开放： - ） span>
非常感谢回复！请将我指出任何与向流水线添加更多算法支持的任何问题（JIRA）？我想跟踪轨道。 span>

【解决方案2】：

我使用以下版本运行 Maven pom.xml 的 H2O 示例，它正在工作

火花 - 1.6
苏打水 - 1.6.8
ai h2o - 3.10.0.8

这里是 maven pom.xml (请参考 GIT repo - https://github.com/seerampavan/H2oTesting/blob/master/pom.xml)

<properties>
    <spark.version>1.6.0-cdh5.7.1</spark.version>
    <scala.version>2.10.4</scala.version>
    <scala.binary.version>2.10</scala.binary.version>
    <top.dir>${project.basedir}/..</top.dir>
    <hadoop.version>2.6.0-cdh5.7.1</hadoop.version>
</properties>

<dependencies>
    <!-- Force import of Spark's servlet API for unit tests -->
    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>javax.servlet-api</artifactId>
        <version>3.0.1</version>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
        <!--<scope>provided</scope>-->
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>

        <exclusions>
            <exclusion>
                <!-- make sure wrong scala version is not pulled in -->
                <groupId>org.scala-lang</groupId>
                <artifactId>scala-library</artifactId>
            </exclusion>
            <exclusion>
                <!-- make sure wrong scala version is not pulled in -->
                <groupId>org.scala-lang</groupId>
                <artifactId>scalap</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>

    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>

    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>

        <exclusions>
            <exclusion>
                <groupId>org.jpmml</groupId>
                <artifactId>pmml-model</artifactId>
            </exclusion>
        </exclusions>

    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>

    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>

    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <type>test-jar</type>
        <classifier>tests</classifier>

    </dependency>
    <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_${scala.binary.version}</artifactId>
        <version>2.2.1</version>

    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>

    </dependency>

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>${hadoop.version}</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>javax.servlet</groupId>
                <artifactId>servlet-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>javax.servlet.jsp</groupId>
                <artifactId>jsp-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.jruby</groupId>
                <artifactId>jruby-complete</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.jboss.netty</groupId>
                <artifactId>netty</artifactId>
            </exclusion>
            <exclusion>
                <groupId>io.netty</groupId>
                <artifactId>netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-reflect</artifactId>
        <version>2.10.5</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-web</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-scala_2.10</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-persist-s3</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-persist-hdfs</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-parquet-parser</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-genmodel</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-core</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-bindings</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-avro-parser</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-app</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>h2o-algos</artifactId>
        <version>3.10.0.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>sparkling-water-repl_2.10</artifactId>
        <version>1.6.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>sparkling-water-ml_2.10</artifactId>
        <version>1.6.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>sparkling-water-examples_2.10</artifactId>
        <version>1.6.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>sparkling-water-core_2.10</artifactId>
        <version>1.6.8</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>deepwater-backend-api</artifactId>
        <version>1.0.0</version>
    </dependency>

    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.9.2</version>
    </dependency>
    <dependency>
        <groupId>org.joda</groupId>
        <artifactId>joda-convert</artifactId>
        <version>1.8.1</version>
    </dependency>
    <dependency>
        <groupId>org.javassist</groupId>
        <artifactId>javassist</artifactId>
        <version>3.22.0-CR1</version>
    </dependency>
    <dependency>
        <groupId>gov.nist.math</groupId>
        <artifactId>jama</artifactId>
        <version>1.0.3</version>
    </dependency>
    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.7</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>reflections</artifactId>
        <version>0.9.11-h2o-custom</version>
    </dependency>
    <dependency>
        <groupId>ai.h2o</groupId>
        <artifactId>google-analytics-java</artifactId>
        <version>1.1.2-H2O-CUSTOM</version>
    </dependency>
    <dependency>
        <groupId>com.github.tony19</groupId>
        <artifactId>named-regexp</artifactId>
        <version>0.2.4</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-s3</artifactId>
        <version>1.11.45</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-kms</artifactId>
        <version>1.11.45</version>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk-core</artifactId>
        <version>1.11.45</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty.aggregate</groupId>
        <artifactId>jetty-servlet</artifactId>
        <version>8.2.0.v20160908</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty.aggregate</groupId>
        <artifactId>jetty-server</artifactId>
        <version>8.2.0.v20160908</version>
    </dependency>
    <dependency>
        <groupId>org.eclipse.jetty.aggregate</groupId>
        <artifactId>jetty-plus</artifactId>
        <version>8.1.17.v20150415</version>
    </dependency>
</dependencies>

【讨论】：