警告：检测到多个版本的 scala 库？答案

【问题标题】：warning:Multiple versions of scala libraries detected?警告：检测到多个版本的 scala 库？
【发布时间】：2017-04-10 09:58:58
【问题描述】：

我的项目是 maven+intellij。我在windows系统中开发。首先我使用最新的 scala libray 版本 2.12.2。为了获取类 SQLContext 等等，我必须导入 spark jar：

但后来有人告诉我，如果我想使用这个 spark jar，我必须降低我的 scala 版本，所以我删除了 libiray 并更改为 2.10....但是现在当我 mvn_clean_install 时。我得到了这个：

[WARNING]  Expected all dependencies to require Scala version: 2.11.7
[WARNING]  com.twitter:chill_2.11:0.5.0 requires scala version: 2.11.7
[WARNING]  com.typesafe.akka:akka-remote_2.11:2.3.11 requires scala version: 2.11.7
[WARNING]  com.typesafe.akka:akka-actor_2.11:2.3.11 requires scala version: 2.11.7
[WARNING]  com.typesafe.akka:akka-slf4j_2.11:2.3.11 requires scala version: 2.11.7
[WARNING]  org.apache.spark:spark-core_2.11:1.6.1 requires scala version: 2.11.7
[WARNING]  org.json4s:json4s-jackson_2.11:3.2.10 requires scala version: 2.11.7
[WARNING]  org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.7
[WARNING]  org.json4s:json4s-ast_2.11:3.2.10 requires scala version: 2.11.7
[WARNING]  org.json4s:json4s-core_2.11:3.2.10 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] E:\...\src\main\scala:-1: info: compiling
[INFO] Compiling 4 source files to E:\...\target\classes at 1491813951772
[ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:2: error: object sql is not a member of package org.apache.spark
[ERROR] import org.apache.spark.sql.SQLContext
[ERROR]                         ^
[ERROR] E:\...\qubole\mapreduce\ConvertToParquetFormat.scala:15: error: not found: type SQLContext
[ERROR]   val sqlContext = new SQLContext(sc)
[ERROR]                        ^
[ERROR] E:\...\mapreduce\ConvertToParquetFormat.scala:24: error: value toDF is not a member of org.apache.spark.rdd.RDD[....qubole.mapreduce.ConvertToParquetFormat.OmnitureHit]
[ERROR] possible cause: maybe a semicolon is missing before `value toDF'?
[ERROR]   .toDF().write.parquet ("file:///C:/Users/Desktop/456")
[ERROR]    ^
[ERROR] three errors found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:00 min
[INFO] Finished at: 2017-04-10T16:45:58+08:00
[INFO] Final Memory: 33M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.0:compile (scala-compile-first) on project packages-omniture-qubole-mapreduce: wrap: org.apache.commons.exec.ExecuteException: Process exited with an er
ror: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

E:\github\mia\packages-omniture-qubole-mapreduce>

所以我删除了 scala 10 版本，它告诉我我没有一个 scala libray。然后我添加了一个 scala 2.11。因为看起来有些 jars 仍然需要 2.11 版本Multiple versions of scala libraries detected?

但是当我 ctrl+left_cilck 单词 SQLContext 时，我可以转到页面，但它是匿名的。

那是因为我忘记删除有关旧 jar 或库的内容吗？这是我的左依赖列表：

这是我的 pom.xml：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.company.www</groupId>
    <artifactId>packages-omniture-qubole-mapreduce</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>com.company.www.platform</groupId>
        <artifactId>platform-parent-spark</artifactId>
        <version>0.1.41</version>
    </parent>

    <properties>
        <spark.mapreduce.mainclass>com.company.www.packages.omniture.qubole.mapreduce.SampleMapReduceJob</spark.mapreduce.mainclass>
    </properties>


    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.major.minor.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.company.www.commons</groupId>
            <artifactId>commons-spark</artifactId>
            <version>[1.0.17, ]</version>
        </dependency>
        <dependency>
            <groupId>com.company.www</groupId>
            <artifactId>exp-user-interaction-messages-v1</artifactId>
            <version>[1.4,]</version>
        </dependency>
        <dependency>
            <groupId>org.scalaj</groupId>
            <artifactId>scalaj-http_${scala.major.minor.version}</artifactId>
            <version>1.1.4</version>
        </dependency>
        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.3</version>
        </dependency>
        <dependency>
            <groupId>org.parboiled</groupId>
            <artifactId>parboiled-java</artifactId>
            <version>1.0.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>


    <build>
        <plugins>
            <plugin>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <finalName>packages-omniture-qubole-mapreduce</finalName>
                            <shadedArtifactAttached>false</shadedArtifactAttached>
                            <artifactSet>
                                <includes>
                                    <include>*:*</include>
                                </includes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>reference.conf</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer">
                                    <resource>log4j.properties</resource>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>${spark.mapreduce.mainclass}</mainClass>
                                </transformer>
                            </transformers>
                            <relocations>
                                <relocation>
                                    <pattern>org.eclipse.jetty</pattern>
                                    <shadedPattern>org.spark-project.jetty</shadedPattern>
                                    <includes>
                                        <include>org.eclipse.jetty.**</include>
                                    </includes>
                                </relocation>
                                <relocation>
                                    <pattern>com.google.common</pattern>
                                    <shadedPattern>org.spark-project.guava</shadedPattern>
                                    <excludes>
                                        <exclude>com/google/common/base/Absent*</exclude>
                                        <exclude>com/google/common/base/Function</exclude>
                                        <exclude>com/google/common/base/Optional*</exclude>
                                        <exclude>com/google/common/base/Present*</exclude>
                                        <exclude>com/google/common/base/Supplier</exclude>
                                    </excludes>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

【问题讨论】：

发布您的 POM 文件。
我已经添加了 pom，请帮助我..

标签： scala maven apache-spark intellij-idea

【解决方案1】：

在你的 pom 中包含以下内容

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_${scala.major.minor.version}</artifactId>
    <version>${spark.version}</version>
</dependency>

【讨论】：

这是做什么的？
SqlContext 类在 spark-core_* 中不可用，它在 spark-sql 中可用。试试看。
是的。我试过了，它成功了！但是当我运行 main 方法时，我得到一个错误...2017-04-10 19:52:40 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes在适用的情况下 2017-04-10 19:52:40 错误外壳：373 - 无法在 hadoop 二进制路径 java.io.IOException 中找到 winutils 二进制文件：无法在 Hadoop 二进制文件中找到可执行的 null\bin\winutils.exe。我该怎么办呢.....
这不会解决问题询问的警告，只是编译错误。而且警告肯定也需要修复。
本次提到的这个问题可以参考这个文档：srccodes.com/p/article/39/…

【解决方案2】：

问题很可能是那些 com.company.www. 依赖项（如果没有指定它们的确切版本，你会变得更糟）。他们对特定的 Scala 版本进行硬编码，您必须查看它们的依赖项才能找到哪个（查找 _2.10、_2.11 或 _2.12 后缀）。

假设这些是您公司的软件包，您需要为不同的 Scala 版本构建单独的工件或确定特定的 Scala 版本（例如，通过要求所有 Spark 项目的公共父 POM）。

【讨论】：