Spark：显示日志消息

Question

我正在使用带有 spark 依赖项的 scala 应用程序。 这就是我所拥有的

log4j.properties

# Here we have defined root logger
log4j.rootLogger=WARN,ERROR,R

#Direct log messages to file
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=./logging.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p  %c{1}:%L- %m%n

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

# Silence akka remoting
#log4j.logger.Remoting=WARN

# Ignore messages below warning level from Jetty, because it's a bit verbose
#log4j.logger.org.eclipse.jetty=WARN

主类

org.apache.log4j.PropertyConfigurator.configure("./log4j.properties")

val sparkSession = SparkSession.builder.master("local").appName("spark session example").getOrCreate
val spark = sparkSession.sqlContext

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

val log = Logger.getLogger(this.getClass.getName)
log.error("This is error");
log.warn("This is warn");
println("log ok")

pom.xml

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
      <modelVersion>4.0.0</modelVersion>

      <groupId>com.ayd</groupId>
      <artifactId>data2</artifactId>
      <version>0.0.1-SNAPSHOT</version>
      <packaging>jar</packaging>

      <name>data2</name>
      <url>http://maven.apache.org</url>

      <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      </properties>

      <dependencies>
       <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-catalyst_2.11</artifactId>
                <version>2.0.0</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.11</artifactId>
                <version>2.0.0</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_2.11</artifactId>
                <version>2.0.0</version>

            </dependency>

            <dependency>
                <groupId>org.postgresql</groupId>
                <artifactId>postgresql</artifactId>
                <version>42.1.3</version>
            </dependency>





            <!-- https://mvnrepository.com/artifact/log4j/log4j -->
            <dependency>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
                <version>1.2.17</version>
            </dependency>
        <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>3.8.1</version>
          <scope>test</scope>
        </dependency>
      </dependencies>
    </project>

我只想得到我显示给我的消息，而不是火花日志。 当我这样做时，我在日志文件中发现这个仍然是 spark 的额外日志消息

 2017-10-18 11:58:38 WARN   NativeCodeLoader:62- Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

        2017-10-18 11:58:41 ERROR  Run$:23- This is error
        2017-10-18 11:58:41 WARN   Run$:24- This is warn

我怎样才能摆脱日志文件中的两个第一行。

提前致谢

Answer 1

您将 spark 类的日志记录级别设置为 WARN，因此您会看到此警告。

如果您想过滤掉所有 Spark 警告（不推荐），您可以在 log4j 属性文件中将级别设置为 ERROR：

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

另一个警告来自 Hadoop 代码（包 org.apache.hadoop） - 因此要禁用该警告（以及所有其他 Hadoop 警告），您也可以将该包的级别指定为 ERROR：

log4j.logger.org.apache.hadoop=ERROR