【发布时间】:2016-12-06 14:53:55
【问题描述】:
我正在尝试在 Maven 上使用 scalatest 和 spark-testing-base 进行 Spark 集成测试。 Spark 作业读取 CSV 文件、验证结果并将数据插入数据库。我正在尝试通过放入已知格式的文件并查看它们是否以及如何失败来测试验证。这个特定的测试只是确保验证通过。不幸的是,scalatest 找不到我的测试。
相关pom插件:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<!-- enable scalatest -->
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<wildcardSuites>com.cainc.data.etl.schema.proficiency</wildcardSuites>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
这是测试类:
class ProficiencySchemaITest extends FlatSpec with Matchers with SharedSparkContext with BeforeAndAfter {
private var schemaStrategy: SchemaStrategy = _
private var dataReader: DataFrameReader = _
before {
val sqlContext = new SQLContext(sc)
import sqlContext._
import sqlContext.implicits._
val dataInReader = sqlContext.read.format("com.databricks.spark.csv")
.option("header", "true")
.option("nullValue", "")
schemaStrategy = SchemaStrategyChooser("dim_state_test_proficiency")
dataReader = schemaStrategy.applySchema(dataInReader)
}
"Proficiency Validation" should "pass with the CSV file proficiency-valid.csv" in {
val dataIn = dataReader.load("src/test/resources/proficiency-valid.csv")
val valid: Try[DataFrame] = Try(schemaStrategy.validateCsv(dataIn))
valid match {
case Success(v) => ()
case Failure(e) => fail("Validation failed on what should have been a clean file: ", e)
}
}
}
当我运行mvn test 时,它找不到任何测试并输出此消息:
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ load-csv-into-db ---
[36mDiscovery starting.[0m
[36mDiscovery completed in 54 milliseconds.[0m
[36mRun starting. Expected test count is: 0[0m
[32mDiscoverySuite:[0m
[36mRun completed in 133 milliseconds.[0m
[36mTotal number of tests run: 0[0m
[36mSuites: completed 1, aborted 0[0m
[36mTests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0[0m
[33mNo tests were executed.[0m
更新
通过使用:
<suites>com.cainc.data.etl.schema.proficiency.ProficiencySchemaITest</suites>
代替:
<wildcardSuites>com.cainc.data.etl.schema.proficiency</wildcardSuites>
我可以让那个测试运行。显然,这并不理想。通配符套件可能已损坏;我要在 GitHub 上开一张票,看看会发生什么。
【问题讨论】:
-
ProficiencySchemaITest在哪里? -
src/test/scala/com/cainc/data/etl/schema/proficiency/ProficiencySchemaITest.scala。我的 pom 中有:
src/main/scala src/test/scala -
我还可以在
target/中看到.class文件已创建。 -
我有同样的问题没有使用
wildcardSuites,所以问题可能更深。
标签: scala maven apache-spark scalatest