将 Blob 连接到 Spark 失败

Question

我正在尝试将 Hortonworks 2.4 发行版中的 spark 连接到 Azure Blob 存储，但出现错误“wasb filesystem notrecognized”。我检查了很多人建议下载 azure-sdk-for-java 并创建包。

我正在尝试使用 mvn 创建包，但同时运行 mvn test 和 mvn package ；该过程在测试时被挂起。没有给出错误，它只是没有返回任何东西。我已更改 testconfiguration.xml 以反映我的 blob 帐户名称。下面是我得到的日志文件。

还有其他事情需要做才能得到罐子吗？

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Microsoft Azure Storage Client SDK 4.3.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ azure-storage ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/adminhorton/kspark/azure-storage-java-master/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ azure-storage ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ azure-storage ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ azure-storage ---
[INFO] Compiling 52 source files to /home/adminhorton/kspark/azure-storage-java-master/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.13:test (default-test) @ azure-storage ---
[INFO] Surefire report directory: /home/adminhorton/kspark/azure-storage-java-master/target/surefire-reports
T E S T S

parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running com.microsoft.azure.storage.StorageAccountTests
Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057 sec
Running com.microsoft.azure.storage.StorageUriTests
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.071 sec

com.microsoft.azure.storage.StorageAccountTests.txt
com.microsoft.azure.storage.StorageUriTests.txt

com.microsoft.azure.storage.StorageAccountTests.txt
com.microsoft.azure.storage.StorageUriTests.txt

Answer 1

根据您的描述，我不确定您是否在项目中正确配置了存储配置。所以我建议您可以按照以下步骤检查您的配置：

1.创建您的 Azure 存储帐户并将其设置为 core-site.xml，格式如下：

<property>
  <name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
  <value>YOUR ACCESS KEY</value>
</property>

2.重启HDP服务，使用Hadoop fs –ls wasb://**.blob.core.windows.net/列出容器中的文件。

我强烈建议你可以参考这个博客how-to-configure-hortonworks-hdp-to-access-azure-windows-storage和官方document。

同时，@Yuval 提供了有关如何使用 Java SDK 连接到 Azure blob 的精彩示例。