无法使用 Azure SDK for Java 通过网络代理访问 Azure 数据湖内容答案

【问题标题】：Unable to access the azure data lake contents through network proxy using Azure SDK for Java无法使用 Azure SDK for Java 通过网络代理访问 Azure 数据湖内容
【发布时间】：2020-11-12 02:49:52
【问题描述】：

我正在尝试通过网络代理使用 azure sdk for java API 列出 azure data Lake Gen 2 容器中的文件，但每次都会出现连接超时错误。遵循 azure 网站本身的示例代码。

示例代码链接：：https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-java

代码

public void connectToAzureDataLake() {
    String endpoint = "endPoint_URL";

    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
            .clientId("clientId")
            .clientSecret("clientSecret")
            .tenantId("tenantId")
            .build();

    int proxyPort = 1111;
    HttpClient httpClient = new NettyAsyncHttpClientBuilder()
            .proxy(new ProxyOptions(ProxyOptions.Type.HTTP,
                    new InetSocketAddress("proxyURL", proxyPort))
                    .setCredentials("proxyUsername", "proxyPassword")).build();


    DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
    DataLakeServiceClient dataLakeServiceClient = builder.credential(clientSecretCredential)
                                                    .endpoint(endpoint).httpClient(httpClient).buildClient();


    DataLakeFileSystemClient dataLakeFileSystemClient = new DataLakeFileSystemClientBuilder()
            .endpoint(endpoint + "/file_system_name")
            .credential(clientSecretCredential)
            .buildClient();
    listFilesInDirectory(dataLakeFileSystemClient);
}

private void listFilesInDirectory(DataLakeFileSystemClient fileSystemClient) {
    ListPathsOptions options = new ListPathsOptions();
    options.setPath("folder_to_list_contents");
    PagedIterable<PathItem> pagedIterable =
            fileSystemClient.listPaths(options, null);
    java.util.Iterator<PathItem> iterator = pagedIterable.iterator();
    PathItem item = iterator.next();

    while (item != null) {
        System.out.println(item.getName());
        if (!iterator.hasNext()) {
            break;
        }
        item = iterator.next();
    }
}

甚至尝试通过 system.properties 和 -D 选项提供代理详细信息，但没有成功。

-D 选项给出：

-Dhttp.proxyHost
-Dhttp.proxyPort
-Dhttp.proxyUser
-Dhttp.proxyPassword
-Dhttps.proxyHost
-Dhttps.proxyPort
-Dhttps.proxyUser
-Dhttps.proxyPassword

任何解决问题的指针都将受到高度赞赏。提前致谢。

【问题讨论】：

标签： azure proxy azure-sdk azure-data-lake-gen2

【解决方案1】：

[否则我会将此作为评论，因为我目前没有任何可以方便测试的代理，但由于字符限制，作为答案发布]。

看来您还需要将代理选项设置为ClientSecretCredentialBuilder

public void connectToAzureDataLake() {
    String endpoint = "endPoint_URL";
    int proxyPort = 1111;
    ProxyOptions proxyOptions = new new ProxyOptions(ProxyOptions.Type.HTTP,
                    new InetSocketAddress("proxyURL", proxyPort))
                    .setCredentials("proxyUsername", "proxyPassword");

    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
            .clientId("clientId")
            .clientSecret("clientSecret")
            .tenantId("tenantId")
            .proxyOptions(proxyOptions)
            .build();

    HttpClient httpClient = new NettyAsyncHttpClientBuilder()
            .proxy(proxyOptions).build();


    DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
    DataLakeServiceClient dataLakeServiceClient = builder.credential(clientSecretCredential)
                                                    .endpoint(endpoint).httpClient(httpClient).buildClient();


    DataLakeFileSystemClient dataLakeFileSystemClient = new DataLakeFileSystemClientBuilder()
            .endpoint(endpoint + "/file_system_name")
            .credential(clientSecretCredential)
            .buildClient();
    listFilesInDirectory(dataLakeFileSystemClient);
}

【讨论】：

现在将测试建议的更改。谢谢
将 httpPipeline 对象传递给 ClientSecretCredential，因为 proxyOptions 已被弃用，之后我能够获取 Microsoft 访问令牌，但在尝试访问数据湖端点 URL 时仍然出现连接超时错误。我认为可能需要将 httpPipeline 对象传递给 DataLakeFileSystemClient。
谢谢克里希格。尝试为 DataLakeFileSystemClient 传递 httpClient，现在一切正常。所以我学到的教训是为每个客户端实例传递 httpClient。

【解决方案2】：

解决方案

public void connectToAzureDataLake() {
    String endpoint = "endPoint_URL";
    int proxyPort = 1111;
    ProxyOptions proxyOptions = new new ProxyOptions(ProxyOptions.Type.HTTP,
                    new InetSocketAddress("proxyURL", proxyPort))
                    .setCredentials("proxyUsername", "proxyPassword");
    HttpClient httpClient = new NettyAsyncHttpClientBuilder()
            .proxy(proxyOptions).build();

    ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
            .clientId("clientId")
            .clientSecret("clientSecret")
            .tenantId("tenantId")
            .httpClient(httpClient)
            .build();


    DataLakeServiceClientBuilder builder = new DataLakeServiceClientBuilder();
    DataLakeServiceClient dataLakeServiceClient = builder.credential(clientSecretCredential)
                                                    .endpoint(endpoint).httpClient(httpClient).buildClient();


    DataLakeFileSystemClient dataLakeFileSystemClient = new DataLakeFileSystemClientBuilder()
            .endpoint(endpoint + "/file_system_name")
            .credential(clientSecretCredential)
            .httpClient(httpClient)
            .buildClient();
    listFilesInDirectory(dataLakeFileSystemClient);
}

【讨论】：