如何使用访问密钥在 databrikcs 中挂载 adls Gen2 帐户答案

【问题标题】：How to mount adls Gen2 account in databrikcs using access keys如何使用访问密钥在 databrikcs 中挂载 adls Gen2 帐户
【发布时间】：2022-08-14 02:41:15
【问题描述】：

我想在 azure databricks 中挂载 adls gen 2 存储帐户。但我使用的是 azure 帐户，我无权创建服务主体。所以我正在尝试使用访问密钥挂载容器，但我一直在获取错误。

spark.conf.set(\"fs.azure.account.key.azadfdatalakegen2.dfs.core.windows.net\",dbutils.secrets.get(scope=\"azdatabricks-adlsgen2SA\", key=\"Azdatrbricks-adlsgen2-accesskeys\"))
    dbutils.fs.mount(
      source = \"abfss://raw@azadfdatalakegen2.dfs.core.windows.net/\",
      mount_point = \"/mnt/raw_adlsmnt\")

我不断收到以下错误消息

> --------------------------------------------------------------------------- ExecutionError                            Traceback (most recent call
> last) <command-555436758533424> in <module>
>       1 spark.conf.set(\"fs.azure.account.key.azadfdatalakegen2.dfs.core.windows.net\",dbutils.secrets.get(scope=\"azdatabricks-adlsgen2SA\",
> key=\"Azdatrbricks-adlsgen2-accesskeys\"))
> ----> 2 dbutils.fs.mount(
>       3   source = \"abfss://raw@azadfdatalakegen2.dfs.core.windows.net/\",
>       4   mount_point = \"/mnt/raw_adlsmnt\")
> 
> /databricks/python_shell/dbruntime/dbutils.py in
> f_with_exception_handling(*args, **kwargs)
>     387                     exc.__context__ = None
>     388                     exc.__cause__ = None
> --> 389                     raise exc
>     390 
>     391             return f_with_exception_handling
> 
> ExecutionError: An error occurred while calling o548.mount. :
> java.lang.NullPointerException: authEndpoint  at
> shaded.databricks.v20180920_b33d810.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
>   at
> shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenUsingClientCreds(AzureADAuthenticator.java:84)
>   at
> com.databricks.backend.daemon.dbutils.DBUtilsCore.verifyAzureOAuth(DBUtilsCore.scala:803)
>   at
> com.databricks.backend.daemon.dbutils.DBUtilsCore.verifyAzureFileSystem(DBUtilsCore.scala:814)
>   at
> com.databricks.backend.daemon.dbutils.DBUtilsCore.createOrUpdateMount(DBUtilsCore.scala:734)
>   at
> com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(DBUtilsCore.scala:776)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)     at
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)  at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)    at
> py4j.Gateway.invoke(Gateway.java:295)     at
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
>   at py4j.commands.CallCommand.execute(CallCommand.java:79)   at
> py4j.GatewayConnection.run(GatewayConnection.java:251)    at
> java.lang.Thread.run(Thread.java:748)

有什么方法可以使用访问密钥挂载 adls gen 2 容器？

标签： azure-databricks azure-data-lake-gen2

【解决方案1】：

如果您想使用 Azure databricks 挂载存储帐户。请遵循以下语法：

dbutils.fs.mount(
    source = "wasbs://pool@vamblob.blob.core.windows.net/",
    mount_point = "/mnt/io234",
    extra_configs = {"fs.azure.account.key.vamblob.blob.core.windows.net":dbutils.secrets.get(scope = "demo_secret", key = "demo123")})

输出：

替代方法

首先在活动目录中创建应用注册，您将获得 client.id,tenent id 。

Client Secret 在 AAD 中创建

请按照以下语法创建挂载存储：

configs = {"fs.azure.account.auth.type": "OAuth",
       "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
       "fs.azure.account.oauth2.client.id": "xxxxxxxxx", 
       "fs.azure.account.oauth2.client.secret": "xxxxxxxxx", 
       "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/xxxxxxxxx/oauth2/v2.0/token", 
       "fs.azure.createRemoteFileSystemDuringInitialization": "true"}

dbutils.fs.mount(
source = "abfss://<container_name>@<storage_account_name>.dfs.core.windows.net/<folder_name>", 
mount_point = "/mnt/<folder_name>",
extra_configs = configs)

有关更多信息，请参阅 Ron L'Esteve 的 article

【讨论】：