火花和安全凤凰不在纱线上工作

Question

我正在尝试使用 JDBC 通过纱线中的 spark 连接到安全凤凰，我可以在日志中看到，它连接成功：

JDBC URL：jdbc:phoenix:zookeeper_quorum:/hbase-secure:someprincial@REALM.COM:/path/to/keytab/someprincipal.keytab

18/02/27 09:30:22 INFO ConnectionQueryServicesImpl: Trying to connect to a secure cluster with keytab:/path/to/keytab/someprincipal.keytab
18/02/27 09:30:22 DEBUG UserGroupInformation: hadoop login
18/02/27 09:30:22 DEBUG UserGroupInformation: hadoop login commit
18/02/27 09:30:22 DEBUG UserGroupInformation: using kerberos user:someprincial@REALM.COM
18/02/27 09:30:22 DEBUG UserGroupInformation: Using user: "someprincial@REALM.COM" with name someprincial@REALM.COM
18/02/27 09:30:22 DEBUG UserGroupInformation: User entry: "someprincial@REALM.COM"
18/02/27 09:30:22 INFO UserGroupInformation: Login successful for user someprincial@REALM.COM using keytab file /path/to/keytab/someprincipal.keytab
18/02/27 09:30:22 INFO ConnectionQueryServicesImpl: Successfull login to secure cluster!!

但后来，当尝试调用 AbstractRpcClient 时，它给了我一个问题，它不再在 UserGroupInformation 中使用 KERBEROS 身份验证，而且它似乎正在获取操作系统用户而不是我在 JDBC 中提供的用户

18/02/27 09:30:23 DEBUG AbstractRpcClient: RPC Server Kerberos principal name for service=ClientService is hbase/hbaseprincipa@REALM.COM
18/02/27 09:30:23 DEBUG AbstractRpcClient: Use KERBEROS authentication for service ClientService, sasl=true
18/02/27 09:30:23 DEBUG AbstractRpcClient: Connecting to some.host.name/10.000.145.544:16020
18/02/27 09:30:23 DEBUG UserGroupInformation: PrivilegedAction as:someuser (auth:SIMPLE) from:org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)
18/02/27 09:30:23 DEBUG HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos principal name is hbase/hbaseprincipa@REALM.COM
18/02/27 09:30:23 DEBUG UserGroupInformation: PrivilegedActionException as:someuser (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/02/27 09:30:23 DEBUG UserGroupInformation: PrivilegedAction as:someuser (auth:SIMPLE) from:org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637)
18/02/27 09:30:23 WARN AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
18/02/27 09:30:23 ERROR AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
    at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:887)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:856)
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1199)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32741)
    at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:379)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:201)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:63)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
    at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
    at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
    at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
    at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
    at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
    ... 26 more

此问题仅在我在 yarn 中运行时发生，但当我在本地运行时，它使用相同的 UserGroupInformation 并且能够毫无问题地连接到 ClientService。

你知道为什么会这样吗？

我已经在我的类路径（执行程序）中包含了所有需要的配置，例如 hbase-site.xml、core-site.xml、hdfs-site.xml，我还设置了 JAAS 配置文件。

我在本地注意到，一开始，UGI 从我的操作系统获取一个，然后由于我尝试连接到 phoenix，phoenix (ConnectionQueryServicesImpl.java) 用我在 JDBC 中指定的 UGI 覆盖，因此，当再次尝试连接时，它使用的是正确的 UGI。

在集群中运行时，似乎不是这样，即使我成功连接到凤凰，当再次尝试使用 UGI 时，它从操作系统获取一个 - 我在同一个执行器中运行。

---

请注意，RpcClientImpl 正在使用基于操作系统用户的 CurrentUser。

在我的驱动程序中，每当我尝试获取 CurrentUser 时，它都会对主体使用 kerberos 身份验证 - 假设 kinit 已完成或在 spark 提交命令中提供了 keytab 和主体

在执行器中，当节点中存在有效令牌时，LoginUser 设置为 kerberos 身份验证，但 CurrentUser 设置为使用操作系统信息的简单身份验证

如何让执行者更改当前用户？

无论如何，我可以通过使用带有 UserGroupInformation.doAs() 方法的 LoginUser 强制执行更新来解决它

Answer 1

几周后，我终于找到了答案。关键是将spark.yarn.security.credentials.hbase.enabled设置为true。

提交spark如下：

spark-submit \
  --master yarn \
  --keytab my-keytab \
  --principal my-principal \
  --conf spark.yarn.security.credentials.hbase.enabled=true \
  # other configs

在执行器中，创建没有keytab和principal的phoenix连接：

String url = "jdbc:phoenix:2.1.8.1:2181:/hbase";
Connection conn = DriverManager.getConnection(url, properties);