【问题标题】:kubernetes ingress with gRPC and HTTP使用 gRPC 和 HTTP 的 kubernetes 入口
【发布时间】:2021-10-05 02:13:13
【问题描述】:

我有一个应用程序部署到 kubernetes (AKS),我在其中混合了 gRPC 和 http 服务。我最初将新 gRPC 服务的路由添加到以前仅提供 http 服务的现有入口。这不起作用并深入研究,我读到我们需要添加 nginx.ingress.kubernetes.io/backend-protocol: GRPC 注释,并且它适用于所有路由,因此我们需要两个单独的入口。我目前正在尝试使用消息nodename nor servname provided, or not known 连接到 gRPC 服务时遇到异常io.grpc.internal.ManagedChannelImpl$NameResolverListener 错误。我猜想虽然当 Ingress 中的多个路径与请求匹配时,最长的匹配路径优先,这不适用于两个入口。所以我需要使用不同的主机,或者更改/* 路径,使其不匹配/results?或者我的配置中还有其他需要更改的地方吗?

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /graphql
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /graphql/*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
      - path: /*
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
            port:
              number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Release.Name }}-{{ .Chart.Name }}-grpc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - hosts:
    - {{ .Values.ingress.hosts.host }}
    secretName: {{ .Values.ingress.tls.secretName }}
  rules:
  - host: {{ .Values.ingress.hosts.host }}
    http:
      paths:
      - path: /results
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}-{{ .Values.services.externalResults.host }}
            port:
              number: 9000

【问题讨论】:

    标签: nginx kubernetes grpc kubernetes-ingress nginx-ingress


    【解决方案1】:

    这最终通过创建指向我们的 k8s 集群的第二个主机名来解决。我把 grpc 服务的路由改成了 ImplementationSpecific 的根路径和 pathType。

          - path: /
            pathType: ImplementationSpecific
    

    两个主机名都需要包含在两个入口的 tls 部分中。更改路由但未更新每个入口的 tls 部分中的主机后,我遇到了 SSL 异常。

    Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
      | => cat io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:721)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    stderr: 
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1771)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:776)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:511)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1079)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:970)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1443)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    stderr: 
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    stderr: 
        at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        ... 1 more
    Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching grpc.aks.dev.app.cycleautomation.com found.
    stderr: 
        at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:462)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:428)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:261)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at io.grpc.netty.shaded.io.netty.handler.ssl.OpenSslTlsv13X509ExtendedTrustManager.checkServerTrusted(OpenSslTlsv13X509ExtendedTrustManager.java:223)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:261)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:700)
        at io.grpc.netty.shaded.io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:595)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1202)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1324)
        at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1367)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1380)
        ... 21 more
        Suppressed: javax.net.ssl.SSLHandshakeException: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
            at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1287)
            at io.grpc.netty.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1248)
            ... 25 more
    

    最终的 yaml 看起来像这样:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: {{ .Release.Name }}-{{ .Chart.Name }}-ingress
      annotations:
        kubernetes.io/ingress.class: nginx
        cert-manager.io/cluster-issuer: letsencrypt
    spec:
      tls:
      - hosts:
        - {{ .Values.ingress.hosts.host }}
        - {{ .Values.ingress.grpc.host }}
        secretName: {{ .Values.ingress.tls.secretName }}
      rules:
      - host: {{ .Values.ingress.hosts.host }}
        http:
          paths:
          - path: /graphql
            pathType: Prefix
            backend:
              service:
                name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
                port:
                  number: 80
          - path: /graphql/*
            pathType: Prefix
            backend:
              service:
                name: {{ .Release.Name }}-{{ .Values.services.graphqlServer.host }}
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
                port:
                  number: 80
          - path: /*
            pathType: Prefix
            backend:
              service:
                name: {{ .Release.Name }}-{{ .Values.services.webUIServer.host }}
                port:
                  number: 80
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: {{ .Release.Name }}-{{ .Chart.Name }}-grpc
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/backend-protocol: GRPC
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        cert-manager.io/cluster-issuer: letsencrypt
    spec:
      tls:
      - hosts:
        - {{ .Values.ingress.hosts.host }}
        - {{ .Values.ingress.grpc.host }}
        secretName: {{ .Values.ingress.tls.secretName }}
      rules:
      - host: {{ .Values.ingress.hosts.host }}
        http:
          paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: {{ .Release.Name }}-{{ .Values.services.externalResults.host }}
                port:
                  number: 9000
    

    然后我能够通过启用 tls 的端口 443 连接到 grpc 服务,并且只使用主机名,而我的连接中没有路径。

    【讨论】:

      猜你喜欢
      • 2020-11-18
      • 2017-11-29
      • 2020-04-16
      • 1970-01-01
      • 1970-01-01
      • 2019-09-14
      • 1970-01-01
      • 2019-07-04
      • 1970-01-01
      相关资源
      最近更新 更多