当 Jenkins 服务器在 kubernetes 集群之外时，我可以使用 Jenkins kubernetes 插件吗？答案

【问题标题】：Can I use Jenkins kubernetes plugin when Jenkins server is outside of a kubernetes cluster?当 Jenkins 服务器在 kubernetes 集群之外时，我可以使用 Jenkins kubernetes 插件吗？
【发布时间】：2017-03-05 00:03:24
【问题描述】：

我目前正在尝试下面的 Jenkins kubernetes 插件，但有一些问题。

https://wiki.jenkins-ci.org/display/JENKINS/Kubernetes+Plugin

就我而言，我的 kubernetes 集群中不存在 Jenkins。这是因为我有三个用于开发、暂存和生产环境的 kubernetes 集群，而不是为每个环境提供三个 Jenkins 服务，我希望有一个统一的 Jenkins 主服务器来操作所有三个集群。

每个环境都在一个独立的 VPC 上，而 Jenkins 服务器在另一个 VPC 上，所以我设置了从 Jenkins VPC 到所有其他 VPC 的 VPC 对等，然后在 DEV 上打开了从 Jenkins 到 k8s master 的 443 端口。

但是当我点击“添加新云”->“kubernetes”上的“测试连接”时，出现错误提示

javax.servlet.ServletException: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:134)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:125)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:57)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:418)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:58)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$DescriptorImpl.doTestConnection(KubernetesCloud.java:590)
    at sun.reflect.GeneratedMethodAccessor1736.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    ... 63 more
Caused by: java.net.NoRouteToHostException: No route to host
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:579)
    at com.squareup.okhttp.internal.Platform.connectSocket(Platform.java:120)
    at com.squareup.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:141)
    at com.squareup.okhttp.internal.io.RealConnection.connect(RealConnection.java:112)
    at com.squareup.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184)
    at com.squareup.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
    at com.squareup.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
    at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
    at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
    at com.squareup.okhttp.Call.getResponse(Call.java:286)
    at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
    at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
    at com.squareup.okhttp.Call.execute(Call.java:80)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:230)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:416)
    ... 74 more

我想确定这个插件是否可以在我的案例中使用，因为在大多数示例中，Jenkins master 似乎也在 kubernetes 集群中作为 k8s pod。

【问题讨论】：

检查了我所有的配置，最后发现在我的 Ubuntu 上，Docker 网络的子网范围和我的 k8s 所在的 VPC 之一相互冲突。我修改了 Docker 网络的 ip 地址范围后，它看起来连接良好。
您尝试连接哪个版本的 kubernetes？ 1.6 ?我正在尝试相同的方法，看起来我面临着不同的问题。这是我的问题陈述stackoverflow.com/questions/46303626/…
您尝试过this 解决方法吗？我想问题只是在前端。

标签： jenkins jenkins-plugins kubernetes

【解决方案1】：

1、为k8s中的kubernetes-plugin准备一个服务账号

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

2、找到对应生成的secret：jenkins-token-xxxxxx

从 k8s 仪表板中，我们可以从 secret 中获取“ca.crt”字符串和“token”字符串。

3、配置jenkins云

在“Kubernetes server certificate key”项中，填写“ca.crt”字符串。

在“Credentials”项中，创建类型为“Secret text”或“OpenShift OAuth token”（即使我们使用纯k8s）的凭据，填写在“token”字符串中，然后使用凭证。

“测试连接”现在应该可以通过了。

【讨论】：

感谢@Roc King，这太棒了。
我正在为 podTemplates 苦苦挣扎，我的连接成功，但 Jenkins 无法启动 pod。我正在使用默认模板进行测试。 podTemplate(label: 'mypod') { node('mypod') { stage('Run shell') { sh 'echo hello world' } } }
当我尝试将令牌添加到 Jenkins 凭据作为“OpenShift OAuth 令牌”时，我没有任何字段 - prntscr.com/mg56o2 任何想法为什么？
高版本 kubernetes-plugin 使“OpenShift OAuth token”丢失token字段。 issues.jenkins-ci.org/browse/JENKINS-49717

【解决方案2】：

回答另一个问题：https://stackoverflow.com/a/48834012/502914

我们在 Kubernetes 集群之外使用 Jenkins，但能够连接到集群，将 pod 作为从属设备启动并在构建完成后销毁（使用默认命名空间）。

【讨论】：