使用 Zipkin TracingStatementInterceptor 启动 c3po 连接池的死锁答案

【问题标题】：Deadlock starting c3po connection pool with Zipkin TracingStatementInterceptor使用 Zipkin TracingStatementInterceptor 启动 c3po 连接池的死锁
【发布时间】：2023-03-31 13:57:01
【问题描述】：

尝试通过在 Spring Boot 应用程序中包含以下库来添加 Zipkin mysql 跟踪检测

compile group: 'io.zipkin.brave', name: 'brave-instrumentation-mysql', version: '5.6.3'

并将以下内容附加到我的 jdbc 连接字符串：

?statementInterceptors=brave.mysql.TracingStatementInterceptor

现有数据库连接正在使用配置如下的 c3po 连接池：

<bean id="dataSourceBean" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close" primary="true">
    <property name="driverClass" value="${db.driver}" />
    <property name="jdbcUrl" value="${db.url}" />
    <property name="user" value="${db.username}" />
    <property name="password" value="${db.password}" />
    <property name="initialPoolSize" value="${db.initial.pool.size}" />
    <property name="minPoolSize" value="${db.min.pool.size}" />
    <property name="maxPoolSize" value="${db.max.pool.size}" />
    <property name="idleConnectionTestPeriod" value="30" />
    <property name="maxIdleTimeExcessConnections" value="120" />
</bean>

但在尝试从池中获取连接时遇到问题启动服务：

2019-05-09 22:54:40.752  WARN [api,,,] 31975 --- [-AdminTaskTimer] c.m.v.a.ThreadPoolAsynchronousRunner     : com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@222a008d -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 3
    Active Threads: 3
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5cf72aaf
            on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@67c13f4f
            on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@dff1fcc
            on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@775717e4
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1290e9f3
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7cc597a9
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@9b1e682
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@46963867
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@53dc8e37
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3f4233a
Pool thread stack traces:
    Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0,5,main]
        org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
        org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
        org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
        brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
        brave.Tracer.nextContext(Tracer.java:252)
        brave.Tracer.newRootContext(Tracer.java:212)
        brave.Tracer.newTrace(Tracer.java:142)
        brave.Tracer.nextSpan(Tracer.java:436)
        brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
        brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
        com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
        com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
        com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
        com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
        com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
    Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1,5,main]
        org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
        org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
        org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
        brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
        brave.Tracer.nextContext(Tracer.java:252)
        brave.Tracer.newRootContext(Tracer.java:212)
        brave.Tracer.newTrace(Tracer.java:142)
        brave.Tracer.nextSpan(Tracer.java:436)
        brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
        brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
        com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
        com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
        com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
        com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
        com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
    Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2,5,main]
        org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:204)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:392)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
        org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
        org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:356)
        org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$570/35986993.getObject(Unknown Source)
        org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:389)
        org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
        org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
        org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
        org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
        brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
        brave.Tracer.nextContext(Tracer.java:252)
        brave.Tracer.newRootContext(Tracer.java:212)
        brave.Tracer.newTrace(Tracer.java:142)
        brave.Tracer.nextSpan(Tracer.java:436)
        brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
        brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
        com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
        com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
        com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
        com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
        com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
        com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
        com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
        com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)

【问题讨论】：

标签： mysql c3p0 zipkin

【解决方案1】：

看起来跟踪正在触发一个老式的倒锁顺序死锁，冻结试图获取新连接的线程。

三个死锁线程中的最后一个是trying to get a lock on some singleton or bean。它已经通过并可能在GenericScope 上获得了锁。

另外两个线程是trying to acquire a lock on a GenericScope，想必是第一个线程有的。

zipkin 代码意外重入 spring 导致死锁。 c3p0 有一个固定大小的线程池，当它的所有线程（这里只有 3 个，c3p0 的默认值）被永久冻结时，它会注意到，然后（在这种情况下非常正确）声明死锁并希望替换阻塞的线程恢复。

c3p0 恢复了吗？这是罕见的还是经常出现的僵局？要防止这种死锁，您可以做的事情并不多，我认为您要么必须忍受它，要么不使用检测工具。

【讨论】：

感谢@stevewaldman 的快速回复。没有 c3po 不会恢复，并且这在启动时始终如一地发生。我会考虑其他一些选项，或者不使用这个工具。
有趣，这是因为勇敢的 Sampler 是一个尚未创建的 bean，但 Tracer 已经创建了。我相信如果您将数据源标记为依赖于所有跟踪 bean，或者相反，您可以避免这种死锁
（我的宿命论可能错位了！如果@BrianDevins 解决方案有效，请随时不接受并鼓励他将他的评论纳入答案。）
将采样器显式设置为连接池的依赖项是一种享受，现在我的应用程序与 Zipkin mysql 工具一起工作。感谢您的所有帮助。 @BrianDevins 请将此添加为答案，我会将其更新为接受。
我希望@BrianDevins 将他的评论转化为答案，并且希望您将绿色标记转给他！