无法在 Tomcat webapp 中关闭 Neo4j Jetty 服务器

Question

目前我正在开发一个使用 Neo4j 的网络应用程序。我们的应用需要部署在 Tomcat 环境中（客户要求）。我们决定嵌入 Neo4j，因为这样我们可以使用 Neo4j 提供的 Java API，它更容易部署并且我们得到改进的性能。但是，我们还需要访问 REST API，因为我们有一个用 Angular 编写的单页 ​​web 应用程序，目前正在使用这个接口。然而，嵌入式 Neo4j 数据库不公开 REST api。 Neo4j-server 工件包含可以使用嵌入式图形数据库引导码头服务器的代码。所以我们的 Tomcat webapp 正在启动一个 Jetty 服务器。我们可以通过部署在Tomcat中的一个端口（8080）上的webapp以及另一个端口（7474）上的Neo4j REST接口和Neo4j浏览器来访问graphDb。虽然这有点奇怪，但它工作正常，除非我们尝试停止我们的 web 应用程序（例如重新部署）。当关闭我们的 web 应用程序时，我们会从 tomcat 收到这些错误：

SEVERE: The web application [/chainmonitor] appears to have started a thread named [GC-Monitor] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [RRD4J Sync Pool [Thread-1]] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [Statistics Gatherer[primitives]] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [pool-1-thread-2] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [DateCache] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-31-selector-3] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-32-selector-0] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-33-selector-1] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-34-selector-2] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-35-selector-4] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-36-selector-5] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-37-acceptor-0-ServerConnector@1425c689{HTTP/1.1}{localhost:7474}] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-38-acceptor-1-ServerConnector@1425c689{HTTP/1.1}{localhost:7474}] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [HashSessionScavenger-0] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-54] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [org.eclipse.jetty.http.HttpFields$1] (value [org.eclipse.jetty.http.HttpFields$1@4a3d3a5e]) and a value of type [org.eclipse.jetty.http.HttpFields.DateGenerator] (value [org.eclipse.jetty.http.HttpFields$DateGenerator@24f0137b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/chainmonitor]
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor\WEB-INF\lib] could not be completely deleted. The presence of the remaining files may cause problems
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor\WEB-INF] could not be completely deleted. The presence of the remaining files may cause problems
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor] could not be completely deleted. The presence of the remaining files may cause problems
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar delete
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor] could not be completely deleted. The presence of the remaining files may cause problems

我们的 pom.xml 看起来像这样：

    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j</artifactId>
        <version>2.0.3</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server</artifactId>
        <version>2.0.3</version>
    </dependency>

这是我们启动嵌入式数据库和启动码头服务器的方式：

graphDb = (EmbeddedGraphDatabase) new GraphDatabaseFactory().
      newEmbeddedDatabaseBuilder(dataDir).loadPropertiesFromURL(Neo4jPropertiesUrl).newGraphDatabase();
bootstrapper = new WrappingNeoServerBootstrapper(graphDb);

目前 neo4j-server.properties 为空。我没有尝试过很多不同的配置，但这样做似乎不是问题。

这就是我们停止嵌入式数据库和码头服务器的方式：

    @Override
    public void contextDestroyed(ServletContextEvent event) {
​            graphDb.shutdown();
            bootstrapper.stop();
    }

这显然是由 Tomcat 的内存泄漏保护 (http://wiki.apache.org/tomcat/MemoryLeakProtection) 引起的。

结果就是我们的webapp只能通过彻底杀死Tomcat进程（kill -9）才能重新部署，这是非常不可取的。

我们尝试了很多方法：

• 未关闭嵌入式 grapbDb。没有效果。
• 未明确关闭引导程序。没有效果。 （引导程序有自己的关闭挂钩）
• 先关闭引导程序，然后关闭嵌入式数据库。
• 线程献祭者：https://github.com/Neo4j/Neo4j/issues/1070。该代码运行并牺牲了 59 个线程本地值，但没有明显效果。有趣的是，当我在关机期间循环运行一分钟时，它一直说它牺牲了 6 个值。对我来说，这表明这个过程根本不起作用，因为 Neo4j 在关闭时需要一些时间。
• 关闭后延迟。我们尝试这样做的原因是因为在 tomcat 收到关闭信号后，我们需要等待几秒钟才能真正终止进程，否则我们在下一次启动时会收到错误，即我们的图形数据已损坏（带有尚未损坏的消息）已正确关闭）。延迟 5+ 秒似乎也有效果，因为之后我们只得到 6 个严重错误而不是 16 个。这也是我认为献祭过程根本不起作用的原因。
• 我们尝试将嵌入式 Neo4j 添加到集群中，从而能够使用 REST api 以及嵌入式 Neo4j。然而，在大约四个小时试图让它发挥作用后，我们放弃了。在嵌入式 Neo4j 数据库启动期间，我们的 Web 应用程序刚刚冻结。经过一些调试，它看起来就像 Neo4j 代码中的死锁。我们尝试将所有 HA 超时配置设置为一些较小的值，但没有任何效果。

问题是由 Neo4j 码头服务器引起的。没有它，这个问题就不会出现。不过，我们需要 REST API。你知道我们如何解决这个问题吗？或者您是否有在 Tomcat 环境中使用嵌入式数据库和 REST API 的替代方法？

我们正在使用：

• Java：1.7.0_60
• Tomcat 7.0.54
• Neo4j：2.0.3（我们尝试了 2.1.1，但得到了完全相同的结果）
• 在 Windows 7 和 Linux Redhat 上观察到了相同的行为

Answer 1

您可能有泄漏的尚未关闭的事务，然后 Neo4j 在关闭时等待长达 20 秒以关闭这些事务。

如果您等待 5 秒以上，报告的哪些问题仍然存在？

Answer 2

我们有非常相似的设置和需求。这是我们针对此问题的解决方案。 JVM 运行时 shutdownHook 在 Tomcat 中不起作用，因此我们持有对包含 GraphDatabaseService 的包装器对象的引用，并将该包装器放在存储库管理器对象中。在存储库管理器对象中的方法上使用 @PreDestroy 注释，这将在存储库管理器类被销毁之前关闭存储库。

import javax.annotation.PreDestroy;

...

@PreDestroy
public void destroy() {
    log.info("Destroying Neo4jRepositoryManager...");
    shutdown(Neo4jPropertyKeys.SANDBOX_DATABASE.getUrlKey());
    shutdown(Neo4jPropertyKeys.PRODUCTION_DATABASE.getUrlKey());
    shutdown(Neo4jPropertyKeys.TEST_DATABASE.getUrlKey());
}

private void shutdown(String urlKey) {
    if(repositories.containsKey(urlKey)) {
        LabeledNodeGraphRepository graphdb = repositories.get(urlKey);
        graphdb.getGraphDatabaseService().shutdown();
        log.info(urlKey + " shutdown...");
    }
}

<bean id="repositoryManager" class="mypath.neo4j.repository.Neo4jRepositoryManager"/>

我们的环境是：

• Java 1.7.0_72
• Tomcat 7.0.56
• Neo4j 2.1.6
• 春季 4.0.5
• 球衣 2.1.4