Java - 具有明显空闲线程的 ExecutorService

Question

我观察到一种无法用 ExecutorService 解释的行为。我有一个应用程序在内存中加载大约 800 个人资料。考虑以下

ExecutorService es = Executors.newFixedThreadPool(8); // 8 cores machine
Runnable enricherService = new Runnable() {

    @Override
    public void run() {
            doEnrichment(conn, tmpCachePerson);
    }
};

// tmpCachePerson is a ConcurrentLinkedQueue<Person>
while (tmpCachePerson.isEmpty() == false) {
    es.execute(enricherService);
}

es.shutdown();

try {
    while (!es.awaitTermination(24L, TimeUnit.HOURS)) {
        System.out.println("Waiting for termination");
    }
} catch (InterruptedException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
}

与 1 个线程的池相比，这部分非常慢。我在代码中放置了一些 println，我可以看到每隔一段时间（约 4 秒），所有线程都会停止，在那里最多停留 16 秒，然后重新开始，几乎就像一个批处理，但在迭代之间会休眠。需要50s才能完成。然后我尝试了以下实现：

Runnable enricher2Thread = new Runnable() {

            @Override
            public void run() {
                while (tmpCachePerson.isEmpty() == false) {
                    doEnrichment(conn, tmpCachePerson);
                }
            }
        };

        Thread t = new Thread(enricher2Thread);
        t.start();
        try {
            t.join();
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

另一方面，这块非常快，只使用一个线程，在控制台中永远不会停止打印，并且在 3 秒内完成。

如果我将第一块中的固定池替换为缓存池，则在生成 800 个线程后，任务将在 3 秒内完成。如果我在固定池中放置 800 个线程，速度相同。任何人都明白为什么固定池会经常暂停，因此为什么它不比 1 个线程快。下面是我看到的 8 个线程的摘录。如果您查看 thread-1，它会在一个简单的 getter 上暂停 5s 。在日志的其他部分，整个任务大约需要 250 毫秒。

2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting executing SQL
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done executing SQL in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting adding
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting Getting Val
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done Getting Val in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting Getting Root
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done Getting Root in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting Getting Path
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done Getting Path in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-6 - Starting
2016-11-15 15:54:04.212 - pool-1-thread-8 - Starting  <-------------- All threads stop
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting executing SQL
2016-11-15 15:54:09.533 - pool-1-thread-6 - Starting executing SQL
2016-11-15 15:54:09.533 - pool-1-thread-8 - Done executing SQL in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Starting Getting Full Path
2016-11-15 15:54:09.533 - pool-1-thread-6 - Done executing SQL in 5320 ms
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting adding
2016-11-15 15:54:09.533 - pool-1-thread-6 - Starting adding
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting Getting Val
2016-11-15 15:54:09.533 - pool-1-thread-6 - Starting Getting Val
2016-11-15 15:54:09.533 - pool-1-thread-8 - Done Getting Val in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting Getting Root
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done Getting Full Path in 5320 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Starting Adding Image
2016-11-15 15:54:09.533 - pool-1-thread-8 - Done Getting Root in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done Adding Image in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting Getting Path
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done adding in 5321 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Starting setting
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done setting in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done in 5321 ms

知道如何改进此代码以及为什么会暂停吗？如果有帮助，我可以发布 doEnrichment() 的代码

编辑：这里是：

private void doEnrichment(Connection conn, ConcurrentLinkedQueue<Person> tmpCachePerson) {
        Person person = tmpCachePerson.poll();

        if (person != null) {
            ImageCollection personImageCollection;
            String query = "SELECT epi.value, i.path FROM Image i "
                    + "INNER JOIN EntityImageRelationship eir ON eir.id_image = i.id "
                    + "INNER JOIN EntityType et ON eir.id_entity_type = et.id "
                    + "INNER JOIN EntityPrimaryImage epi ON epi.type_to_entity_uid = eir.type_to_entity_uid "
                    + "WHERE et.id = ? AND eir.id_entity_id = ? ORDER BY i.id ASC";

            String tagQuery = "SELECT id, value FROM Tag t INNER JOIN EntityTagRelationship etr ON etr.id_tag = t.id WHERE etr.id_entity = ? AND etr.id_entity_type = ?";

            try (PreparedStatement stmnt = conn.prepareStatement(query);
                    PreparedStatement tagStmnt = conn.prepareStatement(tagQuery)) {
                personImageCollection = getEntityImages(conn, stmnt, person.getId(), person.getMovieEntityType());
                person.setImageCollection(personImageCollection);
                person.getImageCollection().setPrimaryImageIcon();

                Set<String> tags = getEntityTags(conn, tagStmnt, person.getId(), person.getMovieEntityType()).keySet();
                person.setTags(tags);

                personCache.add(person);
            } catch (Exception e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }

Answer 1

这可能不是原因，但这里有一个竞争条件：

while (tmpCachePerson.isEmpty() == false) {
    es.execute(enricherService);
}

绝对不能保证上下文切换会发生；即使是这样，它也可能比您预期的要慢；在工人启动之前，该循环可能会运行一百万次。在你解决这个问题之前不要再看下去了；这需要大量的旋转和内存开销。

更好的模式是将轮询放在工作人员中：

Runnable enricherService = new Runnable() {
    @Override
    public void run() {
        while (!tmpCachePerson.isEmpty()) {
            doEnrichment(conn, tmpCachePerson);
            // TODO: error handling? Should a failure in doEnrichment kill the worker?
        }
    }
};

然后启动工人

for (int i = 0; i < 8; ++i) {
    es.execute(enricherService);
}