并发：Java Map

Question

将 2000 万个实体推送到 java 地图对象中的最佳方法是什么？

1. 如果没有多线程，则需要大约 40 秒。
2. 使用 ForkJoinPool 大约需要 25 秒，我创建了 2 个任务，每个任务都推送 1000 万个实体

我相信这两个任务都在 2 个不同的内核中运行。 问题：当我创建 1 个推送 1000 万数据的任务时，大约需要 9 秒，那么当运行 2 个任务，每个任务推送 1000 万数据时，为什么需要大约 26 秒？我是不是做错了什么？

在不到 10 秒的时间内插入 20 M 数据是否有不同的解决方案？

Answer 1

虽然我没有尝试过多个线程，但我确实尝试了 Java 11 提供的 10 种中的所有 7 种适当的 Map 类型。

我的结果都比您报告的 25 到 40 秒快得多。对于 7 个地图类中的 任何，我对 < String , UUID > 的 20,000,000 个条目的结果更像是 3-9 秒。

我正在使用 Java 13：

Model Name: Mac mini
Model Identifier:   Macmini8,1
Processor Name: Intel Core i5
Processor Speed:    3 GHz
Number of Processors:   1
Total Number of Cores:  6
L2 Cache (per Core):    256 KB
L3 Cache:   9 MB
Memory: 32 GB

准备中。

瞬间大小：20000000

uuid 大小：20000000

运行测试。

java.util.HashMap 取：PT3.645250368S

java.util.WeakHashMap 取：PT3.199812894S

java.util.TreeMap 占用：PT8.97788412S

java.util.concurrent.ConcurrentSkipListMap 占用：PT7.347253106S

java.util.concurrent.ConcurrentHashMap 取：PT4.494560252S

java.util.LinkedHashMap 取：PT2.78054883S

java.util.IdentityHashMap 取：PT5.608737472S

我的代码：

System.out.println( "Preparing." );

int limit = 20_000_000; // 20_000_000
Set < String > instantsSet = new TreeSet <>();  // Use `Set` to forbid duplicates.
List < UUID > uuids = new ArrayList <>( limit );

while ( instantsSet.size() < limit )
{
    instantsSet.add( Instant.now().toString() );
}
List < String > instants = new ArrayList <>( instantsSet );
for ( int i = 0 ; i < limit ; i++ )
{
    uuids.add( UUID.randomUUID() );
}
System.out.println( "size of instants: " + instants.size() );
System.out.println( "size of uuids: " + uuids.size() );

System.out.println( "Running test." );
// Using 7 of the 10 `Map` implementations bundled with Java 11.
// Omitting `EnumMap`, as it requires enums for the key.
// Omitting `Map.of` because it is for literals.
// Omitting `HashTable` because it is outmoded, replaced by `ConcurrentHashMap`.
List < Map < String, UUID > > maps = List.of(
        new HashMap <>( limit ) ,
        new WeakHashMap <>( limit ) ,
        new TreeMap <>() ,
        new ConcurrentSkipListMap <>() ,
        new ConcurrentHashMap <>( limit ) ,
        new LinkedHashMap <>( limit ) ,
        new IdentityHashMap <>( limit )
);
for ( Map < String, UUID > map : maps )
{
    long start = System.nanoTime();
    for ( int i = 0 ; i < instants.size() ; i++ )
    {
        map.put( instants.get( i ) , uuids.get( i ) );
    }
    long stop = System.nanoTime();
    Duration d = Duration.of( stop - start , ChronoUnit.NANOS );
    System.out.println( map.getClass().getName() + " took: " + d );

    // Free up memory.
    map = null;
    System.gc(); // Request garbage collector do its thing. No guarantee!
    try
    {
        Thread.sleep( TimeUnit.SECONDS.toMillis( 4 ) );  // Wait for garbage collector to hopefully finish. No guarantee!
    }
    catch ( InterruptedException e )
    {
        e.printStackTrace();
    }
}
System.out.println("Done running test.");

这是我写的比较各种Map 实现的表格。

Answer 2

加法可能需要一个 CPU 周期，因此如果您的 CPU 以 3GHz 运行，那就是 0.3 纳秒。执行 20M 次，即 6000000 纳秒或 6 毫秒。因此，您的测量受启动线程、线程切换、JIT 编译等开销的影响比受您正在执行的操作的影响更大 尝试测量。

垃圾收集也可能发挥作用，因为它可能会减慢您的速度。

我建议您使用专门的库进行微基准测试，例如 jmh。

感谢assylias's 帮助我写下回复的帖子

Answer 3

如果没有看到您的代码，这些不良性能结果的最可能原因是垃圾收集活动。为了演示它，我编写了以下程序：

import java.lang.management.ManagementFactory;
import java.util.*;
import java.util.concurrent.*;

public class TestMap {
  // we assume NB_ENTITIES is divisible by NB_TASKS
  static final int NB_ENTITIES = 20_000_000, NB_TASKS = 2;
  static Map<String, String> map = new ConcurrentHashMap<>();

  public static void main(String[] args) {
    try {
      System.out.printf("running with nb entities = %,d, nb tasks = %,d, VM args = %s%n", NB_ENTITIES, NB_TASKS, ManagementFactory.getRuntimeMXBean().getInputArguments());
      ExecutorService executor = Executors.newFixedThreadPool(NB_TASKS);
      int entitiesPerTask = NB_ENTITIES / NB_TASKS;
      List<Future<?>> futures = new ArrayList<>(NB_TASKS);
      long startTime = System.nanoTime();
      for (int i=0; i<NB_TASKS; i++) {
        MyTask task = new MyTask(i * entitiesPerTask, (i + 1) * entitiesPerTask - 1);
        futures.add(executor.submit(task));
      }
      for (Future<?> f: futures) {
        f.get();
      }
      long elapsed = System.nanoTime() - startTime;
      executor.shutdownNow();
      System.gc();
      Runtime rt = Runtime.getRuntime();
      long usedMemory = rt.maxMemory() - rt.freeMemory();
      System.out.printf("processing completed in %,d ms, usedMemory after GC = %,d bytes%n", elapsed/1_000_000L, usedMemory);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

  static class MyTask implements Runnable {
    private final int startIdx, endIdx;

    public MyTask(final int startIdx, final int endIdx) {
      this.startIdx = startIdx;
      this.endIdx = endIdx;
    }

    @Override
    public void run() {
      long startTime = System.nanoTime();
      for (int i=startIdx; i<=endIdx; i++) {
        map.put("sambit:rout:" + i, "C:\\Images\\Provision_Images");
      }
      long elapsed = System.nanoTime() - startTime;
      System.out.printf("task[%,d - %,d], completed in %,d ms%n", startIdx, endIdx, elapsed/1_000_000L);
    }
  }
}

在处理结束时，此代码通过执行System.gc() 紧跟Runtime.maxMemory() - Runtime.freeMemory() 来计算已用内存的近似值。这表明具有 2000 万个条目的映射大约需要不到 2.2 GB，这是相当可观的。对于 -Xmx 和 -Xms JVM 参数的各种值，我已经使用 1 个和 2 个线程运行它，这里是结果输出（为了清楚起见：2560m = 2.5g）：

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms2560m, -Xmx2560m]
task[0 - 19,999,999], completed in 11,781 ms
processing completed in 11,782 ms, usedMemory after GC = 2,379,068,760 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms2560m, -Xmx2560m]
task[0 - 9,999,999], completed in 8,269 ms
task[10,000,000 - 19,999,999], completed in 12,385 ms
processing completed in 12,386 ms, usedMemory after GC = 2,379,069,480 bytes

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms3g, -Xmx3g]
task[0 - 19,999,999], completed in 12,525 ms
processing completed in 12,527 ms, usedMemory after GC = 2,398,339,944 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms3g, -Xmx3g]
task[0 - 9,999,999], completed in 12,220 ms
task[10,000,000 - 19,999,999], completed in 12,264 ms
processing completed in 12,265 ms, usedMemory after GC = 2,382,777,776 bytes

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms4g, -Xmx4g]
task[0 - 19,999,999], completed in 7,363 ms
processing completed in 7,364 ms, usedMemory after GC = 2,402,467,040 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms4g, -Xmx4g]
task[0 - 9,999,999], completed in 5,466 ms
task[10,000,000 - 19,999,999], completed in 5,511 ms
processing completed in 5,512 ms, usedMemory after GC = 2,381,821,576 bytes

running with nb entities = 20,000,000, nb tasks = 1, VM args = [-Xms8g, -Xmx8g]
task[0 - 19,999,999], completed in 7,778 ms
processing completed in 7,779 ms, usedMemory after GC = 2,438,159,312 bytes

running with nb entities = 20,000,000, nb tasks = 2, VM args = [-Xms8g, -Xmx8g]
task[0 - 9,999,999], completed in 5,739 ms
task[10,000,000 - 19,999,999], completed in 5,784 ms
processing completed in 5,785 ms, usedMemory after GC = 2,396,478,680 bytes

这些结果可以总结在下表中：

--------------------------------
heap      | exec time (ms) for: 
size (gb) | 1 thread | 2 threads
--------------------------------
2.5       |    11782 |     12386
3.0       |    12527 |     12265
4.0       |     7364 |      5512
8.0       |     7779 |      5785
--------------------------------

我还观察到，对于 2.5g 和 3g 堆大小，由于 GC 活动，CPU 活动很高，在整个处理时间内达到 100% 的峰值，而对于 4g 和 8g，仅观察到最后由于System.gc() 调用。

总结：

1. 如果您的堆大小不合适，垃圾回收将扼杀您希望获得的任何性能提升。您应该将其设置得足够大以避免长时间 GC 暂停的副作用。

2. 您还必须注意，使用并发集合（例如 ConcurrentHashMap）会产生显着的性能开销。为了说明这一点，我稍微修改了代码，以便每个任务都使用自己的HashMap，然后最后将所有映射（使用Map.putAll()）聚合到第一个任务的映射中。处理时间降至 3200 毫秒左右