是否建议将大型字典拆分为多个部分？

Question

我正在使用字典来保存相当多 (> 10^7) 的项目。如果将其拆分为多个单独的字典，每个字典都包含数据的一部分/分区，它能否提高查找和/或插入性能？

举个例子，假设我们有一个Dictionary<int,int>。我们可以将其替换为：

var ds = new Dictionary<int,int> [256];
// ...
void Add (int key, int value) {
    // We can assume key is an evenly distributed hash
    ds[key & 0xFF].Add (key, value);
}
// Lookup similar

当然，这需要进行基准测试，但我也对针对这种情况的一般建议感兴趣。令人惊讶的是，我在这里找不到真正相似的问题。

我知道单个字典可以容纳的项目数量是有限的。这个问题假设这个限制不是问题 - 否则，无论如何只有一个解决方案。

Answer 1

我已经考虑了更多。虽然许多数据结构在插入或查找操作中表现出对数成本，但在字典中，这些成本（摊销）假定为 O(1)。

在前一种情况下，通过手动索引（O(1) 操作）分离一部分工作可以通过减少对数参数来减少剩余工作。实际上，我们将在另一个结构之上实现字典。

当然，这也意味着当基础结构本身已经是字典时，这应该不会产生任何重大影响。有很多方法可以实现这些，但据我所知，没有一种方法可以通过减小它们的大小来渐进地受益：它们的平均案例行为（即处理重复项）在时间上是恒定的并且不会增长。

另一方面，手工工作会带来开销。所以我们预计这样一个切片字典的性能会更差。

为了彻底检查这一点，我编写了一个小测试。

Console.WriteLine ("Times in seconds per 10m merged/sliced operations");
foreach (var init in new[] { "empty", "size", "spare" }) {
    for (int n = 10 * 1000 * 1000; n <= 40 * 1000 * 1000; n += 10 * 1000 * 1000) {
        for (int repeat = 0; repeat < 3; repeat++) {
            Stopwatch wmi, wml, wsi, wsl;

            {
                GC.Collect ();
                var r = new Random (0);
                Dictionary<int, object> d;

                if (init == "empty") {
                    d = new Dictionary<int, object> ();
                }
                else if (init == "size") {
                    d = new Dictionary<int, object> (n);
                }
                else {
                    d = new Dictionary<int, object> (2 * n);
                }

                wmi = Stopwatch.StartNew ();
                for (int i = 0; i < n; i++) {
                    var key = r.Next ();
                    d[key] = null;
                }
                wmi.Stop ();

                wml = Stopwatch.StartNew ();
                var dummy = false;
                for (int i = 0; i < n; i++) {
                    dummy ^= d.ContainsKey (i);
                }
                wml.Stop ();
            }

            {
                GC.Collect ();
                var r = new Random (0);
                var ds = new Dictionary<int, object>[256];

                for (int i = 0; i < 256; i++) {
                    if (init == "empty") {
                        ds[i] = new Dictionary<int, object> ();
                    }
                    else if (init == "size") {
                        ds[i] = new Dictionary<int, object> (n / 256);
                    }
                    else {
                        ds[i] = new Dictionary<int, object> (2 * n / 256);
                    }
                }

                wsi = Stopwatch.StartNew ();
                for (int i = 0; i < n; i++) {
                    var key = r.Next ();
                    var d = unchecked(ds[key & 0xFF]);
                    d[key] = null;
                }
                wsi.Stop ();

                wsl = Stopwatch.StartNew ();
                var dummy = false;
                for (int i = 0; i < n; i++) {
                    var d = unchecked(ds[i & 0xFF]);
                    dummy ^= d.ContainsKey (i);
                }
                wsl.Stop ();
            }

            string perM (Stopwatch w) => $"{w.Elapsed.TotalSeconds / n * 10 * 1000 * 1000,5:0.00}";
            Console.WriteLine ($"init = {init,-5}, n = {n,8};"
                + $" insert = {perM (wmi)}/{perM (wsi)},"
                + $" lookup = {perM (wml)}/{perM (wsl)}");
        }
    }
    Console.WriteLine ();
}

每个测试重复 3 次。字典的初始化策略是以下之一： a) 空的（让它自己处理增长） b) init to size（容量等于项目数） c) 初始化为两倍大小

请注意，切片后的字典可能会出现分布不均的情况。

输出（数字越小越好）：

init = empty, n = 10000000; insert =  1.17/ 1.22, lookup =  0.42/ 0.53
init = empty, n = 10000000; insert =  1.13/ 1.24, lookup =  0.41/ 0.53
init = empty, n = 10000000; insert =  1.10/ 1.21, lookup =  0.41/ 0.53
init = empty, n = 20000000; insert =  1.19/ 1.29, lookup =  0.42/ 0.53
init = empty, n = 20000000; insert =  1.18/ 1.28, lookup =  0.42/ 0.54
init = empty, n = 20000000; insert =  1.18/ 1.28, lookup =  0.42/ 0.53
init = empty, n = 30000000; insert =  1.31/ 1.22, lookup =  0.34/ 0.66
init = empty, n = 30000000; insert =  1.35/ 1.23, lookup =  0.35/ 0.66
init = empty, n = 30000000; insert =  1.34/ 1.21, lookup =  0.35/ 0.66
init = empty, n = 40000000; insert =  1.26/ 1.20, lookup =  0.43/ 0.76
init = empty, n = 40000000; insert =  1.26/ 1.19, lookup =  0.43/ 0.76
init = empty, n = 40000000; insert =  1.25/ 1.21, lookup =  0.43/ 0.76

init = size , n = 10000000; insert =  0.82/ 0.89, lookup =  0.48/ 0.79
init = size , n = 10000000; insert =  0.80/ 0.90, lookup =  0.48/ 0.70
init = size , n = 10000000; insert =  0.80/ 0.88, lookup =  0.47/ 0.69
init = size , n = 20000000; insert =  0.84/ 0.91, lookup =  0.48/ 0.69
init = size , n = 20000000; insert =  0.84/ 0.88, lookup =  0.48/ 0.69
init = size , n = 20000000; insert =  0.84/ 0.85, lookup =  0.48/ 0.69
init = size , n = 30000000; insert =  0.88/ 0.90, lookup =  0.49/ 0.75
init = size , n = 30000000; insert =  0.93/ 0.96, lookup =  0.50/ 0.72
init = size , n = 30000000; insert =  0.88/ 0.90, lookup =  0.49/ 0.73
init = size , n = 40000000; insert =  0.85/ 0.90, lookup =  0.48/ 0.76
init = size , n = 40000000; insert =  0.86/ 0.98, lookup =  0.49/ 0.76
init = size , n = 40000000; insert =  0.86/ 0.94, lookup =  0.49/ 0.76

init = spare, n = 10000000; insert =  0.69/ 0.73, lookup =  0.29/ 0.49
init = spare, n = 10000000; insert =  0.70/ 0.71, lookup =  0.29/ 0.49
init = spare, n = 10000000; insert =  0.68/ 0.76, lookup =  0.28/ 0.49
init = spare, n = 20000000; insert =  0.70/ 0.78, lookup =  0.29/ 0.54
init = spare, n = 20000000; insert =  0.70/ 0.78, lookup =  0.29/ 0.53
init = spare, n = 20000000; insert =  0.70/ 0.76, lookup =  0.29/ 0.53
init = spare, n = 30000000; insert =  0.71/ 0.77, lookup =  0.29/ 0.50
init = spare, n = 30000000; insert =  0.73/ 0.78, lookup =  0.30/ 0.51
init = spare, n = 30000000; insert =  0.71/ 0.77, lookup =  0.29/ 0.51
init = spare, n = 40000000; insert =  0.72/ 0.80, lookup =  0.29/ 0.53
init = spare, n = 40000000; insert =  0.72/ 0.81, lookup =  0.29/ 0.53
init = spare, n = 40000000; insert =  0.72/ 0.81, lookup =  0.29/ 0.53

始终如一地，在切片字典中插入和查找都不是更快。我相信在大多数情况下都会如此。

但是，这种切片字典在并行操作中仍有可能的用例。字典的不同部分可以并行处理批量操作的数据的插入、查找等。

无论字典作为一个整体是否同时使用都是如此。但是，如果是，那么切片将允许仅锁定所需的部分而不是所有内容（在幼稚的实现中）。但是，其他从头开始设计用于并发操作的字典（例如 .NET 的 ConcurrentDictionary）则不受此缺点的影响。

Answer 2

将字典保持为一体将为您提供最佳的平均性能，但每次需要 Resize 其内部数组时，您都会获得巨大的成功。

// Create the dictionary
var dict = new Dictionary<int, int>(19998337); // 90 msec

// Populate the dictionary
for (int i = 0; i < 19998337; i++) dict.Add(i, i); // 850 msec

// Add one more entry that requires resize
dict.Add(-1, -1); // 850 msec

这种情况只会发生几次，您可以通过在创建字典时积极分配空间来避免这种情况。将庞大的字典拆分成多个较小的字典当然是另一种分摊成本的方法。

---

更新：我担心每个方法调用 GetHashCode 两次，一次用于选择正确的子字典，另一次在内部由选定的子字典调用，可能会影响性能。 GetHashCode 的一些实现有点贵，比如不区分大小写的字符串 (StringComparer.CurrentCultureIgnoreCase.GetHashCode)。我以为我通过实现一个特殊的EqualityComparer 找到了一个聪明的解决方案，它缓存了最后一次GetHashCode 调用的结果，但实际上性能提升很小。我的基准测试显示，对于int 等简单的键类型没有明显的改进，对于上述不区分大小写的字符串，总体加速不到 10%。同时这个优化有个缺点，就是读操作会改变内部状态，所以ReaderWriterLockSlim不能用于线程同步。

无论如何，这是我对这个想法的实现：

public class SegmentedDictionary<TKey, TValue> : IDictionary<TKey, TValue>
{
    private class CachedComparer : IEqualityComparer<TKey>
    {
        private readonly IEqualityComparer<TKey> _source;
        private int? _cachedHashCode;

        public CachedComparer(IEqualityComparer<TKey> source)
        {
            _source = source ?? EqualityComparer<TKey>.Default;
        }

        public bool Equals(TKey x, TKey y) => _source.Equals(x, y);

        public int GetHashCodeAndCache(TKey key)
        {
            int hashCode = _source.GetHashCode(key);
            _cachedHashCode = hashCode;
            return hashCode;
        }

        public int GetHashCode(TKey key)
        {
            if (_cachedHashCode.HasValue)
            {
                int hashCode = _cachedHashCode.Value;
                _cachedHashCode = null; // Use the cache only once
                return hashCode;
            }
            return _source.GetHashCode(key);
        }
    }

    private readonly CachedComparer _comparer;
    private readonly Dictionary<TKey, TValue>[] _segments;

    public SegmentedDictionary(int segmentsCount, int capacityPerSegment,
        IEqualityComparer<TKey> comparer)
    {
        _comparer = new CachedComparer(comparer);
        _segments = new Dictionary<TKey, TValue>[segmentsCount];
        for (int i = 0; i < segmentsCount; i++)
        {
            _segments[i] = new Dictionary<TKey, TValue>(
                capacityPerSegment, _comparer);
        }
    }

    private Dictionary<TKey, TValue> GetSegment(TKey key)
    {
        var hashCode = _comparer.GetHashCodeAndCache(key);
        var index = Math.Abs(hashCode) % _segments.Length;
        return _segments[index];
    }

    public int Count => _segments.Sum(d => d.Count);

    public TValue this[TKey key]
    {
        get => GetSegment(key)[key];
        set => GetSegment(key)[key] = value;
    }

    public void Add(TKey key, TValue value) => GetSegment(key).Add(key, value);

    public bool ContainsKey(TKey key) => GetSegment(key).ContainsKey(key);

    public bool TryGetValue(TKey key, out TValue value)
        => GetSegment(key).TryGetValue(key, out value);

    public bool Remove(TKey key) => GetSegment(key).Remove(key);

    public void Clear() => Array.ForEach(_segments, d => d.Clear());

    public IEnumerator<KeyValuePair<TKey, TValue>> GetEnumerator()
        => _segments.SelectMany(d => d).GetEnumerator();

    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();

    ICollection<TKey> IDictionary<TKey, TValue>.Keys
        => throw new NotImplementedException();

    ICollection<TValue> IDictionary<TKey, TValue>.Values
        => throw new NotImplementedException();

    void ICollection<KeyValuePair<TKey, TValue>>.Add(
        KeyValuePair<TKey, TValue> item)
        => throw new NotImplementedException();

    bool ICollection<KeyValuePair<TKey, TValue>>.Contains(
        KeyValuePair<TKey, TValue> item)
        => throw new NotImplementedException();

    void ICollection<KeyValuePair<TKey, TValue>>.CopyTo(
        KeyValuePair<TKey, TValue>[] array, int arrayIndex)
        => throw new NotImplementedException();

    bool ICollection<KeyValuePair<TKey, TValue>>.Remove(
        KeyValuePair<TKey, TValue> item)
        => throw new NotImplementedException();

    bool ICollection<KeyValuePair<TKey, TValue>>.IsReadOnly
        => throw new NotImplementedException();
}