如何将迭代器（或类似的东西）存储到硬盘上？

Question

我想在硬盘上存储一个巨大的std::list<my_big_struct>（但在本例中很少）。

此外，我还想将许多std::list<std::list<my_big_struct>::const_iterator> 存储到硬盘上。

如何在没有深拷贝的情况下存档（会用完空间）？

下面是一个简短的例子，my_big_struct 是一个简单的int。

#include <list>
#include <fstream>
#include <boost/serialization/serialization.hpp>
#include <boost/serialization/list.hpp>
#include <boost/archive/binary_oarchive.hpp>

typedef int my_big_struct;

int main()
{
    std::list<my_big_struct> a = {3,2,7,4,9};
    std::list<std::list<my_big_struct>::const_iterator> aa;
    std::list<std::list<my_big_struct>::const_iterator> ab;

    std::list<my_big_struct>::const_iterator find = a.begin();
    aa.emplace(aa.begin(), find++);
    ab.emplace(ab.begin(), find);

    {
        std::ofstream f("test_a.bin", std::ios::binary);
        boost::archive::binary_oarchive o(f);
        o << a; // OK
    }

    // how to store aa and ab onto harddisk without hardcopy ???
}

Answer 1

我不确定设计（这听起来像是保留易失性内存关系的设计味道）。

如果您确信您的设计是合理的，我建议您使用managed_shared_memory（即 Boost Interprocess）。

然后，您可以依赖 std::list¹ 的迭代器稳定性。

演示

这演示了它是如何工作的：

Live On Coliru

#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/list.hpp>
#include <iostream>
namespace bip = boost::interprocess;

typedef int my_big_struct;

namespace shared {
    using segment = bip::managed_mapped_file;
    using manager = segment::segment_manager;

    template <typename T> using alloc = bip::allocator<T, manager>;
    template <typename T> using list  = boost::interprocess::list<T, alloc<T> >;
}

int main()
{
    using List   = shared::list<my_big_struct>;
    using ItList = shared::list<List::const_iterator>;

    std::remove("test.dat");
    {
        shared::segment segment(bip::open_or_create, "test.dat", 10u<<20); // 10 megabyte

        auto& a  = *segment.find_or_construct<List>("a")(segment.get_segment_manager());
        auto& aa = *segment.find_or_construct<ItList>("aa")(segment.get_segment_manager());
        auto& ab = *segment.find_or_construct<ItList>("ab")(segment.get_segment_manager());

        a.assign({3,2,7,4,9});
        List::const_iterator find = a.begin();
        aa.emplace(aa.begin(), find++);

        ab.emplace(ab.begin(), find);
        ab.emplace(ab.begin(), std::next(find,3));
    }

    // the file is persisted
    {
        shared::segment segment(bip::open_only, "test.dat"); // read it back

        auto& a  = *segment.find_or_construct<List>("a")(segment.get_segment_manager());
        auto& ab = *segment.find_or_construct<ItList>("ab")(segment.get_segment_manager());

        for (auto iterator : ab) {
            std::cout << "Iterator in ab points to " << *iterator << " in list a\n";

            my_big_struct const* pointer = &*iterator;
            for (auto& element : a) {
                if (&element == pointer) {
                    std::cout << "Matching element is found in a (by physical address)\n";
                }
            }
        }

    }
}

打印

Iterator in ab points to 9 in list a
Matching element is found in a (by physical address)
Iterator in ab points to 2 in list a
Matching element is found in a (by physical address)

注意事项

为一些碎片开销做好准备，因此请注意构建数据结构的方式（如果可能，请预先管理分配）。

您可以查看我的一些旧答案以获取有关 Boost Interprocess 的灵感

• boost::interprocess::managed_shared_memory
• boost::interprocess::managed_mapped_file
• scoped_allocator_adaptor 更方便

---

¹ Boost Interprocess 通过使用offset_ptr 在内部管理抽象（您不必了解此实现细节）

更多示例

如果您还想在my_big_struct 中包含复杂的成员，您还需要在那里使用段分配器：

Live On Coliru

打印

Iterator in ab points to my_big_struct { name:nine, value: 9 } in list a
Matching element is found in a (by physical address)
Iterator in ab points to my_big_struct { name:two, value: 2 } in list a
Matching element is found in a (by physical address)