通过内省从 ctype 结构中获取元素？

Question

我没有找到任何可以帮助我解决此类问题的方法：我正在尝试获取一个属性的偏移量，该属性是嵌套结构的一部分，例如：

data_types.py

class FirstStructure (ctypes.Structure):
    _fields_ = [('Junk', ctypes.c_bool),
                ('ThisOneIWantToGet', ctypes.c_int8)
                ]


class SecondStructure (ctypes.Structure):
    _fields_ = [('Junk', ctypes.c_double),
                ('Example', FirstStructure)
                ]

要提到的重要一点是，我只知道父结构的名称SecondStructure，我完全不知道那里可以有多少嵌套结构。

我在这里要做的是从SecondStructure 的开头获取ThisOneIWantToGet 属性的偏移量。

我知道有 ctypes.adressof 方法适用于 ctypes 对象。有没有简单的方法来获取嵌套参数的对象，所以我可以这样做：

do_something.py

import data_types as dt
par_struct_obj = getattr(dt, 'SecondStructure')
par_obj = getattr(par_struct_obj , 'ThisOneIWantToGet')
print ctypes.addressof(parameter) - ctypes.addressof(parent_structure)

Answer 1

我将首先指出 ctypes 官方文档：[Python 3.5]: ctypes - A foreign function library for Python。

我定义了一个更复杂的结构树（2 个嵌套级别）。

data_types.py：

import ctypes


PRAGMA_PACK = 0


class Struct2(ctypes.Structure):
    if PRAGMA_PACK:
        _pack_ = PRAGMA_PACK
    _fields_ = [
        ("c_0", ctypes.c_char),  # 1B
        ("s_0", ctypes.c_short),  # 2B
        ("wanted", ctypes.c_int), # 4B
    ]


class Struct1(ctypes.Structure):
    if PRAGMA_PACK:
        _pack_ = PRAGMA_PACK
    _fields_ = [
        ("d_0", ctypes.c_double),  # 8B
        ("c_0", ctypes.c_char),  # 1B
        ("struct2_0", Struct2),
    ]


class Struct0(ctypes.Structure):
    if PRAGMA_PACK:
        _pack_ = PRAGMA_PACK
    _fields_ = [
        ("i_0", ctypes.c_int),  # 4B
        ("s_0", ctypes.c_short),  # 2B
        ("struct1_0", Struct1),
    ]

注意事项：

• 我将感兴趣的成员命名为 wanted（Struct2 的一部分，这是最深的成员）
• 在处理 struct 时有一件很重要的事情，那就是 alignment。更多详情请查看[MSDN]: #pragma pack。

为了说明第2个^nd（上），我准备了一个小例子（与问题无关）。

test_addressof.py：

import sys
import ctypes
import data_types


OFFSET_TEXT = "Offset of '{:s}' member in '{:s}' instance: {:3d} (0x{:08X})"


def offset_addressof(child_structure_instance, parent_structure_instance):
    return ctypes.addressof(child_structure_instance) - ctypes.addressof(parent_structure_instance)


def print_offset_addressof_data(child_structure_instance, parent_structure_instance):
    offset = offset_addressof(child_structure_instance, parent_structure_instance)
    print(OFFSET_TEXT.format(child_structure_instance.__class__.__name__, parent_structure_instance.__class__.__name__, offset, offset))


def main():
    s0 = data_types.Struct0()
    s1 = s0.struct1_0
    s2 = s1.struct2_0
    print("PRAGMA_PACK: {:d} {:s}\n".format(data_types.PRAGMA_PACK, "" if data_types.PRAGMA_PACK else "(default)"))
    print_offset_addressof_data(s1, s0)
    print_offset_addressof_data(s2, s1)
    print_offset_addressof_data(s2, s0)
    print("\nAlignments and sizes:\n\t'{:s}': {:3d} - {:3d}\n\t'{:s}': {:3d} - {:3d}\n\t'{:s}': {:3d} - {:3d}".format(
            s0.__class__.__name__, ctypes.alignment(s0), ctypes.sizeof(s0),
            s1.__class__.__name__, ctypes.alignment(s1), ctypes.sizeof(s1),
            s2.__class__.__name__, ctypes.alignment(s2), ctypes.sizeof(s2)
        )
    )
    #print("Struct0().i_0 type: {:s}".format(s0.i_0.__class__.__name__))


if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

注意事项：

• 原生 C 成员类型转换为 Python 类型，在 ctypes.Structure 中，ctypes.addressof 如果收到这样的消息，则会引发 TypeError一个参数（检查 main 中的注释 print）
• 我尝试在各种 OSes 中使用具有相同大小的 C 类型（例如，我避免了在 Lnxctypes.c_long /em> 和 Win 上的 4 个字节长（当然是在谈论 64 位版本）
• 在两次示例运行之间需要修改源代码。我本可以动态生成这些类，但这会给代码增加不必要的复杂性（并远离我试图提出的观点）

输出：

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python test_addressof.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 0 (default)

Offset of 'Struct1' member in 'Struct0' instance:   8 (0x00000008)
Offset of 'Struct2' member in 'Struct1' instance:  12 (0x0000000C)
Offset of 'Struct2' member in 'Struct0' instance:  20 (0x00000014)

Alignments and sizes:
        'Struct0':   8 -  32
        'Struct1':   8 -  24
        'Struct2':   4 -   8

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>rem change PRAGMA_PACK = 1 in data_types.py

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python test_addressof.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 1

Offset of 'Struct1' member in 'Struct0' instance:   6 (0x00000006)
Offset of 'Struct2' member in 'Struct1' instance:   9 (0x00000009)
Offset of 'Struct2' member in 'Struct0' instance:  15 (0x0000000F)

Alignments and sizes:
        'Struct0':   1 -  22
        'Struct1':   1 -  16
        'Struct2':   1 -   7

struct_util.py：

import sys
import ctypes

import data_types


WANTED_MEMBER_NAME = "wanted"
FIELDS_MEMBER_NAME = "_fields_"


def _get_padded_size(sizes, align_size):
    padded_size = temp = 0
    for size in sizes:
        if temp >= align_size:
            padded_size += temp
            temp = size
        elif temp + size > align_size:
            padded_size += align_size
            temp = size
        else:
            temp += size
    if temp:
        padded_size += max(size, align_size)
    return padded_size


def _get_array_type_sizes(array_type):
    if issubclass(array_type._type_, ctypes.Array):
        return _get_array_type_sizes(array_type._type_) * array_type._type_._length_
    else:
        return [array_type._type_] * array_type._length_


def get_nested_offset_recursive(struct_instance, wanted_member_name):
    if not isinstance(struct_instance, ctypes.Structure):
        return -1
    align_size = ctypes.alignment(struct_instance)
    base_address = ctypes.addressof(struct_instance)
    member_sizes = list()
    for member_name, member_type in getattr(struct_instance, FIELDS_MEMBER_NAME, list()):
        if member_name == wanted_member_name:
            return _get_padded_size(member_sizes, align_size)
        if issubclass(member_type, ctypes.Structure):
            nested_struct_instance = getattr(struct_instance, member_name)
            inner_offset = get_nested_offset_recursive(nested_struct_instance, wanted_member_name)
            if inner_offset != -1:
                return ctypes.addressof(nested_struct_instance) - base_address + inner_offset
            else:
                member_sizes.append(ctypes.sizeof(member_type))
        else:
            if issubclass(member_type, ctypes.Array):
                member_sizes.extend(_get_array_type_sizes(member_type))
            else:
                member_sizes.append(ctypes.sizeof(member_type))
    return -1


def _get_struct_instance_from_name(struct_name):
    struct_class = getattr(data_types, struct_name, None)
    if struct_class:
        return struct_class()


def get_nested_offset(struct_name, wanted_member_name):
    struct_instance = _get_struct_instance_from_name(struct_name)
    return get_nested_offset_recursive(struct_instance, wanted_member_name)


def main():
    struct_names = [
        "Struct2",
        "Struct1",
        "Struct0"
    ]
    wanted_member_name = WANTED_MEMBER_NAME
    print("PRAGMA_PACK: {:d} {:s}\n".format(data_types.PRAGMA_PACK, "" if data_types.PRAGMA_PACK else "(default)"))
    for struct_name in struct_names:
        print("'{:s}' offset in '{:s}' (size: {:3d}): {:3d}".format(wanted_member_name,
                                                                    struct_name,
                                                                    ctypes.sizeof(_get_struct_instance_from_name(struct_name)),
                                                                    get_nested_offset(struct_name, wanted_member_name)))


if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

注意事项：

• 代码比我最初预期的要复杂（waaaay）。我认为有一种更简单的方法，但我就是看不到。希望我没有错过这么明显的东西，整个事情可以用 2-3 行代码完成
• 它应该适用于任何结构，尽管有（很多）情况我没有测试（尤其是结构数组，其中有些情况不起作用）
• 它将在找到的第 1 个^st 成员出现处停止
• 功能（1×1）：
  • get_nested_offset_recursive - 核心功能：递归搜索结构中的成员并计算其偏移量。有2种情况：
    • 成员位于 child 结构中（或 child 的 child，...）：偏移到 child em> 结构是通过减去 2 个结构地址来计算的（使用 ctypes.addressof）
    • 成员在当前结构中（复杂情况）：偏移量的计算考虑了之前成员的大小和结构对齐方式
  • _get_padded_size - 尝试在 align_size 大块中匹配成员大小（在我们关心的那个之前），并返回块大小总和
  • _get_array_type_sizes - 数组不是原子（来自对齐PoV）：char c[10]; 成员可以替换为char c0, c1, ..., c9;。这就是这个函数的作用（递归）
  • _get_struct_instance_from_\name - 辅助或便利函数：返回作为参数给出的结构名称的实例（在 data_types 模块中搜索）
  • get_nested_offset - 包装函数

输出（原理同上）：

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python struct_util.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 0 (default)

'wanted' offset in 'Struct2' (size:   8):   4
'wanted' offset in 'Struct1' (size:  24):  16
'wanted' offset in 'Struct0' (size:  32):  24

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>rem change PRAGMA_PACK = 1 in data_types.py

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python struct_util.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 1

'wanted' offset in 'Struct2' (size:   7):   3
'wanted' offset in 'Struct1' (size:  16):  12
'wanted' offset in 'Struct0' (size:  22):  18

@EDIT0：

正如我在 1^st 和（尤其是）2^nd 注释中指定的那样，我对解决方案不满意，主要是因为即使它适用于当前场景，它不适用于一般场景（嵌套数组和结构）。然后我遇到了[SO]: Ctypes: Get a pointer to a struct field (@MarkTolonen's answer)，并采取了不同的方法。

data_types.py（在前面的内容中添加以下代码）：

class Struct0_1(ctypes.Structure):
    if PRAGMA_PACK:
        _pack_ = PRAGMA_PACK
    _fields_ = [
        ("i_0", ctypes.c_int),  # 4B
        ("s_0", ctypes.c_short),  # 2B
        ("struct1_0_2", Struct1 * 2),
        ("i_1", ctypes.c_int * 2),  # 2 * 4B
        ("struct1_1", Struct1),
        ("i_2", ctypes.c_int),  # 4B
        ("struct1_2_3", Struct1 * 3),
    ]

struct_util_v2.py：

import sys
import ctypes

import data_types


WANTED_MEMBER_NAME = "wanted"

def _get_nested_offset_recursive_struct(struct_ctype, member_name):
    for struct_member_name, struct_member_ctype in struct_ctype._fields_:
        struct_member = getattr(struct_ctype, struct_member_name)
        offset = struct_member.offset
        if struct_member_name == member_name:
            return offset
        else:
            if issubclass(struct_member_ctype, ctypes.Structure):
                inner_offset = _get_nested_offset_recursive_struct(struct_member_ctype, member_name)
            elif issubclass(struct_member_ctype, ctypes.Array):
                inner_offset = _get_nested_offset_recursive_array(struct_member_ctype, member_name)
            else:
                inner_offset = -1
            if inner_offset != -1:
                return inner_offset + offset
    return -1


def _get_nested_offset_recursive_array(array_ctype, member_name):
    array_base_ctype = array_ctype._type_
    for idx in range(array_ctype._length_):
        if issubclass(array_base_ctype, ctypes.Structure):
            inner_offset = _get_nested_offset_recursive_struct(array_base_ctype, member_name)
        elif issubclass(array_base_ctype, ctypes.Array):
            inner_offset = _get_nested_offset_recursive_array(array_base_ctype, member_name)
        else:
            inner_offset = -1
        return inner_offset


def get_nested_offset_recursive(ctype, member_name, nth=1):
    if issubclass(ctype, ctypes.Structure):
        return _get_nested_offset_recursive_struct(ctype, member_name)
    elif issubclass(ctype, ctypes.Array):
        return _get_nested_offset_recursive_array(ctype, member_name)
    else:
        return -1


def main():
    struct_names = [
        "Struct2",
        "Struct1",
        "Struct0",
        "Struct0_1",
    ]
    member_name = WANTED_MEMBER_NAME
    print("PRAGMA_PACK: {:d} {:s}\n".format(data_types.PRAGMA_PACK, "" if data_types.PRAGMA_PACK else "(default)"))
    for struct_name in struct_names:
        struct_ctype = getattr(data_types, struct_name)
        print("'{:s}' offset in '{:s}' (size: {:3d}): {:3d}".format(member_name,
                                                                    struct_name,
                                                                    ctypes.sizeof(struct_ctype),
                                                                    get_nested_offset_recursive(struct_ctype, member_name)))


if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

注意事项：

• 不再使用实例，因为偏移元数据存储在类本身中（addressof 不再需要）
• 对于新添加的结构，之前的代码不起作用
• 新代码的真正强大之处在于处理 get_nested_offset_recursive 的 nth 参数（现在什么都不做 - 可以删除），它告诉哪个事件的偏移量应该报告成员名称（它只对结构数组有意义），但这有点复杂，因此需要更多代码
• 有争议的可能是结构成员是指向结构的指针（有些人可能认为将它们视为数组），但我认为由于此类（内部）结构驻留在另一个内存区域中，只需跳过它们（事实上代码使用这种方法更简单与决策无关）

输出：

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python struct_util_v2.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 0 (default)

'wanted' offset in 'Struct2' (size:   8):   4
'wanted' offset in 'Struct1' (size:  24):  16
'wanted' offset in 'Struct0' (size:  32):  24
'wanted' offset in 'Struct0_1' (size: 168):  24

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>rem change PRAGMA_PACK = 1 in data_types.py

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python struct_util_v2.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 1

'wanted' offset in 'Struct2' (size:   7):   3
'wanted' offset in 'Struct1' (size:  16):  12
'wanted' offset in 'Struct0' (size:  22):  18
'wanted' offset in 'Struct0_1' (size: 114):  18

@EDIT1：

添加了对 nth 参数的支持（重命名为：index）。

struct_util_v3.py：

import sys
import ctypes

import data_types


WANTED_MEMBER_NAME = "wanted"
OFFSET_INVALID = -1

def _get_nested_offset_recursive_struct(struct_ctype, member_name, index):
    current_index = 0
    for struct_member_name, struct_member_ctype in struct_ctype._fields_:
        struct_member = getattr(struct_ctype, struct_member_name)
        offset = struct_member.offset
        if struct_member_name == member_name:
            if index == 0:
                return offset, 0
            else:
                current_index += 1
        else:
            if issubclass(struct_member_ctype, ctypes.Structure):
                inner_offset, occurences = _get_nested_offset_recursive_struct(struct_member_ctype, member_name, index - current_index)
            elif issubclass(struct_member_ctype, ctypes.Array):
                inner_offset, occurences = _get_nested_offset_recursive_array(struct_member_ctype, member_name, index - current_index)
            else:
                inner_offset, occurences = OFFSET_INVALID, 0
            if inner_offset != OFFSET_INVALID:
                return inner_offset + offset, 0
            else:
                current_index += occurences
    return OFFSET_INVALID, current_index


def _get_nested_offset_recursive_array(array_ctype, member_name, index):
    array_base_ctype = array_ctype._type_
    array_base_ctype_size = ctypes.sizeof(array_base_ctype)
    current_index = 0
    for idx in range(array_ctype._length_):
        if issubclass(array_base_ctype, ctypes.Structure):
            inner_offset, occurences = _get_nested_offset_recursive_struct(array_base_ctype, member_name, index - current_index)
        elif issubclass(array_base_ctype, ctypes.Array):
            inner_offset, occurences = _get_nested_offset_recursive_array(array_base_ctype, member_name, index - current_index)
        else:
            inner_offset, occurences = OFFSET_INVALID, 0
        if inner_offset != OFFSET_INVALID:
            return array_base_ctype_size * idx + inner_offset, 0
        else:
            if occurences == 0:
                return OFFSET_INVALID, 0
            else:
                current_index += occurences
    return OFFSET_INVALID, current_index


def get_nested_offset_recursive(ctype, member_name, index=0):
    if index < 0:
        return OFFSET_INVALID
    if issubclass(ctype, ctypes.Structure):
        return _get_nested_offset_recursive_struct(ctype, member_name, index)[0]
    elif issubclass(ctype, ctypes.Array):
        return _get_nested_offset_recursive_array(ctype, member_name, index)[0]
    else:
        return OFFSET_INVALID


def main():
    struct_names = [
        "Struct2",
        "Struct1",
        "Struct0",
        "Struct0_1",
    ]
    member_name = WANTED_MEMBER_NAME
    print("PRAGMA_PACK: {:d} {:s}\n".format(data_types.PRAGMA_PACK, "" if data_types.PRAGMA_PACK else "(default)"))
    for struct_name in struct_names:
        struct_ctype = getattr(data_types, struct_name)
        nth = 1
        ofs = get_nested_offset_recursive(struct_ctype, member_name, index=nth - 1)
        while ofs != OFFSET_INVALID:
            print("'{:s}' offset (#{:03d}) in '{:s}' (size: {:3d}): {:3d}".format(member_name,
                                                                                 nth,
                                                                                 struct_name,
                                                                                 ctypes.sizeof(struct_ctype),
                                                                                 ofs))
            nth += 1
            ofs = get_nested_offset_recursive(struct_ctype, member_name, index=nth - 1)


if __name__ == "__main__":
    print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
    main()

注意事项：

• get_nested_offset_recursive 的 index 参数是成员出现列表中的（0）索引 - 或出现次数在报告偏移量之前跳过（默认值：0 - 表示它将报告第 1^st 次出现的偏移量）
• 没有彻底测试，但我想我涵盖了所有情况
• 对于每个结构，程序会列出所有成员出现的偏移量（直到找不到）
• 现在，代码是我一开始想到的形状

输出：

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python struct_util_v3.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 0 (default)

'wanted' offset (#001) in 'Struct2' (size:   8):   4
'wanted' offset (#001) in 'Struct1' (size:  24):  16
'wanted' offset (#001) in 'Struct0' (size:  32):  24
'wanted' offset (#001) in 'Struct0_1' (size: 192):  24
'wanted' offset (#002) in 'Struct0_1' (size: 192):  48
'wanted' offset (#003) in 'Struct0_1' (size: 192):  72
'wanted' offset (#004) in 'Struct0_1' (size: 192): 104
'wanted' offset (#005) in 'Struct0_1' (size: 192): 136
'wanted' offset (#006) in 'Struct0_1' (size: 192): 160

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>rem change PRAGMA_PACK = 1 in data_types.py

(py35x64_test) e:\Work\Dev\StackOverflow\q050304516>python struct_util_v3.py
Python 3.5.4 (v3.5.4:3f56838, Aug  8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32

PRAGMA_PACK: 1

'wanted' offset (#001) in 'Struct2' (size:   7):   3
'wanted' offset (#001) in 'Struct1' (size:  16):  12
'wanted' offset (#001) in 'Struct0' (size:  22):  18
'wanted' offset (#001) in 'Struct0_1' (size: 130):  18
'wanted' offset (#002) in 'Struct0_1' (size: 130):  34
'wanted' offset (#003) in 'Struct0_1' (size: 130):  50
'wanted' offset (#004) in 'Struct0_1' (size: 130):  74
'wanted' offset (#005) in 'Struct0_1' (size: 130):  94
'wanted' offset (#006) in 'Struct0_1' (size: 130): 110