【问题标题】:Extending setuptools extension to use CMake in setup.py?扩展 setuptools 扩展以在 setup.py 中使用 CMake?
【发布时间】:2017-07-23 22:29:39
【问题描述】:

我正在编写一个链接 C++ 库的 Python 扩展,并且我正在使用 cmake 来帮助构建过程。这意味着现在,我知道如何捆绑它的唯一方法是,我必须先用 cmake 编译它们,然后才能运行 setup.py bdist_wheel。一定有更好的办法。

我想知道是否可以(或任何人尝试过)调用 CMake 作为 setup.py ext_modules 构建过程的一部分?我猜有一种方法可以创建某些东西的子类,但我不确定在哪里看。

我使用 CMake 是因为它让我可以更好地控制构建 c 和 c++ 库扩展,并完全按照我的需要进行复杂的构建步骤。另外,我可以使用 findPythonLibs.cmake 中的 PYTHON_ADD_MODULE() 命令直接使用 cmake 轻松构建 Python 扩展。我只希望这只是一步。

【问题讨论】:

  • 运气好能解决这个问题吗?我面临着一个非常相似的挑战。目前,我添加了一个自定义目标,该目标依赖于构建二进制文件的目标,并将它们组合成一个 setup.py,将它们包含为package_data,但这一切看起来都像是一个大hack。感觉一定有更好的办法
  • 我自己运气不好。这正是我一直在做的事情,感觉很hacky。我希望我知道更好的方法。

标签: python c++ cmake setuptools packaging


【解决方案1】:

您基本上需要做的是覆盖setup.py 中的build_ext 命令类并将其注册到命令类中。在build_ext 的自定义实现中,配置并调用cmake 来配置然后构建扩展模块。不幸的是,官方文档对如何实现自定义distutils 命令相当简洁(参见Extending Distutils);我发现直接研究命令代码更有帮助。例如,这里是build_ext command的源代码。

示例项目

我准备了一个简单的项目,由一个 C 扩展 foo 和一个 python 模块 spam.eggs 组成:

so-42585210/
├── spam
│   ├── __init__.py  # empty
│   ├── eggs.py
│   ├── foo.c
│   └── foo.h
├── CMakeLists.txt
└── setup.py

用于测试设置的文件

这些只是我为测试设置脚本而编写的一些简单存根。

spam/eggs.py(仅用于测试库调用):

from ctypes import cdll
import pathlib


def wrap_bar():
    foo = cdll.LoadLibrary(str(pathlib.Path(__file__).with_name('libfoo.dylib')))
    return foo.bar()

spam/foo.c:

#include "foo.h"

int bar() {
    return 42;
}

spam/foo.h:

#ifndef __FOO_H__
#define __FOO_H__

int bar();

#endif

CMakeLists.txt:

cmake_minimum_required(VERSION 3.10.1)
project(spam)
set(src "spam")
set(foo_src "spam/foo.c")
add_library(foo SHARED ${foo_src})

设置脚本

这就是魔法发生的地方。当然,还有很大的改进空间 - 如果需要,您可以将其他选项传递给 CMakeExtension 类(有关扩展的更多信息,请参阅 Building C and C++ Extensions),通过 setup.cfg 使 CMake 选项可配置覆盖方法 initialize_optionsfinalize_options 等。

import os
import pathlib

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext as build_ext_orig


class CMakeExtension(Extension):

    def __init__(self, name):
        # don't invoke the original build_ext for this special extension
        super().__init__(name, sources=[])


class build_ext(build_ext_orig):

    def run(self):
        for ext in self.extensions:
            self.build_cmake(ext)
        super().run()

    def build_cmake(self, ext):
        cwd = pathlib.Path().absolute()

        # these dirs will be created in build_py, so if you don't have
        # any python sources to bundle, the dirs will be missing
        build_temp = pathlib.Path(self.build_temp)
        build_temp.mkdir(parents=True, exist_ok=True)
        extdir = pathlib.Path(self.get_ext_fullpath(ext.name))
        extdir.mkdir(parents=True, exist_ok=True)

        # example of cmake args
        config = 'Debug' if self.debug else 'Release'
        cmake_args = [
            '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + str(extdir.parent.absolute()),
            '-DCMAKE_BUILD_TYPE=' + config
        ]

        # example of build args
        build_args = [
            '--config', config,
            '--', '-j4'
        ]

        os.chdir(str(build_temp))
        self.spawn(['cmake', str(cwd)] + cmake_args)
        if not self.dry_run:
            self.spawn(['cmake', '--build', '.'] + build_args)
        # Troubleshooting: if fail on line above then delete all possible 
        # temporary CMake files including "CMakeCache.txt" in top level dir.
        os.chdir(str(cwd))


setup(
    name='spam',
    version='0.1',
    packages=['spam'],
    ext_modules=[CMakeExtension('spam/foo')],
    cmdclass={
        'build_ext': build_ext,
    }
)

测试

构建项目的轮子,安装它。测试库是否安装:

$ pip show -f spam
Name: spam
Version: 0.1
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /Users/hoefling/.virtualenvs/stackoverflow/lib/python3.6/site-packages
Requires: 
Files:
  spam-0.1.dist-info/DESCRIPTION.rst
  spam-0.1.dist-info/INSTALLER
  spam-0.1.dist-info/METADATA
  spam-0.1.dist-info/RECORD
  spam-0.1.dist-info/WHEEL
  spam-0.1.dist-info/metadata.json
  spam-0.1.dist-info/top_level.txt
  spam/__init__.py
  spam/__pycache__/__init__.cpython-36.pyc
  spam/__pycache__/eggs.cpython-36.pyc
  spam/eggs.py
  spam/libfoo.dylib

spam.eggs 模块运行包装函数:

$ python -c "from spam import eggs; print(eggs.wrap_bar())"
42

【讨论】:

  • 有什么理由使用 distutils.command 的 build_ext 而不是 from setuptools.command.build_ext import build_ext
  • 确实,我记得我尝试覆盖 build_extension 并遇到了我无法解决的错误。等我回来工作后让我回顾一下,如果你仍然感兴趣,我会更新答案并联系你。
  • 我肯定仍然有兴趣,感谢您的宝贵时间。
  • 对我来说,'extdir.mkdir' 部分使用扩展的共享库的名称创建一个目录,然后在尝试使用该名称创建共享库时导致构建失败。我将其更改为 extdir.parent.mkdir(...) 以解决此问题。
  • 还有一件事;你有一条线:config = 'Debug' if self.debug else 'Release'。如何将self.debug 设置为 true?在安装过程中是否有一些命令行标志?我好像没找到。
【解决方案2】:

我想对此添加我自己的答案,作为对 hoefling 所描述内容的一种补充。

谢谢,hoefling,因为您的回答帮助我走上了正轨,以与我自己的存储库几乎相同的方式编写设置脚本。

序言

编写此答案的主要动机是尝试将缺失的部分“粘合在一起”。 OP 没有说明正在开发的 C/C++ Python 模块的性质;我想先明确一点,以下步骤适用于创建多个 .dll/ .so 文件以及预编译的 *.pyd/so 文件的 C/C++ cmake 构建链一些通用的.py 文件需要放在脚本目录中。

所有这些文件在运行cmake build 命令之后直接生成......很有趣。不建议以这种方式构建 setup.py。

因为 setup.py 意味着您的脚本将成为您的包/库的一部分,并且必须通过库部分声明需要构建的 .dll 文件,并列出源和包含目录,所以没有直观的方法可以告诉 setuptools 在build_ext 中发生的对cmake -b 的一次调用所产生的库、脚本和数据文件都应该放在各自的位置。更糟糕的是,如果您想让 setuptools 跟踪此模块并完全卸载,这意味着用户可以卸载它,并在需要时清除系统中的所有痕迹。

我编写 setup.py 的模块是 bpy,.pyd/.so 相当于将搅拌机构建为 Python 模块,如下所述:

https://wiki.blender.org/wiki//User:Ideasman42/BlenderAsPyModule(更好的指令,但现在是死链接) http://www.gizmoplex.com/wordpress/compile-blender-as-python-module/(可能是更糟糕的说明,但似乎仍然在线)

您可以在这里查看我在 github 上的存储库:

https://github.com/TylerGubala/blenderpy

这就是我写这个答案的动机,希望能帮助其他人尝试完成类似的事情,而不是丢弃他们的 cmake 构建链,或者更糟糕的是,不得不维护两个独立的构建环境。如果离题,我深表歉意。

那么我该怎么做才能做到这一点?

  1. 用我自己的类扩展setuptools.Extension 类,它不包含源或库属性的条目

  2. 用我自己的类扩展setuptools.commands.build_ext.build_ext 类,该类有一个自定义方法来执行我必要的构建步骤(git、svn、cmake、cmake --build)

  3. 用我自己的类扩展distutils.command.install_data.install_data 类(糟糕,distutils... 但是似乎没有 setuputils 等效项),以在 setuptools 记录创建期间标记构建的二进制库(安装-files.txt) 这样

    • 这些库将被记录并卸载,pip uninstall package_name

    • 命令py setup.py bdist_wheel 也可以在本地运行,并且 可用于提供源代码的预编译版本

  4. 用我自己的类扩展 setuptools.command.install_lib.install_lib 类,这将确保将构建的库从其生成的构建文件夹移动到 setuptools 期望它们所在的文件夹中(在 Windows 上,它会将 @987654341 @bin/Release 文件夹中的文件,而不是 setuptools 期望的位置)

  5. 使用我自己的类扩展 setuptools.command.install_scripts.install_scripts 类,以便将脚本文件复制到正确的目录(Blender 期望 2.79 或任何目录位于脚本位置)

    李>
  6. 执行构建步骤后,将这些文件复制到已知目录中,setuptools 将复制到我环境的站点包目录中。此时剩余的 setuptools 和 distutils 类可以接管写入 installed-files.txt 记录,并将完全删除!

示例

这是一个示例,或多或少来自我的存储库,但为了更具体的内容清晰而进行了修剪(您可以随时前往存储库并自己查看)

from distutils.command.install_data import install_data
from setuptools import find_packages, setup, Extension
from setuptools.command.build_ext import build_ext
from setuptools.command.install_lib import install_lib
from setuptools.command.install_scripts import install_scripts
import struct

BITS = struct.calcsize("P") * 8
PACKAGE_NAME = "example"

class CMakeExtension(Extension):
    """
    An extension to run the cmake build

    This simply overrides the base extension class so that setuptools
    doesn't try to build your sources for you
    """

    def __init__(self, name, sources=[]):

        super().__init__(name = name, sources = sources)

class InstallCMakeLibsData(install_data):
    """
    Just a wrapper to get the install data into the egg-info

    Listing the installed files in the egg-info guarantees that
    all of the package files will be uninstalled when the user
    uninstalls your package through pip
    """

    def run(self):
        """
        Outfiles are the libraries that were built using cmake
        """

        # There seems to be no other way to do this; I tried listing the
        # libraries during the execution of the InstallCMakeLibs.run() but
        # setuptools never tracked them, seems like setuptools wants to
        # track the libraries through package data more than anything...
        # help would be appriciated

        self.outfiles = self.distribution.data_files

class InstallCMakeLibs(install_lib):
    """
    Get the libraries from the parent distribution, use those as the outfiles

    Skip building anything; everything is already built, forward libraries to
    the installation step
    """

    def run(self):
        """
        Copy libraries from the bin directory and place them as appropriate
        """

        self.announce("Moving library files", level=3)

        # We have already built the libraries in the previous build_ext step

        self.skip_build = True

        bin_dir = self.distribution.bin_dir

        # Depending on the files that are generated from your cmake
        # build chain, you may need to change the below code, such that
        # your files are moved to the appropriate location when the installation
        # is run

        libs = [os.path.join(bin_dir, _lib) for _lib in 
                os.listdir(bin_dir) if 
                os.path.isfile(os.path.join(bin_dir, _lib)) and 
                os.path.splitext(_lib)[1] in [".dll", ".so"]
                and not (_lib.startswith("python") or _lib.startswith(PACKAGE_NAME))]

        for lib in libs:

            shutil.move(lib, os.path.join(self.build_dir,
                                          os.path.basename(lib)))

        # Mark the libs for installation, adding them to 
        # distribution.data_files seems to ensure that setuptools' record 
        # writer appends them to installed-files.txt in the package's egg-info
        #
        # Also tried adding the libraries to the distribution.libraries list, 
        # but that never seemed to add them to the installed-files.txt in the 
        # egg-info, and the online recommendation seems to be adding libraries 
        # into eager_resources in the call to setup(), which I think puts them 
        # in data_files anyways. 
        # 
        # What is the best way?

        # These are the additional installation files that should be
        # included in the package, but are resultant of the cmake build
        # step; depending on the files that are generated from your cmake
        # build chain, you may need to modify the below code

        self.distribution.data_files = [os.path.join(self.install_dir, 
                                                     os.path.basename(lib))
                                        for lib in libs]

        # Must be forced to run after adding the libs to data_files

        self.distribution.run_command("install_data")

        super().run()

class InstallCMakeScripts(install_scripts):
    """
    Install the scripts in the build dir
    """

    def run(self):
        """
        Copy the required directory to the build directory and super().run()
        """

        self.announce("Moving scripts files", level=3)

        # Scripts were already built in a previous step

        self.skip_build = True

        bin_dir = self.distribution.bin_dir

        scripts_dirs = [os.path.join(bin_dir, _dir) for _dir in
                        os.listdir(bin_dir) if
                        os.path.isdir(os.path.join(bin_dir, _dir))]

        for scripts_dir in scripts_dirs:

            shutil.move(scripts_dir,
                        os.path.join(self.build_dir,
                                     os.path.basename(scripts_dir)))

        # Mark the scripts for installation, adding them to 
        # distribution.scripts seems to ensure that the setuptools' record 
        # writer appends them to installed-files.txt in the package's egg-info

        self.distribution.scripts = scripts_dirs

        super().run()

class BuildCMakeExt(build_ext):
    """
    Builds using cmake instead of the python setuptools implicit build
    """

    def run(self):
        """
        Perform build_cmake before doing the 'normal' stuff
        """

        for extension in self.extensions:

            if extension.name == 'example_extension':

                self.build_cmake(extension)

        super().run()

    def build_cmake(self, extension: Extension):
        """
        The steps required to build the extension
        """

        self.announce("Preparing the build environment", level=3)

        build_dir = pathlib.Path(self.build_temp)

        extension_path = pathlib.Path(self.get_ext_fullpath(extension.name))

        os.makedirs(build_dir, exist_ok=True)
        os.makedirs(extension_path.parent.absolute(), exist_ok=True)

        # Now that the necessary directories are created, build

        self.announce("Configuring cmake project", level=3)

        # Change your cmake arguments below as necessary
        # Below is just an example set of arguments for building Blender as a Python module

        self.spawn(['cmake', '-H'+SOURCE_DIR, '-B'+self.build_temp,
                    '-DWITH_PLAYER=OFF', '-DWITH_PYTHON_INSTALL=OFF',
                    '-DWITH_PYTHON_MODULE=ON',
                    f"-DCMAKE_GENERATOR_PLATFORM=x"
                    f"{'86' if BITS == 32 else '64'}"])

        self.announce("Building binaries", level=3)

        self.spawn(["cmake", "--build", self.build_temp, "--target", "INSTALL",
                    "--config", "Release"])

        # Build finished, now copy the files into the copy directory
        # The copy directory is the parent directory of the extension (.pyd)

        self.announce("Moving built python module", level=3)

        bin_dir = os.path.join(build_dir, 'bin', 'Release')
        self.distribution.bin_dir = bin_dir

        pyd_path = [os.path.join(bin_dir, _pyd) for _pyd in
                    os.listdir(bin_dir) if
                    os.path.isfile(os.path.join(bin_dir, _pyd)) and
                    os.path.splitext(_pyd)[0].startswith(PACKAGE_NAME) and
                    os.path.splitext(_pyd)[1] in [".pyd", ".so"]][0]

        shutil.move(pyd_path, extension_path)

        # After build_ext is run, the following commands will run:
        # 
        # install_lib
        # install_scripts
        # 
        # These commands are subclassed above to avoid pitfalls that
        # setuptools tries to impose when installing these, as it usually
        # wants to build those libs and scripts as well or move them to a
        # different place. See comments above for additional information

setup(name='my_package',
      version='1.0.0a0',
      packages=find_packages(),
      ext_modules=[CMakeExtension(name="example_extension")],
      description='An example cmake extension module',
      long_description=open("./README.md", 'r').read(),
      long_description_content_type="text/markdown",
      keywords="test, cmake, extension",
      classifiers=["Intended Audience :: Developers",
                   "License :: OSI Approved :: "
                   "GNU Lesser General Public License v3 (LGPLv3)",
                   "Natural Language :: English",
                   "Programming Language :: C",
                   "Programming Language :: C++",
                   "Programming Language :: Python",
                   "Programming Language :: Python :: 3.6",
                   "Programming Language :: Python :: Implementation :: CPython"],
      license='GPL-3.0',
      cmdclass={
          'build_ext': BuildCMakeExt,
          'install_data': InstallCMakeLibsData,
          'install_lib': InstallCMakeLibs,
          'install_scripts': InstallCMakeScripts
          }
    )

以这种方式编写setup.py 后,构建python 模块就像运行py setup.py 一样简单,它将运行构建并生成输出文件。

建议您为网速慢或不想从源代码构建的用户制作轮子。为此,您需要安装 wheel 包 (py -m pip install wheel) 并通过执行 py setup.py bdist_wheel 生成一个轮子分发,然后像任何其他包一样使用 twine 上传它。

【讨论】:

  • 嗨,我有一个项目,其中包含来自 python.h 的包装 C 文件,并提供了一些用于调用 C/C++ api 并将其输出转换为 @987654351 的 api @ 类型。是否有一个选项可以在 cmake 中编译它并使用输出 .so 库文件在 egg 文件中生成 python 模块(到目前为止,我只能通过直接从 extension 对象编译来做到这一点)
  • @Zohar81 如果您能够使用Extension 对象进行编译,请拥有一个类似结构的setup.py 文件并安装最新版本的wheel (pip install wheel) 然后运行@987654358 @ 应该会给你想要的结果。如果没有,或者您有任何其他问题,请告诉我。
  • 我根据我的需要调整了您的代码,并注意到您不需要InstallCMakeLibsData,而是可以覆盖InstallCMakeLibs.get_outputs。但是,您的代码帮助我找到了 setuputils 和 distutils 代码的入口。
  • @John 这是一个有趣的观察结果,尽管我仍然倾向于通过子类化install_data 感觉更安全一些,但这取决于开发人员的偏好和必要性。应有的注意。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 2018-12-29
  • 2010-12-28
  • 2022-10-06
  • 1970-01-01
  • 1970-01-01
  • 2020-01-26
  • 2012-04-09
相关资源
最近更新 更多