自定义 c++ 扩展错误：粘贴“pybind11_init_”和“‘sigmoid’”未提供有效的预处理令牌

Question

我正在尝试使用 pytorch 添加自定义 c++ 扩展。

我正在关注https://pytorch.org/tutorials/advanced/cpp_extension.html的教程

我创建了两个文件。

project/
  main.py
  sigmoid.cpp

ma​​in.py

from torch.utils.cpp_extension import load 
lltm_cpp = load(name=‘sigmoid’, sources=[‘sigmoid.cpp’], verbose=True)

sigmoid.cpp

#include <torch/extension.h>

#include <iostream>

torch::Tensor d_sigmoid(torch::Tensor z) {
  auto s = torch::sigmoid(z);
  return (1 - s) * s;
}

PYBIND11_MODULE('sigmoid', m) {
  m.def('d_sigmoid', &d_sigmoid, 'sigmoid');
}

当我运行 main.py 时出现以下错误

错误：粘贴“pybind11_init_”和“‘sigmoid’”没有给出有效的 预处理令牌

我该如何解决这个错误？

完整的错误报告

Using /tmp/torch_extensions as PyTorch extensions root…
Emitting ninja build file /tmp/torch_extensions/sigmoid/build.ninja…
Building extension module sigmoid…
[1/2] c++ -MMD -MF sigmoid.o.d -DTORCH_EXTENSION_NAME=sigmoid -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /home/wickrama/anaconda3/envs/pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -c /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp -o sigmoid.o
FAILED: sigmoid.o
c++ -MMD -MF sigmoid.o.d -DTORCH_EXTENSION_NAME=sigmoid -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /home/wickrama/anaconda3/envs/pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -c /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp -o sigmoid.o
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “PyInit_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:11:9: warning: character constant too long for its type
m.def(‘d_sigmoid’, &d_sigmoid, ‘sigmoid’);
^~~~~~~~~~~
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:11:34: warning: character constant too long for its type
m.def(‘d_sigmoid’, &d_sigmoid, ‘sigmoid’);
^~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 949, in _build_extension_module
check=True)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/subprocess.py”, line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/wickrama/projects/torch_cpp_ext/main.py”, line 3, in
lltm_cpp = load(name=‘sigmoid’, sources=[‘sigmoid.cpp’], verbose=True)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 644, in load
is_python_module)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 813, in _jit_compile
with_cuda=with_cuda)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 866, in _write_ninja_file_and_build
_build_extension_module(name, build_directory, verbose)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 962, in _build_extension_module
raise RuntimeError(message)
RuntimeError: Error building extension ‘sigmoid’

Answer 1

我的猜测：与 Python 不同，单引号 (') 不等同于 c++ 中的双引号 (")。前者用于构造char字面量，因此只能换行单个字符，后者用于构造字符串字面量（严格来说const char *字面量）。所以尝试替换

PYBIND11_MODULE('sigmoid', m) {
  m.def('d_sigmoid', &d_sigmoid, 'sigmoid');
}

与

PYBIND11_MODULE("sigmoid", m) {
  m.def("d_sigmoid", &d_sigmoid, "sigmoid");
}

看看有没有帮助。