spacy 2.2.3 FileNotFoundError: [Errno 2] No such file or directory: 'thinc\\neural\\_custom_kernels.cu' in pyinstaller

Question

我试图使用 pyinstaller 创建一个可执行文件。执行问题时遇到以下问题。

 File "test_env2_live\main.py", line 2, in <module>
 File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "test_env2_live\controller\main.py", line 2, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "test_env2_live\controller\inflow\job_controller.py", line 175, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\spacy\__init__.py", line 10, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\__init__.py", line 4, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\_classes\model.py", line 11, in <module>
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\train.py", line 7, in <module>
File "optimizers.pyx", line 14, in init thinc.neural.optimizers
File "ops.pyx", line 24, in init thinc.neural.ops
ImportError: cannot import name _custom_kernels

所以我在构建可执行文件时添加了 --hiddenimport thinc.neural._custom_kernels。下面是构建可执行文件的代码

pyinstaller main.py --hiddenimport preshed.maps --hiddenimport srsly.msgpack.util --hidden-import="sklearn.utils._cython_blas" --hidden-import="sklearn.neighbors.typedefs" --hidden-import="sklearn.neighbors.quad_tree" --hidden-import="sklearn.tree._utils" --hidden-import cymem.cymem --hidden-import thinc.linalg --hidden-import murmurhash.mrmr --hidden-import cytoolz.utils --hidden-import cytoolz._signatures --hidden-import spacy.strings --hidden-import spacy.morphology --hidden-import spacy.lexeme --hidden-import spacy.tokens --hidden-import spacy.gold --hidden-import spacy.tokens.underscore --hidden-import spacy.parts_of_speech --hidden-import dill --hidden-import spacy.tokens.printers --hidden-import spacy.tokens._retokenize --hidden-import spacy.syntax --hidden-import spacy.syntax.stateclass --hidden-import spacy.syntax.transition_system --hidden-import spacy.syntax.nonproj --hidden-import spacy.syntax.nn_parser --hidden-import spacy.syntax.arc_eager --hidden-import thinc.extra.search --hidden-import spacy.syntax._beam_utils --hidden-import spacy.syntax.ner --hidden-import thinc.neural._classes.difference --hidden-import spacy.vocab --hidden-import spacy.lemmatizer --hidden-import spacy._ml --hidden-import spacy.lang.en --hiddenimport thinc.neural._custom_kernels

但现在可执行文件给出以下错误

File "site-packages\thinc\neural\train.py", line 7, in <module>
File "optimizers.pyx", line 14, in init thinc.neural.optimizers
File "ops.pyx", line 24, in init thinc.neural.ops
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "C:\Users\rajesh.das\AppData\Local\Continuum\anaconda3\envs\test_env2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
exec(bytecode, module.__dict__)
File "site-packages\thinc\neural\_custom_kernels.py", line 36, in <module>
File "pathlib.py", line 1183, in open
File "pathlib.py", line 1037, in _opener
File "pathlib.py", line 387, in wrapped
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\rajesh\\python\\console\\test_env2_live\\dist\\main\\thinc\\neural\\_custom_kernels.cu'

不确定为什么会出现 FileNotFoundError 错误。谁能帮我？

配置：Python 3.6.7、conda 4.8.0、spacy v2.2.3、W​​indows 10

我可以使用较低版本的 spacy(2.0.18) 成功运行可执行文件。但在 2.2.3 版本中会报错。

提前致谢

Answer 1

FileNotFound 错误是因为 PyInstaller 没有正确打包 thinc； thinc 需要一个钩子。我发现包含from spacy import * 的脚本将与下面的钩子文件一起使用。我使用的命令是：

pyinstaller test-spacy.py --additional-hooks-dir=.

这很有效，因为我在与test-spacy.py 脚​​本相同的目录中为spacy（及其子模块）添加了一个钩子。只需将以下文本复制到一个名为 hook-spacy.py 的文件中，该文件与您的脚本位于同一目录中。

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]
# This hook file is a bit of a hack - really, all of the libraries should be in seperate hook files. (Eg hook-blis.py with the blis part of the hook)

如果您需要更多帮助，请咨询 cmets。

Answer 2

我同意@Legorooj。除了他的回答之外，我最后不得不为 srsly.msgpack.util 添加另一个隐藏的导入。同样在我的脚本中，我使用了“en_core_web_sm”：

import en_core_web_sm
nlp = en_core_web_sm.load()

所以我也为“en_core_web_sm”添加了一个挂钩文件。 hook-spacy.py 文件变为：

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- OTHER ----------------------------

hiddenimports += ['srsly.msgpack.util']

而 hook-en_core_web_sm 文件是：

from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files("en_core_web_sm")

Answer 3

# -*- mode: python ; coding: utf-8 -*-

block_cipher = None

import PyInstaller

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all
from PyInstaller.utils.hooks import collect_data_files

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- OTHER ----------------------------

hiddenimports += ['srsly.msgpack.util']


# ----------------------------- Sacy Model -------------------------
datas += collect_data_files("en_core_web_sm")


a = Analysis(['ttest_spacy.py'],
             pathex=['PATH/TO/PYFILE'],
             binaries=binaries,
             datas=datas,
             hiddenimports = hiddenimports,
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          [],
          exclude_binaries=True,
          name='ttest_spacy',
          debug=False,
          bootloader_ignore_signals=False,
          strip=False,
          upx=True,
          console=True )
coll = COLLECT(exe,
               a.binaries,
               a.zipfiles,
               a.datas,
               strip=False,
               upx=True,
               upx_exclude=[],
               name='ttest_spacy')

Answer 4

我调整了 .spec 文件并添加了包含以下详细信息的 main.spec 文件，它可以正常工作。

# -*- mode: python ; coding: utf-8 -*-

import PyInstaller

datas = []
datas.extend(PyInstaller.utils.hooks.collect_data_files('spacy.lang', include_py_files = True))
datas.extend(PyInstaller.utils.hooks.collect_data_files('thinc'))

block_cipher = None
a = Analysis(['main.py'],
         pathex=['D:\\rajesh\\python\\console\\live'],
         binaries=[],
         datas=datas,
         hiddenimports = [
            'spacy.kb',
            'spacy.lexeme',
            'spacy.matcher._schemas',
            'spacy.morphology',
            'spacy.parts_of_speech',
            'spacy.syntax._beam_utils',
            'spacy.syntax._parser_model',
            'spacy.syntax.arc_eager',
            'spacy.syntax.ner',
            'spacy.syntax.nn_parser',
            'spacy.syntax.stateclass',
            'spacy.syntax.transition_system',
            'spacy.tokens._retokenize',
            'spacy.tokens.morphanalysis',
            'spacy.tokens.underscore',

            'spacy._align',

            'blis',
            'blis.py',

            'cymem',
            'cymem.cymem',

            'murmurhash',
            'murmurhash.mrmr',

            'preshed.maps',

            'srsly.msgpack.util',

            'thinc.extra.search',
            'thinc.linalg',
            'thinc.neural._aligned_alloc',
            'thinc.neural._custom_kernels',

            'sklearn.utils._cython_blas',
            'sklearn.neighbors.typedefs',
            'sklearn.neighbors.quad_tree',
            'sklearn.tree._utils'
        ],
         hookspath=[],
         runtime_hooks=[],
         excludes=[],
         win_no_prefer_redirects=False,
         win_private_assemblies=False,
         cipher=block_cipher,
         noarchive=False)
   pyz = PYZ(a.pure, a.zipped_data,
         cipher=block_cipher)
    exe = EXE(pyz,
      a.scripts,
      [],
      exclude_binaries=True,
      name='main',
      debug=False,
      bootloader_ignore_signals=False,
      strip=False,
      upx=True,
      console=True )
   coll = COLLECT(exe,
           a.binaries,/
           a.zipfiles,
           a.datas,
           strip=False,
           upx=True,
           upx_exclude=[],
           name='main')