为包含编码字节字符串的文件生成 sphinx autodoc答案

【问题标题】：Generate sphinx autodoc for files that contain encoded byte strings为包含编码字节字符串的文件生成 sphinx autodoc
【发布时间】：2023-03-16 07:53:01
【问题描述】：

我有几个包含结束编码字节字符串的文件，例如：

variable_name = (b'编码字符串')

但是，当我运行 make html 通过 Sphinx 生成自动文档时，我收到以下错误。

Encoding error:

'utf-8' codec can't decode byte 0xc0 in position 3: invalid start byte

我仍然需要 sphinx 为这些文件和我的项目中的其余文件生成文档而不是失败，或者最坏的情况仍然为不包含这些字节字符串的每个其他文件生成文档并跳过那些做的。是否有一个配置变量我可以添加到我的 conf.py 文件中以使其识别字节字符串或只是一起忽略它，这是否意味着不记录该变量或跳过整个文件然后继续记录剩余的文件?

重现步骤：

使用 autodoc 创建 sphinx 项目
将“vari = (b'\n$\x00\xc0\x91 \xa0l')”添加到任何 .py 文件中
使用“make html”生成文档 - 失败并出现上述错误

非常感谢任何帮助！提前谢谢！

# Sphinx version: 2.1.2
# Python version: 3.5.2 (CPython)
# Docutils version: 0.14 
# Jinja2 version: 2.10.1
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/sphinx/cmd/build.py", line 283, in build_main
    args.tags, args.verbosity, args.jobs, args.keep_going)
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/sphinx/application.py", line 268, in __init__
    self._init_builder()
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/sphinx/application.py", line 329, in _init_builder
    self.events.emit('builder-inited')
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/sphinx/events.py", line 103, in emit
    results.append(callback(self.app, *args))
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/extension.py", line 97, in run_autoapi
    patterns=file_patterns, dirs=normalized_dirs, ignore=ignore_patterns
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/mapper.py", line 213, in load
    data = self.read_file(path=path)
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/mapper.py", line 224, in read_file
    parsed_data = Parser().parse_file(path)
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/parser.py", line 46, in parse_file
    return self.parse(node)
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/parser.py", line 240, in parse
    data = parse_func(node)
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/parser.py", line 227, in parse_module
    child_data = self.parse(child)
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/parser.py", line 240, in parse
    data = parse_func(node)
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/parser.py", line 72, in parse_assign
    value = self._encode(assign_value[1])
  File "/home/joshsullivan/.py_virtualenvs/verb_py3/lib/python3.5/site-packages/autoapi/mappers/python/parser.py", line 25, in _encode
    return _TEXT_TYPE(to_encode, self._encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 3: invalid start byte

我的文件.py

'''This is my docstring'''
vari = (b'\n$\x00\xc0\x91 \xa0l')

我的 Conf.py 文件：

# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config

# -- Path setup --------------------------------------------------------------

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))


# -- Project information -----------------------------------------------------

project = 'vstars'
copyright = '2019, S.P.A.D.E'
author = 'S.P.A.D.E'

# The short X.Y version
version = '1.0'

# The full version, including alpha/beta/rc tags
release = '1.0.1'


# -- General configuration ---------------------------------------------------

# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '2.0.1'

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
    'sphinx.ext.autodoc',
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'

# The master toctree document.
master_doc = 'index'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
#html_theme = 'alabaster'

import sphinx_rtd_theme

html_theme = 'sphinx_rtd_theme'

html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

# Theme options are theme-specific and customize the look and feel of a theme
# further.  For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself.  Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}


# -- Options for HTMLHelp output ---------------------------------------------

# Output file base name for HTML help builder.
htmlhelp_basename = 'vstarsdoc'


# -- Options for LaTeX output ------------------------------------------------

latex_elements = {
    # The paper size ('letterpaper' or 'a4paper').
    #
    # 'papersize': 'letterpaper',

    # The font size ('10pt', '11pt' or '12pt').
    #
    # 'pointsize': '10pt',

    # Additional stuff for the LaTeX preamble.
    #
    # 'preamble': '',

    # Latex figure (float) alignment
    #
    # 'figure_align': 'htbp',
}

# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
#  author, documentclass [howto, manual, or own class]).
latex_documents = [
    (master_doc, 'vstars.tex', 'vstars\\_sphinx\\_documentation Documentation',
     'Joshua Sullivan', 'manual'),
]


# -- Options for manual page output ------------------------------------------

# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
    (master_doc, 'vstars', 'vstars Documentation',
     [author], 1)
]


# -- Options for Texinfo output ----------------------------------------------

# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
#  dir menu entry, description, category)
texinfo_documents = [
    (master_doc, 'vstars', 'vstars Documentation',
     author, 'vstars', 'One line description of project.',
     'Miscellaneous'),
]


# -- Options for Epub output -------------------------------------------------

# Bibliographic Dublin Core info.
epub_title = project

# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''

# A unique identification for the text.
#
# epub_uid = ''

# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']


# -- Extension configuration -------------------------------------------------
extensions = ['autoapi.extension']
# Document Python Code
autoapi_type = 'python'
autoapi_dirs = ['../']

我的 Index.rst 文件：

.. vstars documentation master file, created by
   sphinx-quickstart on Wed Jul  3 10:03:20 2019.
   You can adapt this file completely to your liking, but it should at least
   contain the root `toctree` directive.

Welcome to vstars's documentation!
==================================

.. toctree::
   :maxdepth: 2
   :caption: Contents:



Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

我的 Makefile：

# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line.
SPHINXOPTS    =
SPHINXBUILD   = sphinx-build
SOURCEDIR     = .
BUILDDIR      = _build

# Generate __init__.py files for all subdirectories of vstars
init:
    find ../ -type d -not -path "../doc*" -exec touch {}/__init__.py \;

# Put it first so that "make" without argument is like "make help".
help:
    @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
    @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

文件夹结构：

vstars
- my_file.py
- 文档
  - Conf.py
  - 索引.rst
  - 生成文件

【问题讨论】：

bytes 文字中包含文字非 ASCII 字符是语法错误。我怀疑解码错误是由其他原因引起的。
如果我在 Python 交互式会话中执行 b'\n$\x00\xc0\x91 \xa0l'.decode()，我会得到与问题中相同的错误。但是在运行 Sphinx 时我无法重现该问题。
我看不出为什么 Sphinx 会尝试使用 UTF-8 解码一些任意字节对象，因为这很容易失败。我希望 Sphinx 使用 UTF-8 解码源代码（Python 3 中的默认值），但这不应该导致此错误，因为有问题的字节在源中被转义（如\xc0，即四个 ASCII 字符） .狮身人面像是否给你一个回溯，即。你能确定是字节文字导致了异常吗？
@lenz 我添加了tracebrack，这样你就可以看到了。它似乎是由字节文字引起的。我还通过简单地注释掉文字并执行“make html”调用和 sphinx 正确构建来证实了这一点。
我可以重现该问题。堆栈跟踪中的最后一项是指 Sphinx AutoAPI 扩展源代码中的这一行：github.com/readthedocs/sphinx-autoapi/blob/master/autoapi/…。

标签： python-3.x encoding utf-8 python-sphinx

【解决方案1】：

原因

UnicodeDecodeError 源于 Sphinx AutoAPI 扩展 (https://sphinx-autoapi.readthedocs.io)。堆栈跟踪中的最后一项是指autoapi/mappers/python/parser.py 中的第 25 行。

_TEXT_TYPE 是str 的别名，所以_TEXT_TYPE(to_encode, self._encoding) 在运行时表示str(b'\n$\x00\xc0\x91 \xa0l', "UTF-8")，相当于b'\n$\x00\xc0\x91 \xa0l'.decode("UTF-8")。

它看起来像一个错误。 “香草”狮身人面像不会出现此问题。

解决方法

将以下猴子补丁添加到 conf.py：

from autoapi.mappers.python.parser import Parser

def patched_encode(self, to_encode):
    if self._encoding:
        try:
            if not(isinstance(bytes, to_encode)):  # <- The patch
                return _TEXT_TYPE(to_encode, self._encoding)
        except TypeError:
            # The string was already in the correct format
            pass

    return to_encode

Parser._encode = patched_encode

【讨论】：