Spacy - 错误的 config.file

Question

在使用自定义标签训练 ner 时，我以完全相同的方式创建了一个 .json 文件，但使用了 example 中所述的我自己的数据。 然后我尝试使用以下命令将它（训练/开发）转换为训练所需的二进制格式：

python -m spacy convert train.json ./ -t spacy

确实导致创建了 2 个文件。
我在启动训练过程时遇到的错误：

[E923] It looks like there is no proper sample data to initialize the Model of component 'ner'. To check your input data paths and annotation, run: python -m spacy debug data config.cfg

调试命令输出相同。

Answer 1

问题在于存在重叠的实体。每个单词应该只有一个标签。

问题的解决方法可以是（代码来自spacy_convert_script）：

import srsly
import spacy
for f in ["train.json", "dev.json"]:
nlp = spacy.blank("en")
db = DocBin()
for text, annot in srsly.read_json(f):
    doc = nlp.make_doc(text)
    ents = []
    try:
        for start, end, label in annot["entities"]:
            span = doc.char_span(start, end, label=label)
            if span is None:
                msg = f"Skipping entity [{start}, {end}, {label}] in the following text because the character span '{doc.text[start:end]}' does not align with token boundaries:\n\n{repr(text)}\n"
                warnings.warn(msg)
            else:
                ents.append(span)
        doc.ents = ents
        db.add(doc)
    except:
        print(doc.text, ents) #see which texts cause the problem
        continue
db.to_disk(f.split('.')[0]+'.spacy')

这只会导致跳过导致问题的文本。选择重叠实体之一：

        try:
        x = 0
        for start, end, label in annot["entities"]:
            span = doc.char_span(start, end, label=label)
            if span is None:
                msg = f"Skipping entity [{start}, {end}, {label}] in the following text because the character span '{doc.text[start:end]}' does not align with token boundaries:\n\n{repr(text)}\n"
                warnings.warn(msg)
            else:
                if start > x and end > x:
                    x = end
                    ents.append(span)