如何从管道中提取词汇

Question

我可以通过以下方式从 CountVecotizerModel 中提取词汇

fl = StopWordsRemover(inputCol="words", outputCol="filtered")
df = fl.transform(df)
cv = CountVectorizer(inputCol="filtered", outputCol="rawFeatures")
model = cv.fit(df)

print(model.vocabulary)

上面的代码将打印带有索引的词汇列表，因为它是 ids。

现在我已经创建了上述代码的管道，如下所示：

rm_stop_words = StopWordsRemover(inputCol="words", outputCol="filtered")
count_freq = CountVectorizer(inputCol=rm_stop_words.getOutputCol(), outputCol="rawFeatures")

pipeline = Pipeline(stages=[rm_stop_words, count_freq])
model = pipeline.fit(dfm)
df = model.transform(dfm)

print(model.vocabulary) # This won't work as it's not CountVectorizerModel

它会抛出以下错误

print(len(model.vocabulary))

AttributeError: 'PipelineModel' 对象没有属性 'vocabulary'

那么如何从管道中提取Model属性呢？

Answer 1

与任何其他阶段属性一样，提取stages：

stages = model.stages

找到你感兴趣的（-s）：

from pyspark.ml.feature import CountVectorizerModel

vectorizers = [s for s in stages if isinstance(s, CountVectorizerModel)]

并获得所需的字段：

[v.vocabulary for v in vectorizers]