查找文件中所有重复的模式

Question

我有一个文件，其中包含一组几千个独特的单词/术语。 它看起来像：

high school teacher
high school student
library
pencil stand
college professor
college graduate

我需要所有重复模式的列表，所以在这种情况下，我需要以下结果：

high
school
high school
college

在 unix/vim 中有什么方法可以实现吗？

要求的补充说明：

问。重复必须在一行上，还是可以分成几行？

• 理想情况下，每个模式都应该在一个新行中

问。模式是否都是单词序列（一个或多个单词）

• 是的，它们都是单词序列

问。一行内的间距重要吗？大写？标点符号？

• 空格和标点符号都算作模式的一部分。我们可以忽略大小写

即。

• School == School != school
• this pat.tern == this pat.tern != this pattern

Answer 1

这对我有用（脚本放在script.awk 文件中）：

{
    for (i = 1; i <= NF; i++)
    {
        count[$i]++
        sequence = $i
        for (j = i + 1; j <= NF; j++)
        {
            sequence = sequence " " $j
            count[sequence]++
        }
    }
}
END {
    for (i in count)
    {
        if (count[i] > 1)
           print i
    }
}

“每一行”代码在行上构建单词序列并使用这些序列来计算序列。 END 块循环遍历序列，打印计数超过 1 的那些（因此重复了单词序列）。

给定（扩展的）数据文件（称为data）：

high school teacher
high school student
library
pencil stand
college professor
college graduate
coelacanths are ancient fish
coelacanths are ancient but still alive
coelacanths are ancient and long lived
coelacanths are ancient and can live to be 100 years old
coelacanths are ancient living fossils
coelacanths can live to be ancient
coelacanths are long-lived
coelacanths are slow to mature
coelacanths are denizens of the deep sea
coelacanths can be found off Africa and Indonesia

awk -f script.awk data | sort 的输出为：

ancient
ancient and
and
are
are ancient
are ancient and
be
can
can live
can live to
can live to be
coelacanths
coelacanths are
coelacanths are ancient
coelacanths are ancient and
coelacanths can
college
high
high school
live
live to
live to be
school
to
to be

数据仔细有一些更长的重复序列，最多四个单词；更长的单词序列将被同样有效地跟踪。