用千篇一律的短语分割文本

Question

我有一个列表，其中包含一些不可分割的单词，例如

List<String> lookUp = new ArrayList<>();
lookUp.add("New York");
lookUp.add("Big Apple");

对于一个句子，我想将它拆分成单词，但不要拆分我列表中给出的密不可分的单词。举个例子

String sentence = "New York is also called Big Apple";

它应该还给我

["New York", "is", "also", "called", "Big Apple"]

我开始编写一个算法，首先用空格分割句子，然后我做一个循环：对于每个单词，我检查这个单词和它的右邻是否出现在查找列表中，如果是，则一起解析这些单词.

1) 想象一下，我的查找列表还包含包含两个以上单词的密不可分的短语，例如“George W. Bush”-> 我的算法只会查找“George W”。和“W. Bush”，并且不会在查找列表中找到它，因此会将其拆分为 3 个单词。

2) 更重要的问题（您可以忽略问题 1）：是否已经有库甚至 GATE 插件（这样我就不必重新发明轮子了）？德语短语也存在这种情况吗？没找到=(

Answer 1

Java 7 上的另一个实现，它不使用 regular expressions：

    List<String> lookUp = new ArrayList<>();
    lookUp.add("New York");
    lookUp.add("New Jersey");
    lookUp.add("Big Apple");
    lookUp.add("George W. Bush");

    String sentence = "New York is also called Big Apple . New Jersey is located near to New York . George W. Bush doesn't live in New Mexico`";

    String currentPhrase = "";
    List<String> parseResult = new ArrayList<>();

    for (String word : sentence.split("\\s+")) {
        currentPhrase += (currentPhrase.isEmpty() ? "" : " ") + word;
        if (lookUp.contains(currentPhrase)) {
            parseResult.add(currentPhrase);
            currentPhrase = "";
            continue;
        }
        boolean phraseFound = false;
        for (String look : lookUp)
            if (look.startsWith(currentPhrase)) {
                phraseFound = true;
                break;
            }

        if (!phraseFound) {
            parseResult.addAll(Arrays.asList(currentPhrase.split("\\s+")));
            currentPhrase = "";
        } 
    }

    System.out.println(parseResult);

输出是：

[New York, is, also, called, Big Apple, ., New Jersey, is, located, near, to, New York, ., George W. Bush, doesn't, live, in, New, Mexico]