从 Java 中的字符串中删除停用词

Question

我有一个包含很多单词的字符串，我有一个包含一些停用词的文本文件，我需要从我的字符串中删除这些停用词。 假设我有一个字符串

s="I love this phone, its super fast and there's so much new and cool things with jelly bean....but of recently I've seen some bugs."

删除停用词后，字符串应该是这样的：

"love phone, super fast much cool jelly bean....but recently bugs."

我已经能够做到这一点，但我面临的问题是，当字符串中有相邻的停用词时，它只删除第一个停用词，我得到的结果是：

"love phone, super fast there's much and cool with jelly bean....but recently seen bugs"  

这是我的 stopwordslist.txt 文件： Stopwords

我该如何解决这个问题。这是我到目前为止所做的：

int k=0,i,j;
ArrayList<String> wordsList = new ArrayList<String>();
String sCurrentLine;
String[] stopwords = new String[2000];
try{
        FileReader fr=new FileReader("F:\\stopwordslist.txt");
        BufferedReader br= new BufferedReader(fr);
        while ((sCurrentLine = br.readLine()) != null){
            stopwords[k]=sCurrentLine;
            k++;
        }
        String s="I love this phone, its super fast and there's so much new and cool things with jelly bean....but of recently I've seen some bugs.";
        StringBuilder builder = new StringBuilder(s);
        String[] words = builder.toString().split("\\s");
        for (String word : words){
            wordsList.add(word);
        }
        for(int ii = 0; ii < wordsList.size(); ii++){
            for(int jj = 0; jj < k; jj++){
                if(stopwords[jj].contains(wordsList.get(ii).toLowerCase())){
                    wordsList.remove(ii);
                    break;
                }
             }
        }
        for (String str : wordsList){
            System.out.print(str+" ");
        }   
    }catch(Exception ex){
        System.out.println(ex);
    }

Answer 1

最近的一个项目需要在浏览了一些博客和文章后，从给定的文本或文件中过滤停止/词干和脏话的功能。 创建了一个简单的库来过滤数据/文件并在 Maven 中可用。希望这可以帮助一些人。

https://github.com/uttesh/exude

     <dependency>
        <groupId>com.uttesh</groupId>
        <artifactId>exude</artifactId>
        <version>0.0.2</version>
    </dependency>

Answer 2

似乎你做了一个停止一个停止词被删除一个句子移动到另一个停止词：你需要删除每个句子中的所有停止词。

您应该尝试更改您的代码：

发件人：

for(int ii = 0; ii < wordsList.size(); ii++){
    for(int jj = 0; jj < k; jj++){
        if(stopwords[jj].contains(wordsList.get(ii).toLowerCase())){
            wordsList.remove(ii);
            break;
        }
    }
}

类似于：

for(int ii = 0; ii < wordsList.size(); ii++)
{
    for(int jj = 0; jj < k; jj++)
    {
        if(wordsList.get(ii).toLowerCase().contains(stopwords[jj])
        {
            wordsList.remove(ii);
        }
    }
}

请注意，break 已删除，stopword.contains(word) 更改为 word.contains(stopword)。

Answer 3

你可以像这样使用全部替换功能

String yourString ="I love this phone, its super fast and there's so much new and cool things with jelly bean....but of recently I've seen some bugs."
yourString=yourString.replaceAll("stop" ,"");

Answer 4

尝试将停用词存储在集合中，然后将您的字符串标记为列表。 之后您可以简单地使用“removeAll”来获得结果。

Set<String> stopwords = new Set<>()
//fill in the set with your file

String s="I love this phone, its super fast and there's so much new and cool things with jelly bean....but of recently I've seen some bugs.";
List<String> listOfStrings = asList(s.split(" "));

listOfStrings.removeAll(stopwords);
StringUtils.join(listOfStrings, " ");

不需要 for 循环 - 它们通常意味着问题。

Answer 5

这是一个更优雅的解决方案（恕我直言），仅使用正则表达式：

    // instead of the ".....", add all your stopwords, separated by "|"
    // "\\b" is to account for word boundaries, i.e. not replace "his" in "this"
    // the "\\s?" is to suppress optional trailing white space
    Pattern p = Pattern.compile("\\b(I|this|its.....)\\b\\s?");
    Matcher m = p.matcher("I love this phone, its super fast and there's so much new and cool things with jelly bean....but of recently I've seen some bugs.");
    String s = m.replaceAll("");
    System.out.println(s);

Answer 6

错误是因为您从迭代的列表中删除了元素。 假设您有包含|word0|word1|word2| 的wordsList 如果ii 等于1 并且if 测试为真，则调用wordsList.remove(1);。之后您的列表是|word0|word2|。然后ii 递增并等于2，现在它超过了列表的大小，因此word2 永远不会被测试。

有几种解决方案。例如，您可以将值设置为“”，而不是删除值。或者创建一个特殊的“结果”列表。

Answer 7

试试下面的程序。

String s="I love this phone, its super fast and there's so" +
            " much new and cool things with jelly bean....but of recently I've seen some bugs.";
    String[] words = s.split(" ");
    ArrayList<String> wordsList = new ArrayList<String>();
    Set<String> stopWordsSet = new HashSet<String>();
    stopWordsSet.add("I");
    stopWordsSet.add("THIS");
    stopWordsSet.add("AND");
    stopWordsSet.add("THERE'S");

    for(String word : words)
    {
        String wordCompare = word.toUpperCase();
        if(!stopWordsSet.contains(wordCompare))
        {
            wordsList.add(word);
        }
    }

    for (String str : wordsList){
        System.out.print(str+" ");
    }

输出： 爱手机，它超级快，有很多新的很酷的东西，果冻豆....但最近我看到了一些错误。

Answer 8

下面试试：

   String s="I love this phone, its super fast and there's so much new and cool things with jelly bean....but of recently I've seen some bugs.";
   String stopWords[]={"love","this","cool"};
   for(int i=0;i<stopWords.length;i++){
       if(s.contains(stopWords[i])){
           s=s.replaceAll(stopWords[i]+"\\s+", ""); //note this will remove spaces at the end
       }
   }
   System.out.println(s);

这样你的最终输出将没有你不想要的单词。只需获取数组中的停用词列表并替换为所需的字符串。
我的停用词的输出：

I   phone, its super fast and there's so much new and  things with jelly bean....but of recently I've seen some bugs.

Answer 9

相反，您为什么不使用以下方法。它会更容易阅读和理解：

for(String word : words){
    s = s.replace(word+"\\s*", "");
}
System.out.println(s);//It will print removed word string.

Answer 10

尝试使用replaceAll api 之类的字符串：

String myString = "I love this phone, its super fast and there's so much new and cool things with jelly bean....but of recently I've seen some bugs.";
String stopWords = "I|its|with|but";
String afterStopWords = myString.replaceAll("(" + stopWords + ")\\s*", "");
System.out.println(afterStopWords);

OUTPUT: 
love this phone, super fast and there's so much new and cool things jelly bean....of recently 've seen some bugs.