Java 从字符串数组的文件搜索中删除重复项 [0]答案

【问题标题】：Java Remove Duplicates from file search for String Array [0]Java 从字符串数组的文件搜索中删除重复项 [0]
【发布时间】：2015-06-10 23:39:34
【问题描述】：

我有一个长文本文件。

现在我将从文件中删除重复项。问题是搜索参数是列表中的第一个单词，用“：”分割

例如：

文件行：

11234567:229283:29833204:2394803 11234567:4577546765:655776:564456456 43523:455543:54335434:53445 11234567:43455:544354:5443

现在我会在这里：

11234567:229283:29833204:2394803 43523:455543:54335434:53445

我需要从重复项中获取第一行，其他将被忽略。

我试过这个：

Set<String> lines11;
try (BufferedReader reader11 = new BufferedReader(new FileReader("test.txt"))) {
    lines11 = new HashSet<>(10000); // maybe should be bigger
    String line11;
    while ((line11 = reader11.readLine()) != null) {
        lines11.add(line11);
    }
} // maybe should be bigger
try (BufferedWriter writer11 = new BufferedWriter(new FileWriter("test.txt"))) {
    for (String unique : lines11) {
        writer11.write(unique);
        writer11.newLine();
    }
}

这是有效的，但只有在复制完整行时才会删除。

如何更改它，使其在每行中查找第一个单词并在此处检查重复项；当没有找到重复时，保存完整的行；如果重复则忽略该行？

【问题讨论】：

标签： java file collections buffered reader

【解决方案1】：

您需要维护一个Set<String>，它只包含每行的第一个单词。

List<String> lines11;
Set<String> dups;
try (BufferedReader reader11 = new BufferedReader(new FileReader("test.txt"))) {
    lines11 = new ArrayList<>();
    dups = new HashSet<>();
    String line11;
    while ((line11 = reader11.readLine()) != null) {
        String first = line11.split(":")[0]; // assuming your separator is :
        if (!dups.contains(first)) {
            lines11.add(line11);
            dups.add(first);
        }
    }
}
try (BufferedWriter writer11 = new BufferedWriter(new FileWriter("test.txt"))) {
    for (String unique : lines11) {
        writer11.write(unique);
        writer11.newLine();
    }
}

【讨论】：

【解决方案2】：

我将写关于添加到列表的部分使用HashMap

    String tmp[] = null;
    HashMap<String, String> lines = new HashMap<String, String>();
    String line11 = "";

    while ((line11 = reader11.readLine()) != null) {
        tmp = line11.split(":");
        if(!lines.containsKey(tmp[0])){
            lines.put(tmp[0], line11);
        }
    }

所以循环只会添加唯一的行，使用第一个单词作为键

【讨论】：

【解决方案3】：

    You can add the data in list and take one more set in which you will add first word in that set and try add every time first of new line if it is in set, then it will not be added and return false. On that basis you can add data in list or directly in you new bufferreader.


List<String> lines11;
     Set<String> uniqueRecords;
                try (BufferedReader reader11 = new BufferedReader(new FileReader("test.txt"))) {
                    lines11 = new ArrayList<>(); // no need to give size it will increase dynamically
    uniqueRecords = new HashSet<>();
                    String line11;
                    while ((line11 = reader11.readLine()) != null) {
                           String firstWord = line11.substring(0, firstWord.firstIndexOf(" "));
                           if(uniqueRecords.add(firstWord )){
                               lines11.add(line11);
                                  }



                    }
                } // maybe should be bigger
                try (BufferedWriter writer11 = new BufferedWriter(new FileWriter("test.txt"))) {
                    for (String unique : lines11) {
                        writer11.write(unique);
                        writer11.newLine();

                    }
                }

【讨论】：