使用java从文本文件中删除重复行

Question

我可以逐行读取文本文件直到分隔符 -- 并将这些行写入新文件吗？

然后我想读取两个分隔符 -- 之间的下一行，并将它们与前面的行进行比较。

如果三行或更多行重复，则不要将它们写入文件。

以此类推直到结束。

 public void removeDuplicateErr(String data) throws IOException {
    String contents = new String(Files.readAllBytes(Paths.get(data)));
    String[] blocks = contents.split("--");

    String fileName = "output.txt";
    PrintWriter pw = new PrintWriter(fileName);

    int count = 0;
    int count1 = 0;

    for (String block : blocks) {
        boolean flag = false;
        if(count > 0) {
            String contents1 = new String(Files.readAllBytes(Paths.get(fileName)));
            String[] blocks1 = contents1.split("--");
            for(String block1 : blocks1) {
                BufferedReader br1 = new BufferedReader(new StringReader(block1));
                String line1 = br1.readLine();
                while (line1 != null) {
                    BufferedReader br2 = new BufferedReader(new StringReader(block));
                    String line2 = br2.readLine();
                    while (line2 != null) {
                        if(line1.equals(line2)) {
                            count1++;
                            if(count1 >= 3) {
                                flag = true;
                                break;
                            }
                        }
                        line2 = br2.readLine();
                    }
                    line1 = br1.readLine();
                }
                if (!flag) {
                    pw.print(block);
                    pw.print("--");
                    pw.flush();
                }
            }
        }
        if(count < 1) {
            pw.print(block);
            pw.print("--");
            pw.flush();
        }
        count++;
    }
    pw.close();
}

 

输入样本

test 1
test 2
test 3
test 4
test 5
--
test 6
test 2
test 3
test 4
test 12
--
test 8
test 9 
test 10 
test 11 
test 12
--
test 1
test 3
test 4 
test 21
test 22
--
test 1
test 2
test 3
test 4 
test 5
--
test 50
test 51
test 52 
test 53
test 54 
test 55
--
test 53
test 54
test 55
test 56
test 57

预期结果

test 1
test 2
test 3
test 4
test 5
--
test 8 
test 9 
test 10 
test 11 
test 12
--
test 50 
test 51 
test 52
test 53
test 54
test 55

Answer 1

很棒的起始代码（在您的原始版本中）。

(!) 免责声明： 在有人已经回答之后更改问题中的代码不是很支持。当您使用给定的答案来改进您的代码时，情况会更糟。

你的代码/问题的变化，迫使所有答案也改变他们的解决方案。

---

问题

“重复”块的标准是：

[该块的] 3 行或更多行 [出现在其他块中]

（备注：我在括号“[]”内添加了一些规范）

配料配方

• 将单个输入文件作为一个整体读入单个缓冲区 String contents = new String(Files.readAllBytes(Paths.get("/path/to/file")));
• 将内容提取到块（由--分隔）使用String[] blocks = contents.split("--")
• 逐块循环：for (block in blocks) { blocks_out.append( deduplicateFrom(block, blocks); }
• 将生成的块写回文件（记得在每个块之后附加分隔符）

现在关键部分详细说明： blocks_out.append( deduplicateFrom(block, blocks);

• 在循环内部将当前块与其他块进行比较
• 比较可以根据您的标准进行测试（重复 3 行或更多行）
• 如果未找到重复项，则将（去重的）块添加到结果中

你的程序可以记录什么

以下是控制台上打印的操作：

Split input into 7 blocks. Comparing them for duplicates in others. 
    comparing block 0
        duplicate lines of block 0 in 1: 3
    rejecting 'duplicated' block 0
    comparing block 1
        duplicate lines of block 1 in 0: 3
    rejecting 'duplicated' block 1
    comparing block 2
        duplicate lines of block 2 in 0: 0
        duplicate lines of block 2 in 1: 1
        duplicate lines of block 2 in 3: 0
        duplicate lines of block 2 in 4: 0
        duplicate lines of block 2 in 5: 0
        duplicate lines of block 2 in 6: 0
    adding 'unique' block 2
    comparing block 3
        duplicate lines of block 3 in 0: 3
    rejecting 'duplicated' block 3
    comparing block 4
        duplicate lines of block 4 in 0: 5
    rejecting 'duplicated' block 4
    comparing block 5
        duplicate lines of block 5 in 0: 0
        duplicate lines of block 5 in 1: 0
        duplicate lines of block 5 in 2: 0
        duplicate lines of block 5 in 3: 0
        duplicate lines of block 5 in 4: 0
        duplicate lines of block 5 in 6: 3
    rejecting 'duplicated' block 5
    comparing block 6
        duplicate lines of block 6 in 0: 0
        duplicate lines of block 6 in 1: 0
        duplicate lines of block 6 in 2: 0
        duplicate lines of block 6 in 3: 0
        duplicate lines of block 6 in 4: 0
        duplicate lines of block 6 in 5: 3
    rejecting 'duplicated' block 6
Total 'unique' blocks in output: 1

我能识别的唯一唯一块有数字：8 到 12。其中只有 12 从之前的块中以 repeated 的形式出现（1 个重复行）。

在 SO 或网络上研究所有成分

• How to read a file into string in java?

对于第一个项目符号/成分，依此类推。

Answer 2

以下是实施建议：

public class BlockFilter {

    public void removeDuplicateErr(String data) throws IOException {
        String contents = new String(Files.readAllBytes(Paths.get(data)));
        String fileName = "output.txt";
        Files.writeString(Paths.get(fileName), getFilteredContent(contents));
        // TODO this method is not yet tested (but getFilteredContent() below)
    }

    public String getFilteredContent(String fileContent) {
        return getValidBlocks(fileContent).stream()
                .map(Block::getAsString)
                .collect(Collectors.joining("\n--"));
    }

    private Collection<Block> getValidBlocks(String fileContent) {
        // create a list of Blocks...
        List<Block> allBlocks = Stream.of(fileContent.split("--"))
                .map(Block::new)
                .collect(Collectors.toList());

        // ...and collect all, that are valid
        return allBlocks.stream()
            .filter(block -> isBlockValid(allBlocks, block))
            .collect(Collectors.toList());
    }

    private boolean isBlockValid(List<Block> allBlocks, Block block) {
        for (Block otherBlock : allBlocks) {
            if (otherBlock == block) {
                // we've reached 'block' and did not hit the invalid criterion so far
                return true;
            } else if (block.countEqualLines(otherBlock) >= 3) {
                // the block is invalid
                return false;
            }
        }

        throw new IllegalArgumentException("Argument 'block' must be contained in 'allBlocks'!");
    }

    private static class Block {

        private List<String> lines;

        private Block(String block) {
            Objects.requireNonNull(block);
            lines = Stream.of(block.split("\n")).collect(Collectors.toList());
        }

        private long countEqualLines(Block otherBlock) {
            return otherBlock.lines.stream()
                    .filter(line -> lines.contains(line))
                    .count();
        }

        private String getAsString() {
            return lines.stream().collect(Collectors.joining("\n"));
        }
    }
}

至少通过以下单元测试：;-)

    class BlockFilterTest {
    
        @org.junit.jupiter.api.Test
        void getValidBlocks() {
    
            // Java 15 Multi-line String
            String fileContent = """
                    test 1
                    test 2
                    test 3
                    test 4
                    test 5
                    --
                    test 6
                    test 2
                    test 3
                    test 4
                    test 12
                    --
                    test 8
                    test 9
                    test 10
                    test 11
                    test 12
                    --
                    test 1
                    test 3
                    test 4
                    test 21
                    test 22
                    --
                    test 1
                    test 2
                    test 3
                    test 4
                    test 5
                    --
                    test 50
                    test 51
                    test 52
                    test 53
                    test 54
                    test 55
                    --
                    test 53
                    test 54
                    test 55
                    test 56
                    test 57""";
    
            String expected = """
                    test 1
                    test 2
                    test 3
                    test 4
                    test 5
                    --
                    test 8
                    test 9
                    test 10
                    test 11
                    test 12
                    --
                    test 50
                    test 51
                    test 52
                    test 53
                    test 54
                    test 55""";
    
            assertEquals(expected, new BlockFilter().getFilteredContent(fileContent));
        }
    }

注意，由于 BlockFilter 没有状态，所有方法也可以是静态的。