Java中UTF-8编码中2GB txt文件中每个单词的词频

Question

我正在做一个项目，在那里我需要找出每个单词在超过 1 亿个孟加拉语单词的大型语料库中出现的频率。文件大小约为 2GB。我实际上需要频率计数最频繁的 20 个单词和最不频繁的 20 个单词。我在 PHP 中完成了相同的代码，但花了很长时间（代码在一周后仍在运行）。因此，我试图在 Java 中做到这一点。

在这段代码中，它应该如下工作，

-从语料库中读取一行nahidd_filtered.txt

-使用空格分割

• 对于每个吐出的单词，读取整个频率文件freq3.txt

  • 如果找到单词，则增加频率计数并存储在该文件中

  • else count = 1（新词）并将频率计数存储在该文件中

我尝试使用循环从 nahidd_filtered.txt 语料库中读取文本块，并且频率的单词存储在 freq3.txt 中。 freq3.txt文件存储频率计数是这样的，

Word1 Frequuncy1（中间有一个空格）

Word2 频率2

...........

简单地说，我需要从 UTF-8 编码的大型语料库文件中获取前 20 个最频繁和 20 个最不频繁的单词以及它们的频率计数。请检查代码并建议我为什么这不起作用或任何其他建议。非常感谢。

import java.io.*;
import java.util.*;
import java.util.concurrent.TimeUnit;

public class Main {


private static String fileToString(String filename) throws IOException {
    FileInputStream inputStream = null;
    Scanner reader = null;
    inputStream = new FileInputStream(filename);
    reader = new Scanner(inputStream, "UTF-8");

    /*BufferedReader reader = new BufferedReader(new FileReader(filename));*/
    StringBuilder builder = new StringBuilder();


    // For every line in the file, append it to the string builder
    while (reader.hasNextLine()) {
        String line = reader.nextLine();
        builder.append(line);
    }

    reader.close();
    return builder.toString();
}

public static final String UTF8_BOM = "\uFEFF";

private static String removeUTF8BOM(String s) {
    if (s.startsWith(UTF8_BOM)) {
        s = s.substring(1);
    }
    return s;
}

public static void main(String[] args) throws IOException {

    long startTime = System.nanoTime();
    System.out.println("-------------- Start Contents of file: ---------------------");
    FileInputStream inputStream = null;
    Scanner sc = null;
    String path = "C:/xampp/htdocs/thesis_freqeuncy_2/nahidd_filtered.txt";
    try {
        inputStream = new FileInputStream(path);
        sc = new Scanner(inputStream, "UTF-8");
        int countWord = 0;
        BufferedWriter writer = null;
        while (sc.hasNextLine()) {
            String word = null;
            String line = sc.nextLine();
            String[] wordList = line.split("\\s+");

            for (int i = 0; i < wordList.length; i++) {
                word = wordList[i].replace("।", "");
                word = word.replace(",", "").trim();
                ArrayList<String> freqword = new ArrayList<>();
                String freq = fileToString("C:/xampp/htdocs/thesis_freqeuncy_2/freq3.txt");
                /*freqword = freq.split("\\r?\\n");*/
                Collections.addAll(freqword, freq.split("\\r?\\n"));
                int flag = 0;
                String[] freqwordsp = null;
                int k;
                for (k = 0; k < freqword.size(); k++) {
                    freqwordsp = freqword.get(k).split("\\s+");
                    String word2 = freqwordsp[0];
                    word = removeUTF8BOM(word);
                    word2 = removeUTF8BOM(word2);
                    word.replaceAll("\\P{Print}", "");
                    word2.replaceAll("\\P{Print}", "");
                    if (word2.toString().equals(word.toString())) {

                        flag = 1;
                        break;
                    }
                }

                int count = 0;
                if (flag == 1) {
                    count = Integer.parseInt(freqwordsp[1]);
                }
                count = count + 1;
                word = word + " " + count + "\n";
                freqword.add(word);

                System.out.println(freqword);
                writer = new BufferedWriter(new FileWriter("C:/xampp/htdocs/thesis_freqeuncy_2/freq3.txt"));
                writer.write(String.valueOf(freqword));
            }
        }
        // writer.close();
        System.out.println(countWord);
        System.out.println("-------------- End Contents of file: ---------------------");
        long endTime = System.nanoTime();
        long totalTime = (endTime - startTime);
        System.out.println(TimeUnit.MINUTES.convert(totalTime, TimeUnit.NANOSECONDS));

        // note that Scanner suppresses exceptions
        if (sc.ioException() != null) {
            throw sc.ioException();
        }
    } finally {
        if (inputStream != null) {
            inputStream.close();
        }
        if (sc != null) {
            sc.close();
        }
    }

}

}

Answer 1

首先：

对于每个吐出的单词，读取整个频率文件freq3.txt

别这样！磁盘 IO 操作非常非常慢。您是否有足够的内存将文件读入内存？看来，是的：

String freq = fileToString("C:/xampp/htdocs/thesis_freqeuncy_2/freq3.txt");
Collections.addAll(freqword, freq.split("\\r?\\n"));

如果您确实需要此文件，请加载一次并使用内存。同样在这种情况下，地图（单词到频率）可能比列表更舒服。计算完成后将集合保存在磁盘上。

接下来，你可以bufferize你的输入流，它可能会显着提高性能：

inputStream = new BufferedInputStream(new FileInputStream(path));

并且不要忘记关闭流/读取器/写入器。显式地或使用try-with-resource 语句。

一般来说，代码可能会根据所使用的 API 进行简化。例如：

public class DemoApplication {

    public static final String UTF8_BOM = "\uFEFF";

    private static String removeUTF8BOM(String s) {
        if (s.startsWith(UTF8_BOM)) {
            s = s.substring(1);
        }
        return s;
    }

    private static final String PATH = "words.txt";

    private static final String REGEX = " ";

    public static void main(String[] args) throws IOException {

        Map<String, Long> frequencyMap;
        try (BufferedReader reader = new BufferedReader(new FileReader(PATH))) {
            frequencyMap = reader
                    .lines()
                    .flatMap(s -> Arrays.stream(s.split(REGEX)))
                    .map(DemoApplication::removeUTF8BOM)
                    .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));
        }

        frequencyMap
                .entrySet()
                .stream()
                .sorted(Comparator.comparingLong(Map.Entry::getValue))
                .limit(20)
                .forEach(System.out::println);
    }
}