如何在 Python 中同时读取 ANSI 和 Unicode 编码文件

Question

我正在尝试使用 Python 编写一个函数，其中用户输入根目录和要搜索的关键短语。然后我的函数搜索整个目录以从包含输入的关键短语的文件中查找和输出行。目前，我的脚本能够从使用 ANSI 编码的文件中读取和输出行，但不是 Unicode。请让我知道如何更改我的代码，以便我的脚本可以搜索这两种类型的编码文件。我对 Python 比较陌生，谢谢！

我的 Python 脚本：

import os

def myFunction(rootdir, keyPhrases):

    path = rootdir # Enter the root directory you want to search from

    key_phrases = [keyPhrases] # Enter here the key phrases in the lines you hope to find 
    key_phrases = [i.replace('\n','') for i in key_phrases] #In case an \n is added to the end of the string when the parameter is passed to the function

    # This for loop allows all sub directories and files to be searched
    for (path, subdirs, files) in os.walk(path): 
        files = [f for f in os.listdir(path) if f.endswith('.txt') or f.endswith('.log')] # Specify here the format of files you hope to search from (ex: ".txt" or ".log")
        files.sort() # file is sorted list

        files = [os.path.join(path, name) for name in files] # Joins the path and the name, so the files can be opened and scanned by the open() function

        # The following for loop searches all files with the selected format
        for filename in files:

                # Opens the individual files and to read their lines
                with open(filename) as f:
                    f = f.readlines()

                # The following loop scans for the key phrases entered by the user in every line of the files searched, and stores the lines that match into the "important" array
                for line in f:
                    for phrase in key_phrases: 
                        if phrase in line:
                            print(line)
                            break 

    print("The end of the directory has been reached, if no lines are printed then that means the key phrase does not exist in the root directory you entered.")

Answer 1

在 Windows“Unicode”(UTF16) 编码文件中，前 2 个字节通常是字节顺序标记 (BOM)，值为 0xFF 0xFE。这表示 UTF16 little endian 编码。 “ANSI”（通常是 Windows-1252）文件没有标记。

当您尝试读取 UTF16 文件时，就好像它是使用不同的 8 位编码进行编码一样，例如UTF8、Windows-1252 或 ASCII 您将看到 UnicodeDecodeError 异常，因为 0xFF 在这些编码中不是有效字节（或 UTF-8 的有效起始字节）。

因此，如果您确定文件将采用 UTF-16-LE 或 Windows-1252 编码，那么您可以在文件开头测试 UTF16 BOM 并使用该编码打开文件，如果检测到：

import sys
from codecs import BOM_UTF16_LE

def get_file_encoding(filename, default=None):
    with open(filename, 'rb') as f:
        if f.read(2) == BOM_UTF16_LE:
            return 'utf_16'
        return default if default else sys.getdefaultencoding()

with open(filename, encoding=get_file_encoding(filename, 'windows_1252')) as f:
    for line in f:
        for phrase in key_phrases: 
            if phrase in line:
                print(line)
                break 

另外，您可以考虑使用正则表达式进行短语匹配，而不是遍历可能的短语。