将两个文件与歌曲名称列表进行比较的最简单方法

Question

我有两个歌曲标题列表，每个列表都在一个纯文本文件中，它们是许可歌词文件的文件名 - 我想检查较短的列表标题（针）是否在较长的列表（干草堆）中。脚本/应用程序应返回不在大海捞针中的标题列表。

我更喜欢使用 Python 或 shell 脚本 (BASH)，或者只使用可以处理所需模糊性的可视差异程序。

主要问题是标题需要模糊匹配以解决数据输入错误以及可能的字序问题。

Haystack 示例（注意一些重复和接近重复的行，匹配突出显示）：

Yearn
Yesterday, Today And Forever
Yesterday, Today, Forever
You
You Alone
You Are Here (The Same Power)
You Are Holy
You Are Holy (Prince Of Peace)
You Are Mighty
You Are Mine
You Are My All In All
You Are My Hiding Place
You Are My King (Amazing Love)
You Are Righteous (Hope)
You Are So Faithful
You Are So Good to Me
You Are Worthy Of My Praise
You Have Been Good
You Led Me To The Cross
You Reign
You Rescued Me
You Said
You Sent Your Own
You Set Me Apart (Dwell In Your House)
You alone are worthy (Glory in the highest)
You are God in heaven (let my words be few)
You are always fighting for us (Hallelujah you have overcome)
You are beautiful (I stand in awe)
You are beautiful beyond description
You are mighty
You are my all in all
You are my hiding place
You are my passion
You are still Holy
You are the Holy One (We exalt Your name)
You are the mighty King
You are the mighty warrior
You are the vine
**You chose the cross (Lost in wonder)**
You have shown me favour unending
You hold the broken hearted
You laid aside Your majesty
You said
You're Worthy Of My Praise
You're calling me (Unashamed love)
You're the God of this city
You're the Lion of Judah
You're the word of God the Father (Across the lands)
You've put a new song in my heart
Your Beloved
Your Grace is Enough
Your Great Name We Praise
Your Great Name We Praise-2
Your Light (You Have Turned)
Your Light Is Over Me (His Love)
**Your Love**
**Your Love Is Amazing**
Your Love Is Deep
Your Love Is Deeper - Jesus, Lord of Heaven (Phil Wickham)
Your Love Oh Lord
Your Love Oh Lord (Psalm 36)
Your Love is Extravagant
Your Power (Send Me)
Your blood speaks a better word
Your everlasting love
**Your grace is enough**
**Your grace is enough (Great is Your faithfulness)**
Your mercy is falling
Your mercy taught us how to dance (Dancing generation)
Your voice stills the oceans (nothing is impossible)
Yours Is The Kingdom

针样：

You Are Good (I Want To Scream It Out)
You Are My Strength (In The Fullness)
You Are My Vision O King Of My Heart
You Are The King Of Glory (Hosanna To The Son)
**You Chose The Cross (Lost In Wonder)**
**Your Grace Is Enough (This Is Our God)**
**Your Love Is Amazing Steady And Unchanging**
**Your Love Shining Like The Sun**

请注意，针标题“Your Love Shining Like The Sun”只是“Your Love”的可能匹配项。最好不要匹配失败，因此任何不确定的标题匹配都应该出现在输出中。

comm -1 -3 <(sort haystack.txt) <(sort needle.txt)

没有找到任何匹配项。 diff 或 grep 似乎他们也有同样的问题，不够模糊。 Kdiff3 和 diffnow.com 与手动比较一样快，因为我仍然必须扫描几乎所有匹配项，它们只能处理空格和字母大小写的差异。

来自 prestosoft.com 的ExamDiffPro 看起来很有可能，但它仅适用于 MS Windows，在我使用 WINE 或 VirtualBox 之前，我更喜欢原生 Linux 解决方案。

针实际上是一个 CSV，所以我考虑过使用 LibreOffice 并将其视为数据库并执行 SQL 查询或使用带有 hlookup 或其他东西的电子表格...Another 问题引导我到OpenRefine (formerly google-refine)

似乎这是一个常见的问题类别（它基本上是经常使用 [Levenshtein] 编辑距离计算的“记录链接”），我应该如何处理它？请给点建议？

Answer 1

我在 MySQL 中做了类似的事情。我使用以下代码来定义 Levenshtein 距离和比率函数（我从this question 的答案中得到）：

DROP FUNCTION IF EXISTS `levenshtein`;
CREATE FUNCTION `levenshtein`(s1 text, s2 text) RETURNS int(11) DETERMINISTIC
BEGIN 
    DECLARE s1_len, s2_len, i, j, c, c_temp, cost INT;
    DECLARE s1_char CHAR; 
    DECLARE cv0, cv1 text; 
    SET s1_len = CHAR_LENGTH(s1), s2_len = CHAR_LENGTH(s2), cv1 = 0x00, j = 1, i = 1, c = 0; 
    IF s1 = s2 THEN 
      RETURN 0; 
    ELSEIF s1_len = 0 THEN 
      RETURN s2_len; 
    ELSEIF s2_len = 0 THEN 
      RETURN s1_len; 
    ELSE 
      WHILE j <= s2_len DO 
        SET cv1 = CONCAT(cv1, UNHEX(HEX(j))), j = j + 1; 
      END WHILE; 
      WHILE i <= s1_len DO 
        SET s1_char = SUBSTRING(s1, i, 1), c = i, cv0 = UNHEX(HEX(i)), j = 1; 
        WHILE j <= s2_len DO 
          SET c = c + 1; 
          IF s1_char = SUBSTRING(s2, j, 1) THEN  
            SET cost = 0; ELSE SET cost = 1; 
          END IF; 
          SET c_temp = CONV(HEX(SUBSTRING(cv1, j, 1)), 16, 10) + cost; 
          IF c > c_temp THEN
            SET c = c_temp;
          END IF; 
          SET c_temp = CONV(HEX(SUBSTRING(cv1, j+1, 1)), 16, 10) + 1; 
          IF c > c_temp THEN  
            SET c = c_temp;  
          END IF; 
          SET cv0 = CONCAT(cv0, UNHEX(HEX(c))), j = j + 1; 
        END WHILE; 
        SET cv1 = cv0, i = i + 1; 
      END WHILE; 
    END IF; 
    RETURN c; 
END;

DROP FUNCTION IF EXISTS `levenshtein_ratio`;
CREATE FUNCTION `levenshtein_ratio`(s1 text, s2 text) RETURNS int(11) DETERMINISTIC
BEGIN 
    DECLARE s1_len, s2_len, max_len INT;
    SET s1_len = LENGTH(s1), s2_len = LENGTH(s2); 
    IF s1_len > s2_len THEN  
      SET max_len = s1_len;  
    ELSE  
      SET max_len = s2_len;  
    END IF; 
    RETURN ROUND((1 - LEVENSHTEIN(s1, s2) / max_len) * 100); 
END;

假设您将列表导入为以下两个表：

needle (title VARCHAR)
haystack (title VARCHAR)

然后您可以使用如下查询来比较两个表。

SELECT title, best_match,
    levenshtein_ratio(TRIM(LOWER(title)), TRIM(LOWER(best_match))) AS ratio
FROM (
    SELECT n.title AS title, (
                SELECT h.title
                FROM haystack h
                ORDER BY levenshtein_ratio(TRIM(LOWER(n.title)), TRIM(LOWER(h.title))) DESC
                LIMIT 1
           ) AS best_match
    FROM needle n
) x
ORDER BY ratio DESC

选择一个截止值，低于该值的所有行都没有很好的匹配。如果您想直接处理编辑距离，可以使用 levenshtein() 而不是 levenshtein_ratio()，在本例中是 ORDER BY 到 ASC。

请注意，这对词序差异没有任何特殊规定。此外，如果您的列表很大，比较可能会很慢。

Answer 2

如果你想走 OpenRefine 的方式，最好设置一个本地对帐服务器 - 我推荐reconcile csv。

将您的 haystack 加载到协调服务器中，让 OpenRefine 处理您的 needle 文件并将其发送到协调服务。

协调服务器会为每个提案返回一个分数，以便您可以（我只是以这些分数为例，不要认为它们是理所当然的）：

• 批量接受超过 0.9 的所有内容
• 手动查看 0.8 到 0.9 之间的所有内容
• 丢弃低于 0.8 的所有内容

Answer 3

你可能想看看fuzzywuzzy (https://github.com/seatgeek/fuzzywuzzy)。

from fuzzywuzzy import process
needles = open('needle').read().split("\n")
haystack = open('haystack').read().split("\n")
for a in needles:
    print a + ' -> ',
    print process.extractBests(a, haystack, score_cutoff=90)

提取函数的有用参数是限制、记分器和处理器。

Answer 4

Fuzzywuzzy 可以让你梦想成真：

import csv
from fuzzywuzzy import fuzz

# Grab CSV data: 
with open('needles.csv', 'U') as z:
    reader = csv.reader(z)
    needles = list(reader)

with open('haystack.csv', 'U') as w:
    reader = csv.reader(w)
    haystack = list(reader)

# Calculate matches and append to list
NeedlesNotInHaystack = []

Fuzziness = 80 # ADJUST THIS VALUE TO FINE-TUNE RESULTS 
for x in needle:
    for y in haystack:
        if fuzz.ratio(x,y) > Fuzziness: 
            NeedlesNotInHaystack.append(x)

#Export results to CSV:
with open('Results', 'wb') as csvfile:
    temp = csv.writer(csvfile)
    temp.writerows(NeedlesNotInHaystack)

而不是 fuzz.ratio，你可能会得到更好的结果：

fuzz.token_sort_ratio OR fuzz.token_set_ratio

IMO，this tutorial 是 Python 中模糊匹配的最佳 + 最简洁概述