如何从文本文件中删除重复并包含某些单词的行？

Question

我正在尝试从抓取的数据中删除重复的行和包含某些单词的行。我搜索了各种代码，但它们不起作用:(

这是代码。只有第一部分有效，删除了重复的行：

openFile = open("links.txt", "r") 
writeFile = open("updatedfile.txt", "w") 
#Store traversed lines
tmp = set() 
for txtLine in openFile: 
#Check new line
    if txtLine not in tmp: 
        writeFile.write(txtLine) 
#Add new traversed line to tmp 
        tmp.add(txtLine)         
openFile.close() 
writeFile.close()

sleep(5)

with open("updatedfile.txt", "r") as fp:
    lines = fp.readlines()

with open("updatedfile.txt", "w") as fp:
    for line in lines:
        if line.strip("\n") != "search":
            fp.write(line)

这是 links.txt 文件

https://twitter.com/search?q=%23BTC&src=hashtag_click
https://twitter.com/search?q=%23ADA&src=hashtag_click
https://twitter.com/search?q=%23LTC&src=hashtag_click
https://twitter.com/search?q=%23CAKE&src=hashtag_click
https://twitter.com/Marie62943337
https://twitter.com/Marie62943337
https://twitter.com/Fathur0501
https://twitter.com/Fathur0501
https://twitter.com/BogdanMar93
https://twitter.com/BogdanMar93
https://t.[spaced because body cannot contain short url]co/74ZzkVwa2W
https://t. co/Gv2tyiWfAk

我希望输出是：

https://twitter.com/Marie62943337
https://twitter.com/Fathur0501
https://twitter.com/BogdanMar93

感谢您的帮助。

Answer 1

检查此代码。我认为它有效

with open("test.txt", "r") as fp:
    lines = fp.readlines()
fp.close()

unique = set() 

with open("test.txt", "w") as fp:
    for line in lines:
        if "search" not in line and line not in unique and "twitter.com" in line:
            fp.write(line)
            unique.add(line)

请在下面的评论中分享查询。

Answer 2

也许你想用这个，用'in'：

lines = ['https://twitter.com/search?q=%23CAKE&src=hashtag_click', 'https://twitter.com/Marie62943337']
for line in lines:
    if 'search' not in line:
        print(line)