如何获得具有多个值的相同名称在Python中获得唯一结果

Question

我有一个比较大的 csv 文件来比较我的 txt 文件的 URL

如何获得具有多个值的相同名称在 Python 中获得唯一结果，有没有办法更好地比较两个文件的速度？因为它的最小 csv 文件大小为 1 gb

file1.csv

[01/Nov/2019:09:54:26 +0900] ","","102.12.14.22","34.122.104.106","200","CONNECT","http://www.amazon.com/asdd/asd/","555976","1508"
[01/Nov/2019:09:54:26 +0900] ","","164.16.37.75","52.222.194.116","200","CONNECT","http://www.google.com:443","555976","1508"
[01/Nov/2019:09:54:26 +0900] ","","167.27.14.62","34.122.104.106","200","CONNECT","http://www.amazon.com/asdd/asd/","555976","1508"
[01/Nov/2019:09:54:26 +0900] ","","192.10.77.95","21.323.12.96","200","CONNECT","http://www.wakers.com/sg/wew/","555976","1508"
[01/Nov/2019:09:54:26 +0900] ","","167.27.14.62","34.122.104.106","200","CONNECT","http://www.amazon.com/asdd/asd/","555976","1508"
[01/Nov/2019:09:54:26 +0900] ","","197.99.94.32","34.122.104.106","200","CONNECT","http://www.amazon.com/asdd/asd/","555976","1508"
[01/Nov/2019:09:54:26 +0900] ","","157.87.34.72","34.122.104.106","200","CONNECT","http://www.amazon.com/asdd/asd/","555976","1508"

file2.txt

1 www.amazon.com shop
1 wakers.com shop

脚本：

import csv
with open("file1.csv", 'r') as f: 
    reader = csv.reader(f)
    for k in reader:
        ko = set()
        srcip = k[2]
        url = k[6]
        lines = url.replace(":443", "").replace(":8080", "")
        war = lines.split("//")[-1].split("/")[0].split('?')[0]
        ko.add((war,srcip))
        for to in ko:
            with open("file2.txt", "r") as f:
                all_val = set()
                for i in f:
                    val = i.strip().split(" ")[1]
                    if val in to[0]:
                        all_val.add(to)
                for ki in all_val:
                  print(ki)

我的输出：

('www.amazon.com', '102.12.14.22')
('www.amazon.com', '167.27.14.62')
('www.wakers.com', '192.10.77.95')
('www.amazon.com', '167.27.14.62')
('www.amazon.com', '197.99.94.32')
('www.amazon.com', '157.87.34.72')

如果url相同如何获取，获取具有唯一值的总值

如何得到这样的结果？

amazon.com    102.12.14.22 
              167.27.14.62 
              197.99.94.32
              157.87.34.72
wakers.com    192.10.77.95

Answer 1

简短回答：您不能直接这样做。可以，但性能较差。

CSV 是一种很好的存储格式，但如果您想做类似的事情，您可能希望将所有内容存储在另一个自定义数据文件中。您可以首先将文件解析为只有唯一 ID 而不是长字符串（如 amazon = 0、wakers = 1 等），以提高性能并降低比较成本。

问题是，这些东西对于变量 csv、内存映射或从您的 csv 构建数据库可能也很糟糕（并且在数据库上进行更改，仅在需要时转储 csv）

查看：How do quickly search through a .csv file in Python 以获得更完整的答案。

Answer 2

问题解决方法

import csv
import re


def possible_urls(filename, category, category_position, url_position):
  # Here we will read a txt file to create a list of domains, that could correspond to shops
  domains = []
  with open(filename, "r") as file:
      file_content = file.read().splitlines()
  for line in file_content:
      info_in_line = line.split(" ")
      # Here i use a regular expression, to prase domain from url.
      domain = re.sub('www.', '', info_in_line[url_position])
      if info_in_line[category_position] == category:
          domains.append(domain)
  return domains


def read_from_csv(filename, ip_position, url_position, possible_domains):
  # Here we will create a dictionary, where will 
  # all ips that this domain can have.
  # Dictionary will look like this:
  # {domain_name: [list of possible ips]}

  domain_ip = {domain: [] for domain in possible_domains}
  with open(filename, 'r') as f:
      reader = csv.reader(f)
      for line in reader:
          if len(line) < max(ip_position, url_position):
              print(f'Not enough items in line {line}, to obtain url or ip')
              continue
          ip = line[ip_position]
          url = line[url_position]
          # Using python regular expression to get a domain name
          # from url.
          domain = re.search('//[w]?[w]?[w]?\.?(.[^/]*)[:|/]', url).group(1)
          if domain in domain_ip.keys():
              domain_ip[domain].append(ip)
  return domain_ip


def print_fomatted_result(result):
  # Prints formatted result
  for shop_domain in result.keys():
      print(f'{shop_domain}: ')
      for shop_ip in result[shop_domain]:
          print(f'              {shop_ip}')


def create_list_of_shops():
  # Function that first creates a list of possible domains, and
  # then read ip for that domains from csv
  possible_domains = possible_urls('file2.txt', 'shop', 2, 1)
  shop_domains_with_ip = read_from_csv('file1.csv', 2, 6, possible_domains)
  # Display result, we get in previous operations
  print(shop_domains_with_ip)
  print_fomatted_result(shop_domains_with_ip)


create_list_of_shops()

输出

域是键的 ip 字典，因此您可以通过提供该域的名称来获取域的所有可能 ip：

{'amazon.com': ['102.12.14.22', '167.27.14.62', '167.27.14.62', '197.99.94.32', '157.87.34.72'], 'wakers.com': ['192.10.77.95']}

amazon.com: 
              102.12.14.22
              167.27.14.62
              167.27.14.62
              197.99.94.32
              157.87.34.72
wakers.com: 
              192.10.77.95

正则表达式

您可以从解决方案中学到的一个非常有用的东西是正则表达式。正则表达式是允许您以非常方便的方式从行中过滤或检索信息的工具。它还大大减少了代码量，使代码更具可读性和安全性。 让我们考虑一下您从字符串中删除端口的代码，并考虑如何用正则表达式替换它。

        lines = url.replace(":443", "").replace(":8080", "")

以这种方式替换端口很容易受到攻击，因为您永远无法确定 url 中实际上可以包含哪些端口号。如果出现端口号 5460 或端口号 1022 等怎么办？对于每个这样的端口，您将添加新的替换，很快您的代码将看起来像这样

lines = url.replace(":443", "").replace(":8080", "").replace(":5460","").replace(":1022","")...

不是很可读。但是通过常规表达，您可以描述一种模式。好消息是我们实际上知道带有端口号的 url 模式。他们都看起来像这样： :some_digits。因此，如果我们知道模式，我们可以用正则表达式来描述它，并告诉 python 找到所有匹配它并替换为空字符串''： re.sub(':\d+', '', url) 它告诉python正则表达式引擎： 查找字符串url 中所有位于: 之后的数字，并将它们替换为空字符串。与使用替换链的解决方案相比，此解决方案更短、更安全且更具可读性，因此我建议您阅读一下它们。学习正则表达式的好资源是 this site。在这里你可以test你的正则表达式。

代码中正则表达式的解释

re.sub('www.', '', info_in_line[url_position])

在字符串info_in_line[url_position] 中查找所有www. 并将其替换为空字符串。

re.search('www.(.[^/]*)[:|/]', url).group(1)

让我们把它分成几部分：

1. [^/] - 这里可以是除/ 之外的所有内容
2. (.[^/]*) - 这里我使用了match group。它告诉引擎我们感兴趣的解决方案将在哪里。
3. [:|/] - 这意味着可以留在那个地方的角色。长话短说：捕获组后可能是: 或（|）/。 这么总结。正则表达式可以用如下文字表示： 查找所有以www. 开头并以: 或\ 结尾的子字符串，并将它们之间的所有内容返回给我。 group(1) - 表示获得第一场比赛。

希望答案会有所帮助！

Answer 3

如果您使用 URL 作为字典中的键，并将您的 IP 地址设置为字典的元素，那会达到您的预期吗？

my_dict = {
    'www.amazon.com' = {
        '102.12.14.22',
        '167.27.14.62',
        '197.99.94.32',
        '157.87.34.72',
    },
    'www.wakers.com' = {'192.10.77.95'},
}

Answer 4

## I have used your code & Pandas to get your desired output
## Copy paste the code & execute to get the result

import csv

url_dict = {}

## STEP 1: Open file2.txt to get url names
with open("file2.txt", "r") as f:      
    for i in f:                      
        val = i.strip().split(" ")[1]       
        url_dict[val] = []      

## STEP 2: 2.1 Open csv file 'file1.csv' to extract url name & ip address
##         2.2 Check if url from file2.txt is available from the extracted url from 'file1.csv'
##         2.3 Create a dictionary with the matched url & its ip address 
##         2.4 Remove duplicates in ip addresses from same url          
with open("file1.csv", 'r') as f: ## 2.1
    reader = csv.reader(f)    

    for k in reader:
        #ko = set()        
        srcip = k[2]
        #print(srcip)
        url = k[6]
        lines = url.replace(":443", "").replace(":8080", "")
        war = lines.split("//")[-1].split("/")[0].split('?')[0]  

        for key, value in url_dict.items(): 
            if key in war:                   ## 2.2
            url_dict[key].append(srcip)   ## 2.3          

## 2.4
for key, value in url_dict.items():    
    url_dict[key] = list(set(value))

## STEP 3: Print dictionary output to .TXT file 
file3 = open('output_text.txt', 'w')    
for key, value in url_dict.items():                  
    file3.write('\n' + key + '\n')            
    for item in value:   
        file3.write(' '*15 + item + '\n')

file3.close()