我的标题在我的 txt 文件的第一列。我想创建一个 Pandas DF

Question

文本文件中的示例数据

[User]
employeeNo=123
last_name=Toole
first_name=Michael
language=english
email = michael.toole@123.ie
department=Marketing
role=Marketing Lead
[User]
employeeNo=456
last_name= Ronaldo
first_name=Juan
language=Spanish
email=juan.ronaldo@sms.ie
department=Data Science
role=Team Lead
Location=Spain
[User]
employeeNo=998
last_name=Lee
first_name=Damian
language=english
email=damian.lee@email.com
[User]

想知道是否有人可以帮助我，您可以在上面查看我的示例数据集。我想做的（请告诉我是否有更有效的方法）是循环遍历第一列，并且无论出现唯一 ID 列表的任何位置（例如 first_name、last_name、role 等）都将相应行中的值附加到该列表并执行每个唯一 ID 的操作，以便我留下以下内容。 我读过关于多索引的文章，我不确定这是否是一个更好的解决方案，但我无法让它工作（我对 python 很陌生）

enter image description here

# Define a list of selected persons
selectedList = textFile
# Define a list of searching person
searchList = ['uid']
# Define an empty list
foundList = []

# Iterate each element from the selected list
for index, sList in enumerate(textFile):
  # Match the element with the element of searchList
  if sList in searchList:
    # Store the value in foundList if the match is found
    foundList.append(selectedList[index])

Answer 1

您有一个文本文件，其中每条记录都以[User] 行开头，数据行具有key=value 格式。我知道没有模块能够自动处理它，但很容易手动解析它。代码可能是：

with open('file.txt') as fd:
    data = []                          # a list of records
    for line in fd:
        line = line.strip()            # strip end of line
        if line == '[User]':           # new record
            row = {}                   # row will be a key: value dict
            data.append(row)
        else:
            k,v = line.split('=', 1)   # split on the = character
            row[k] = v

df = pd.DataFrame(data)                # list of key: value dicts => dataframe

通过显示的示例数据，我们得到：

  employeeNo last_name first_name language                 email     department            role                 email Location
0        123     Toole    Michael  english   michael.toole@123.ie     Marketing  Marketing Lead                   NaN      NaN
1        456   Ronaldo       Juan  Spanish                    NaN  Data Science       Team Lead   juan.ronaldo@sms.ie    Spain
2        998       Lee     Damian  english                    NaN           NaN             NaN  damian.lee@email.com      NaN

Answer 2

我确信有一种更优化的方法可以做到这一点，但它是获取一个唯一的行名列表，这次在循环过程中提取它们并将它们组合到一个新的数据帧中。最后，使用所需的列名对其进行更新。

import pandas as pd
import numpy as np
import io

data = '''
[User]
employeeNo=123
last_name=Toole
first_name=Michael
language=english
email=michael.toole@123.ie
department=Marketing
role="Marketing Lead"
[User]
employeeNo=456
last_name= Ronaldo
first_name=Juan
language=Spanish
email=juan.ronaldo@sms.ie
department="Data Science"
role=Team Lead
Location=Spain
[User]
employeeNo=998
last_name=Lee
first_name=Damian
language=english
email=damian.lee@email.com
[User]
'''

df = pd.read_csv(io.StringIO(data), sep='=', comment='[', header=None)

new_cols = df[0].unique()
new_df = pd.DataFrame()
for col in new_cols:
    tmp = df[df[0] == col]
    tmp.reset_index(inplace=True)
    new_df = pd.concat([new_df, tmp[1]], axis=1)
new_df.columns = new_cols
new_df['User'] = None
new_df = new_df[['User','employeeNo','last_name','first_name','language','email','department','role','Location']]

new_df
    User    employeeNo  last_name   first_name  language    email   department  role    Location
0   None    123     Toole   Michael     english     michael.toole@123.ie    Marketing   Marketing Lead  Spain
1   None    456     Ronaldo     Juan    Spanish     juan.ronaldo@sms.ie     Data Science    Team Lead   NaN
2   None    998     Lee     Damian  english     damian.lee@email.com    NaN     NaN     NaN

Answer 3

根据之前版本偏移值的测试重写

import pandas as pd
# Revised from previous answer - ensures key value pairs are contained to the same
# record - previous version assumed the first record had all the expected keys - 
# inadvertently assigned (Location) value of second record to the first record 
# which did not have a Location key 
# This version should perform better - only dealing with one single df
#  - and using pandas own pivot() function

textFile = 'file.txt'
filter = '[User]'

# Decoration - enabling a check and balance - how many users are we processing?
textFileOpened = open(textFile,'r')
initialRead = textFileOpened.read()
userCount = initialRead.count(filter)  # sample has 4 [User] entries - but only three actual unique records
print ('User Count {}'.format(userCount))

# Create sets so able to manipulate and interrogate
allData = []
oneRow = []
userSeq = 0

#Iterate through file - assign record key and [userSeq] Key to each pair
with open(textFile, 'r') as fp:
    for fileLineSeq, line in enumerate(fp):
        if filter in str(line):
            userSeq = userSeq + 1 # Ensures each key value pair is grouped
        else: userSeq = userSeq
        oneRow = [fileLineSeq, userSeq, line]
        allData.append(oneRow)

df = pd.DataFrame(allData)

df.columns = ['FileRow','UserSeq','KeyValue']  # rename columns
userSeparators = df[df['KeyValue'] == str(filter+'\n') ].index # Locate [User Records]
df.drop(userSeparators, inplace = True) # Remove [User] records
df = df.replace(' = ' ,  '=' , regex=True ) # Input data dirty - cleaning up
df = df.replace('\n' ,  '' , regex=True ) # remove the new lines appended during the list generation

# print(df) # Test as necessary here

# split KeyValue column into two
df[['Key', 'Value']] = df.KeyValue.str.split('=', expand=True)
# very powerful function - convert to table
df = df.pivot(index='UserSeq', columns='Key', values='Value') 
print(df)

结果

User Count 4
Key     Location    department                 email employeeNo first_name language last_name            role
UserSeq                                                                                                      
1            NaN     Marketing  michael.toole@123.ie        123    Michael  english     Toole  Marketing Lead
2          Spain  Data Science   juan.ronaldo@sms.ie        456       Juan  Spanish   Ronaldo       Team Lead
3            NaN           NaN  damian.lee@email.com        998     Damian  english       Lee             NaN