加入两个 csv 文件（1-N 关系）

Question

csv 文件以制表符分隔

file1.csv：

id_album   name        date
001        Nevermind   24/09/1991
...

file2.csv：

id_song   id_album   name  
001       001        Smells Like Teen Spirit
002       001        In Bloom
...

我想获得这个 output.csv ：

id_album   name        date         songs
001        Nevermind   24/09/1991   001,Smells Like Teen Spirit,002,In Bloom,...

您是否看到了在 Bash（最好）或 Python 中执行此操作的方法？

我的 csv 文件中有很多记录（数百万行）。

编辑

我尝试了 join / sed / awk 但无法管理 1 对 N 关系

Answer 1

发现 awk 语言：

awk -F'[[:space:]][[:space:]]+' 'NR==FNR{ if(NR>1) a[$2]=($2 in a? a[$2]",":"")$1","$3; next}
       FNR==1{ print $0,"songs" }
       $1 in a{ print $0,a[$1] }' file2.csv OFS='\t' file1.csv > output.csv

output.csv 内容：

id_album   name        date songs
001        Nevermind   24/09/1991   001,Smells Like Teen Spirit,002,In Bloom

Answer 2

TL;DR

from io import StringIO
file1 = """id_album,name,date
001,Nevermind,24/09/1991"""

file2 = """id_song,id_album,name
001,001,Smells Like Teen Spirit
002,001,In Bloom"""

df1 = pd.read_csv(StringIO(file1))
df1 = df1.rename(columns={'name':'album_name'})

df2 = pd.read_csv(StringIO(file2))
df2 = df2.rename(columns={'name':'song_name'})


df3 = df1.merge(df2, on='id_album')
df4 = pd.DataFrame(list({album['id_album'].unique()[0]:','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])}.items()), columns=['id_album', 'song_id_name'])

df_want = df1.merge(df4)

[出]：

>>> df_want
   id_album album_name        date                          song_id_name
0         1  Nevermind  24/09/1991  1,Smells Like Teen Spirit,2,In Bloom

---

长期

给定：

>>> from io import StringIO
>>> file1 = """id_album,name,date
... 001,Nevermind,24/09/1991"""

>>> file2 = """id_song,id_album,name
... 001,001,Smells Like Teen Spirit
... 002,001,In Bloom"""

>>> df1 = pd.read_csv(StringIO(file1))
>>> df1 = df1.rename(columns={'name':'album_name'})

>>> df2 = pd.read_csv(StringIO(file2))
>>> df2 = df2.rename(columns={'name':'song_name'})

>>> df1
   id_album album_name        date
0         1  Nevermind  24/09/1991

>>> df2
   id_song  id_album                   name  
0        1         1  Smells Like Teen Spirit
1        2         1                 In Bloom

首先合并id_album列上的2个DataFrame：

>>> df3 = df1.merge(df2, on='id_album')
>>> df3
   id_album album_name        date  id_song                song_name
0         1  Nevermind  24/09/1991        1  Smells Like Teen Spirit
1         1  Nevermind  24/09/1991        2                 In Bloom

现在来点pandas 技巧：

1. First group the rows by the `id_album` column:
2. In each group, get the `id_song` and `song_name` columns and stack them

>> [','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])]
['1,Smells Like Teen Spirit,2,In Bloom']

以类似的方式，从.groupby()获取album_name：

>>> [album['album_name'].unique()[0] for idx, album in df3.groupby(['id_album'])]
['Nevermind']

让我们结合两个groupby 操作：

>>> {album['album_name'].unique()[0]:','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])}
{'Nevermind': '1,Smells Like Teen Spirit,2,In Bloom'}

>>> album2songs = {album['album_name'].unique()[0]:','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])}

将album2songs 放入数据框：

>>> df4 = pd.DataFrame(list(album2songs.items()), columns=['album_name', 'song_id_name'])
>>> df4
  album_name                          song_id_name
0  Nevermind  1,Smells Like Teen Spirit,2,In Bloom

现在加入df1和df4：

>>> df1.merge(df4)
   id_album album_name        date                          song_id_name
0         1  Nevermind  24/09/1991  1,Smells Like Teen Spirit,2,In Bloom

顺便说一句，@RomanPerekhrest awk 解决方案更酷！