删除仅包含不同列的重复行答案

【问题标题】：Removing duplicate line with only a column that is different删除仅包含不同列的重复行
【发布时间】：2018-04-10 21:47:18
【问题描述】：

我有一个文件如下：

ENSG00000087266:I10_35  chr4:2815905-2816105    +   I   B
ENSG00000087266:I10_35  chr4:2815905-2816105    +   I   U
ENSG00000087266:I10_36  chr4:2815955-2816155    +   I   B
ENSG00000087266:I10_36  chr4:2815955-2816155    +   I   U
ENSG00000087266:I16_18  chr4:2826151-2826351    +   I   B
ENSG00000087266:I3_27   chr4:2797290-2797490    +   I   B
ENSG00000087266:I3_28   chr4:2797340-2797540    +   I   B
ENSG00000087266:I3_28   chr4:2797340-2797540    +   I   U
ENSG00000087266:I3_29   chr4:2797390-2797590    +   I   B
ENSG00000087266:I3_30   chr4:2797440-2797640    +   I   B
ENSG00000087266:I3_30   chr4:2797440-2797640    +   I   U
ENSG00000087266:I3_31   chr4:2797490-2797690    +   I   B
ENSG00000087266:I3_38   chr4:2797840-2798040    +   I   B
ENSG00000000003:E1_1    chrX:99894942-99895142  -   E   U
ENSG00000000003:E2_10   chrX:99891638-99891838  -   E   U
ENSG00000000003:E2_11   chrX:99891688-99891888  -   E   U
ENSG00000000003:E2_12   chrX:99891738-99891938  -   E   U
ENSG00000000003:E2_13   chrX:99891788-99891988  -   E   U
ENSG00000000003:E2_14   chrX:99891838-99892038  -   E   U
ENSG00000000003:E2_15   chrX:99891888-99892088  -   E   U
ENSG00000000003:E2_16   chrX:99891938-99892138  -   E   U
ENSG00000000003:E2_1    chrX:99891188-99891388  -   E   U
ENSG00000000003:E2_2    chrX:99891238-99891438  -   E   U

我想删除所有在第 5 列中包含“U”的重复列，以便文件为

ENSG00000087266:I10_35  chr4:2815905-2816105    +   I   B
ENSG00000087266:I10_36  chr4:2815955-2816155    +   I   B
ENSG00000087266:I16_18  chr4:2826151-2826351    +   I   B
ENSG00000087266:I3_27   chr4:2797290-2797490    +   I   B
ENSG00000087266:I3_28   chr4:2797340-2797540    +   I   B
ENSG00000087266:I3_29   chr4:2797390-2797590    +   I   B
ENSG00000087266:I3_30   chr4:2797440-2797640    +   I   B
ENSG00000087266:I3_31   chr4:2797490-2797690    +   I   B
ENSG00000087266:I3_38   chr4:2797840-2798040    +   I   B
ENSG00000000003:E1_1    chrX:99894942-99895142  -   E   U
ENSG00000000003:E2_10   chrX:99891638-99891838  -   E   U
ENSG00000000003:E2_11   chrX:99891688-99891888  -   E   U
ENSG00000000003:E2_12   chrX:99891738-99891938  -   E   U
ENSG00000000003:E2_13   chrX:99891788-99891988  -   E   U
ENSG00000000003:E2_14   chrX:99891838-99892038  -   E   U
ENSG00000000003:E2_15   chrX:99891888-99892088  -   E   U
ENSG00000000003:E2_16   chrX:99891938-99892138  -   E   U
ENSG00000000003:E2_1    chrX:99891188-99891388  -   E   U
ENSG00000000003:E2_2    chrX:99891238-99891438  -   E   U

简而言之，如果除最后一列之外的所有列的行都是重复的，则保留以“B”结尾的行并删除以“U”结尾的重复行

到目前为止，我已经尝试过使用

  while read id; do  sed -i '/"$id"/{/U/d}' file_that_contains_duplicates.txt; done < list_of_duplicate_ids.txt

没有输出任何东西；目前我正在跑步

 while read id; do  awk -v id=$id '$1!=id && $5!="U"' file_that_contains_duplicates.txt; done < list_of_duplicate_ids.txt | sort -u which somehow works but outputs repeated lines and also takes some time.

【问题讨论】：

sed '/U$/d' filename，如果你喜欢 sed。
Sed remove matching lines script的可能重复
@EdMorton：我刚刚添加了一些行来说明我的情况

标签： shell awk duplicates

【解决方案1】：

$ awk '!(seen[$1,$2,$3,$4]++ && ($5 == "U"))' file
ENSG00000087266:I10_35  chr4:2815905-2816105    +   I   B
ENSG00000087266:I10_36  chr4:2815955-2816155    +   I   B
ENSG00000087266:I16_18  chr4:2826151-2826351    +   I   B
ENSG00000087266:I3_27   chr4:2797290-2797490    +   I   B
ENSG00000087266:I3_28   chr4:2797340-2797540    +   I   B
ENSG00000087266:I3_29   chr4:2797390-2797590    +   I   B
ENSG00000087266:I3_30   chr4:2797440-2797640    +   I   B
ENSG00000087266:I3_31   chr4:2797490-2797690    +   I   B
ENSG00000087266:I3_38   chr4:2797840-2798040    +   I   B
ENSG00000000003:E1_1    chrX:99894942-99895142  -   E   U
ENSG00000000003:E2_10   chrX:99891638-99891838  -   E   U
ENSG00000000003:E2_11   chrX:99891688-99891888  -   E   U
ENSG00000000003:E2_12   chrX:99891738-99891938  -   E   U
ENSG00000000003:E2_13   chrX:99891788-99891988  -   E   U
ENSG00000000003:E2_14   chrX:99891838-99892038  -   E   U
ENSG00000000003:E2_15   chrX:99891888-99892088  -   E   U
ENSG00000000003:E2_16   chrX:99891938-99892138  -   E   U
ENSG00000000003:E2_1    chrX:99891188-99891388  -   E   U
ENSG00000000003:E2_2    chrX:99891238-99891438  -   E   U

【讨论】：

【解决方案2】：

在这种情况下，我不知道您所说的“重复”是什么意思，因为您只是删除了所有以“D”结尾的行。那是easily done:

sed '/U$/d'

（匹配以“U”结尾的行，然后是 delete。）

如果您只想要以“B”结尾的行：

sed -n '/B$/p'

（匹配以“B”结尾的行，然后print。）

【讨论】：

如果您看到 id 为 ENSG00000087266:I3_28 的行，除了最后一列之外的所有内容都相同，我想保留最后一列中包含“B”的行。