finding dupliates based on key-values from list of dict in python

Question

I have following list of dict records, where I need to extract out all the duplicates (based on the 'label') and leave one per 'label' in the original records. Also when the items get removed by 'label', always removed one with the 'headings' = True over the 'headings' = False:

Input:

records = [{"label":'x', "headings":False, "key":300}, {"label":'x', "headings":True, "key":301}, 
           {"label":'x', "headings":False, "key":302},{"label":'x', "headings":False, "key":303}, 
           {"label":'y', "headings":False, "key":304}, {"label":'y', "headings":True, "key":305},
           {"label":'z', "headings":True, "key":306}, {"label":'z', "headings":True, "key":307}]

output

duplicate_items :

records = [{"label":'x', "headings":False, "key":300},{"label":'x', "headings":True, "key":301},{"label":'x', "headings":False, "key":302},{"label":'y', "headings":True, "key":305},{"label":'z', "headings":True, "key":306}]

Answer 1

You left a few questions unanswered (see cmets). You also did not provide your own code and any unexpected output/error you got with it, so we have nothing to work with/fix. This is bad form.

But I found this to be a fun exercise, so here is what I came up with:

from typing import TypedDict


class Record(TypedDict):
    label: str
    headings: bool
    key: int


def remove_duplicates(records: list[Record]) -> list[Record]:
    # First, decide which records (by index) _not_ to remove.
    # Map labels to 2-tuples of (index, headings boolean):
    keep: dict[str, tuple[int, bool]] = {}
    for idx, record in enumerate(records):
        label, headings = record["label"], record["headings"]
        # We keep it, if this is the first time we see that label OR
        # we did encounter it, but this record's `headings` value is `False`,
        # whereas the previous one was `True`:
        if label not in keep or (not headings and keep[label][1]):
            keep[label] = (idx, headings)
    # Combine all indices we want to keep into one set for easy lookup:
    keep_indices = {idx for idx, _ in keep.values()}
    # Iterate over all record indices in reverse order
    # and pop the corresponding records if necessary:
    removed = []
    for idx in reversed(range(len(records))):
        if idx not in keep_indices:
            removed.append(records.pop(idx))
    return removed

The original list is mutated in-place, but a new list is created and returned from the removed dictionaries/duplicates. The algorithm creates a few helper-datastructures sacrificing a bit of memory, but should be fairly efficient in terms of time, i.e. approximatelyO(n)withnbeing the number of records.

To test it:

...

def main() -> None:
    from pprint import pprint
    records = [
        {"label": "x", "headings": False, "key": 300},
        {"label": "x", "headings": True, "key": 301},
        {"label": "x", "headings": False, "key": 302},
        {"label": "x", "headings": False, "key": 303},
        {"label": "y", "headings": False, "key": 304},
        {"label": "y", "headings": True, "key": 305},
        {"label": "z", "headings": True, "key": 306},
        {"label": "z", "headings": True, "key": 307},
    ]
    removed = remove_duplicates(records)  # type: ignore[arg-type]
    print("remaining:")
    pprint(records)
    removed.reverse()
    print("removed:")
    pprint(removed)


if __name__ == "__main__":
    main()

Output:

remaining:
[{'headings': False, 'key': 300, 'label': 'x'},
 {'headings': False, 'key': 304, 'label': 'y'},
 {'headings': True, 'key': 306, 'label': 'z'}]
removed:
[{'headings': True, 'key': 301, 'label': 'x'},
 {'headings': False, 'key': 302, 'label': 'x'},
 {'headings': False, 'key': 303, 'label': 'x'},
 {'headings': True, 'key': 305, 'label': 'y'},
 {'headings': True, 'key': 307, 'label': 'z'}]