Remove duplicate dictionaries from list Python | Example code

Using set can remove duplicate dictionaries from list Python. First, transform the initial dict into a list of tuples, then put them into a set (that removes duplicate entries), and then back into a dict.

[dict(t) for t in {tuple(d.items()) for d in l}]

Example remove duplicate dict in list in Python

Simple example code.

Convert the list of dictionaries to a list of tuples where the tuples contain the items of the dictionary. Since the tuples can be hashed, you can remove duplicates using set (using a set comprehension here, an older python alternative would be set(tuple(d.items()) for d in l)) and, after that, re-create the dictionaries from tuples with dict.

l = [{'a': 10, 'b': 4},
     {'a': 20, 'b': 4},
     {'a': 10, 'b': 4}]

seen = set()
new_l = []
for d in l:
    t = tuple(d.items())
    if t not in seen:
        seen.add(t)
        new_l.append(d)

print(new_l)

Output:

Source: stackoverflow.com

Remove Duplicate Dictionaries characterized by Key

test_list = [{"A": 6, "is": 19, "best": 1},
             {"A": 8, "is": 11, "best": 9},
             {"A": 2, "is": 16, "best": 1},
             {"A": 12, "is": 11, "best": 8},
             {"A": 22, "is": 16, "best": 8}]

# initializing Key 
K = "best"

memo = set()
res = []
for sub in test_list:

    # testing for already present value
    if sub[K] not in memo:
        res.append(sub)

        # adding in memo if new value
        memo.add(sub[K])

# printing result 
print(str(res))

Output: [{‘A’: 6, ‘is’: 19, ‘best’: 1}, {‘A’: 8, ‘is’: 11, ‘best’: 9}, {‘A’: 12, ‘is’: 11, ‘best’: 8}]

Do comment if you have any doubts and suggestions on this Python dictionaries topic.

Note: IDE: PyCharm 2021.3.3 (Community Edition)

Windows 10

Python 3.10.1

All Python Examples are in Python 3, so Maybe its different from python 2 or upgraded versions.