Python find duplicates in list of lists | Example code

Python find duplicates in the list of lists Example

Given a list of lists, check there are if there are any duplicates in lists of lists that have the same values and order.

Count the occurrences in a list comprehension, converting them to a tuple so you can hash & apply unicity:

routes = [[1, 2, 4, 6, 10], [1, 3, 8, 9, 10], [1, 2, 4, 6, 10]]
dups = {tuple(x) for x in routes if routes.count(x) > 1}

print(dups)

Output:

Simple enough, but a lot of looping under the hood because of repeated calls to count. There’s another way, which involves hashing but has a lower complexity would be to use collections.Counter:

from collections import Counter

routes = [[1, 2, 4, 6, 10], [1, 3, 8, 9, 10], [1, 2, 4, 6, 10]]

c = Counter(map(tuple,routes))
dups = [k for k,v in c.items() if v>1]

print(dups)

Find duplicate elements in a list of lists

If there is a duplicate element that is common between any or all the lists, return True. If all of the elements are unique, return False.

def find_dup(l):
    dupes = []
    flat = [item for sublist in l for item in sublist]
    for f in flat:
        if flat.count(f) > 1:
            if f not in dupes:
                dupes.append(f)

    if dupes:
        return True
    else:
        return False


l = [[20, 21, 22], [17, 18, 19, 20], [10, 11, 12, 13]]
print(find_dup(l))

Output: True

Finding duplicates in a list of lists and merger them

For example, de-duplicate a list of lists and merge the values of the duplicates.

original_list = [['a', 1], ['b', 1], ['a', 1], ['b', 1], ['b', 2], ['c', 2], ['b', 3]]

totals = {}
for k, v in original_list:
    totals[k] = totals.get(k, 0) + v

print(totals)

Output:

{‘a’: 2, ‘b’: 7, ‘c’: 2}

Source: stackoverflow.com

Do comment if you have any doubts and suggestions on this Python List topic.

Note: IDE: PyCharm 2021.3.3 (Community Edition)

Windows 10

Python 3.10.1

All Python Examples are in Python 3, so Maybe its different from python 2 or upgraded versions.