Tag Archives: Unique

How to remove duplicates from a list while preserving order in Python

You can remove duplicates from a list easily by putting the elements in a set and then making a new list from the set’s contents:

def unique(sequence):
    return list(set(sequence))

However, this will put the elements in an arbitrary order. To preserve the order, you can:

  1. Use a set to keep track of which elements you’ve seen
  2. At the same time, populate the new list by a list comprehension
def unique(sequence):
    seen = set()
    return [x for x in sequence if not (x in seen or seen.add(x))]

Note that this relies on the fact that set.add() returns None.