Fastest way: set()

items = [1, 2, 2, 3, 3, 3]
unique = list(set(items))
print(unique)   # [1, 2, 3]

A set can only hold unique values, so duplicates disappear automatically. The catch: a set does not guarantee order.

Remove duplicates but keep the order

items = ["a", "b", "a", "c", "b"]
unique = list(dict.fromkeys(items))
print(unique)   # ["a", "b", "c"]

Dictionary keys are unique and, since Python 3.7, remember their insertion order — exactly what you want here.

The explicit loop version

seen = set()
unique = []
for item in items:
    if item not in seen:
        seen.add(item)
        unique.append(item)

This keeps order too and is easy to read when you are learning.

Common mistakes

  • Expecting set() to preserve order. It will not — use dict.fromkeys when order matters.
  • Trying to dedupe a list of lists. Sets need hashable items, so convert inner lists to tuples first.

Frequently asked questions

Why did set() change my order?

Sets are unordered by design. If the order of your data matters, use list(dict.fromkeys(items)) instead.

How do I remove duplicate dictionaries?

Dictionaries are not hashable, so track a key you care about (like an id) in a seen set and append only unseen items.

Want to master lists, sets and dictionaries? Try our free Python course.