This article focuses on one of the operations of getting the unique list from a list that contains a possible duplicated. Remove duplicates from list operation has large number of applications and hence, it’s knowledge is good to have.
Method 1 : Naive method In naive method, we simply traverse the list and append the first occurrence of the element in new list and ignore all the other occurrences of that particular element.
|
test_list = [1, 3, 5, 6, 3, 5, 6, 1] print ["The original list is : " + str[test_list]] res = [] for i in test_list: if i not in res: res.append[i] print ["The list after removing duplicates : " + str[res]] |
Output :
The original list is : [1, 3, 5, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]Method 2 : Using list comprehension This method has working similar to the above method, but this is just a one-liner shorthand of longer method done with the help of list comprehension.
|
test_list = [1, 3, 5, 6, 3, 5, 6, 1] print ["The original list is : " + str[test_list]] res = [] [res.append[x] for x in test_list if x not in res] print ["The list after removing duplicates : " + str[res]] |
Output :
The original list is : [1, 3, 5, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]
Method 3 : Using set[] This is the most popular way by which the duplicated are removed from the list. But the main and notable drawback of this approach is that the ordering of the element is lost in this particular method.
|
test_list = [1, 5, 3, 6, 3, 5, 6, 1] print ["The original list is : " + str[test_list]] test_list = list[set[test_list]] print ["The list after removing duplicates : " + str[test_list]] |
Output :
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]Method 4 : Using list comprehension + enumerate[] list comprehension coupled with enumerate function can also achieve this task. It basically looks for already occurred elements and skips adding them. It preserves the list ordering.
|
test_list = [1, 5, 3, 6, 3, 5, 6, 1] print ["The original list is : " + str[test_list]] res = [i for n, i in enumerate[test_list] if i not in test_list[:n]] print ["The list after removing duplicates : " + str[res]] |
Output :
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]Method 5 : Using collections.OrderedDict.fromkeys[] This is fastest method to achieve the particular task. It first removes the duplicates and returns a dictionary which has to be converted to list. This works well in case of strings also.
|
from collections import OrderedDict test_list = [1, 5, 3, 6, 3, 5, 6, 1] print ["The original list is : " + str[test_list]] res = list[OrderedDict.fromkeys[test_list]] print ["The list after removing duplicates : " + str[res]] |
Output :
The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]Article Tags :
Practice Tags :
- Apr 30, 2021
- 5 Minute Read
In this article, we will learn what is a list in python. As a python list is a collection of multiple elements even containing duplicates, sometimes it is necessary to make the list unique. Here, we are going to study the multiple ways to remove duplicates from the list in python. So, let's get started!
What is a List?
The list is the most important data type in python language. In Python language, the list is written as the list of commas separated values inside the square bracket. The most important advantage of the list is that the elements inside the list are not compulsorily of the same data type and negative indexing. Also, all the operation of the string is similarly applied on list data type such as slicing, concatenation, etc. Also, we can create the nested list i.e list containing another list.
For example:
# creating a list of items with different data types sample_list = [6,"mark",[A,I]] print[sample_list]
Output:
5 Ways to Remove Duplicates from a List in Python
There are many ways to remove the duplicates from a list in python. Let’s study them below:
Method 1: Naïve Method
In this method, we traverse the list and then append the first occurrence of the element in the new list, and then all other elements are ignored.
Example:
# removing duplicated from the list using naive methods # initializing list sam_list = [11, 13, 15, 16, 13, 15, 16, 11] print ["The list is: " + str[sam_list]] # remove duplicated from list result = [] for i in sam_list: if i not in result: result.append[i] # printing list after removal print ["The list after removing duplicates : " + str[result]]
Output:
The list is: [11, 13, 15, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 13, 15, 16]
Method 2: Using a list comprehensive
This method is similar to the above method, but this method is shorter than the above one.
Example:
# removing duplicated from the list using list comprehension # initializing list sam_list = [11, 13, 15, 16, 13, 15, 16, 11] print ["The list is: " + str[sam_list]] # to remove duplicated from list result = [] [result.append[x] for x in sam_list if x not in result] # printing list after removal print ["The list after removing duplicates: " + str[result]]
Output:
The list is: [11, 13, 15, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 13, 15, 16]
Method 3: Using set[]
This method is the most popular method to remove the duplicate from the python list. This is because the set data structure does not allow duplicates. But the drawback of this method is that the order of the elements is lost.
Example:
# removing duplicated from the list using set[] # initializing list sam_list = [11, 15, 13, 16, 13, 15, 16, 11] print ["The list is: " + str[sam_list]] # to remove duplicated from list sam_list = list[set[sam_list]] # printing list after removal # ordering distorted print ["The list after removing duplicates: " + str[sam_list]]
Output:
The list is: [11, 15, 13, 16, 13, 15, 16, 11]
The list after removing duplicates: [16, 11, 13, 15]
Method 4: Using list comprehensive + enumerate[]
List comprehensive when merged with enumerate function we can remove the duplicate from the python list. Basically in this method, the already occurred elements are skipped, and also the order is maintained.
Example:
# removing duplicated from the list using list comprehension + enumerate[] # initializing list sam_list = [11, 15, 13, 16, 13, 15, 16, 11] print ["The list is: " + str[sam_list]] # to remove duplicated from list result = [i for n, i in enumerate[sam_list] if i not in sam_list[:n]] # printing list after removal print ["The list after removing duplicates: " + str[result]]
Output:
The list is: [11, 13, 15, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 13, 15, 16]
Method 5: Using collections.OrderedDict.fromkeys[]
This is the fastest method to achieve the target of removing duplicates from the python list. This method will first remove the duplicates and return a dictionary that has converted to a list. Also, this method works well in the case of a string.
Example:
# removing duplicated from list using collections.OrderedDict.fromkeys[] from collections import OrderedDict # initializing list sam_list = [11, 15, 13, 16, 13, 15, 16, 11] print ["The list is: " + str[sam_list]] # to remove duplicated from list result = list[OrderedDict.fromkeys[sam_list]] # printing list after removal print ["The list after removing duplicates: " + str[result]]
Output:
The list is: [11, 15, 13, 16, 13, 15, 16, 11]
The list after removing duplicates: [11, 15, 13, 16]
These are a few of the methods with which we can remove the duplicate from the python list.
Conclusion
Hence, in this article, we learned about the python list and different methods to remove the duplicate elements from the list in python. Also, we studied the example along with the output for different methods.