The Best Ways to Compare Two Lists in Python
Check if two lists are equal, which elements match, get the difference between two lists, compare lists of dictionaries, list of strings and more!
Table of contents
- Comparing if two lists are equal in python
- Comparing two lists of float numbers
- Comparing if two lists without order (unordered lists) are equal
- How to compare two lists and return matches
- How to compare two lists in python and return differences
- How to compare two lists of strings
- How to compare two lists of dictionaries
- How to compare two list of lists
- How to compare two lists of objects
- How to compare two lists of numpy arrays
- Conclusion
A while ago I wrote a guide on how to compare two dictionaries in Python 3, and how this task is not as simple as it might sound. It turns out comparing two lists in Python is just so tricky as comparing dict
s.
The way we've been taught to compare two objects in Python is a bit misleading. Most books and tutorials teach object comparison by using either the ==
or the is
operator. In reality, these two operators cover just a small fraction of the most frequent use cases.
For example:
- what if we want to compare a list of floating-point numbers considering a certain tolerance?
- what if we wish to contrast two lists but ignoring the order in which the elements appear?
- maybe we need to compare two lists and return the elements that intersect both
- sometimes we might want to get the difference between two lists
- what if we have two lists of strings and need to compare them by ignoring the string cases?
- what if we're given a list of
numpy
arrays to compare each other, what can we do? - or maybe we have a list of custom objects, or a list of dictionaries.
The list goes on and on, and for all of these use cases using ==
doesn't help.
That's what we are going to see in this article. We’ll learn the best ways of comparing two lists in Python for several use cases where the ==
operator is not enough.
Ready? Let's go!
Comparing if two lists are equal in python
The easiest way to compare two lists for equality is to use the ==
operator. This comparison method works well for simple cases, but as we'll see later, it doesn't work with advanced comparisons.
An example of a simple case would be a list of int
or str
objects.
>>> numbers = [1, 2, 3]
>>> target = [1, 2, 3]
>>> numbers == target
True
>>> [1, 2, 3] == [1, 3, 2]
False
>>> ['name', 'lastname'] == ['name', 'lastname']
True
>>> ['name', 'lastname'] == ['name', 'last name']
False
Pretty simple, right? Unfortunately, the world is complex, and so is production grade code. In the real world, things get complicated really fast. As an illustration, consider the following cases.
Suppose you have a list of floating points that is built dynamically. You can add single elements, or elements derived from a mathematical operation such as 0.1 + 0.1
.
>>> numbers = []
>>> numbers.append(0.1 + 0.1 + 0.1) # derive the element based on a summation
>>> numbers.append(0.2) # add a single element
>>> target = [0.3, 0.2]
>>> numbers == target # compares the lists
False
>>> numbers # Ooopppssss....
[0.30000000000000004, 0.2]
>>> target
[0.3, 0.2]
Clearly, floating point arithmetic has its limitations, and sometimes we want to compare two lists but ignore precision errors, or even define some tolerance. For cases like this, the ==
operator won’t suffice.
Things can get more complicated if the lists have custom objects or objects from other libraries, such as numpy
.
In [1]: import numpy as np
In [2]: numbers = [np.ones(3), np.zeros(2)]
In [3]: numbers
Out[3]: [array([1., 1., 1.]), array([0., 0.])]
In [4]: target = [np.ones(3), np.zeros(2)]
In [5]: numbers == target
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-5-b832db4b039d> in <module>
----> 1 numbers == target
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
You might also like to compare the lists and return the matches. Or maybe compare the two lists and return the differences. Or perhaps you want to compare two lists ignoring the duplicates, or compare a list of dictionaries in Python.
In every single case, using ==
is not the answer, and that's what we are going to see next: how to perform complex comparison operations between two lists in Python.
Comparing two lists of float numbers
In the previous section, we saw that floating point arithmetic can cause precision errors. If we have a list of floats and want to compare it with another list, chances are that the ==
operator won't help.
Let's revisit the example from the previous section and see what is the best way of comparing two lists of floats.
>>> numbers = []
>>> numbers.append(0.1 + 0.1 + 0.1) # derive the element based on a summation
>>> numbers.append(0.2) # add a single element
>>> target = [0.3, 0.2]
>>> numbers == target # compares the lists
False
>>> numbers # Ooopppssss....
[0.30000000000000004, 0.2]
>>> target
[0.3, 0.2]
As you see, 0.1 + 0.1 + 0.1 = 0.30000000000000004
, which causes the comparison to fail. Now, how can we do better? Is it even possible?
There are a few ways of doing approaching this task. One would be to create our own custom function, that iterates over the elements and compare it one by one using the math.isclose()
function.
Fortunately we don't have to reinvent the wheel. As I showed in the "how to compare two dicts" article, we can use a library called deepdiff
for that. This library supports different types of objects and lists are one of them.
The example below starts off by setting up the two lists we want to compare. We then pass it to the deepdiff.DeepDiff
constructor which returns the difference. That's great, the returned value is much more informative than a simple boolean.
Since we want to ignore the precision error, we can set the number of digits AFTER the decimal point to be used in the comparison.
The result is an empty dict, which means the lists are equal. If we try comparing a list with a float number that differs in more than 3 significant digits, the library will return that diff.
For reproducibility, in this article I used the latest version of deepdiff
which is 5.6.0
.
In [1]: from deepdiff import DeepDiff
In [2]: numbers = []
In [3]: numbers.append(0.1 + 0.1 + 0.1) # derive the element based on a summation
In [4]: numbers.append(0.2) # add a single element
In [5]: target = [0.3, 0.2]
# if we don't specify the number of significant digits, the comparison will use ==
In [6]: DeepDiff(numbers, target)
Out[6]:
{'values_changed': {'root[0]': {'new_value': 0.3,
'old_value': 0.30000000000000004}}}
# 0.30000000000000004 and 0.3 are equal if we only look at the first 3 significant digits
In [7]: DeepDiff(numbers, target, significant_digits=3)
Out[7]: {}
In [8]: numbers
Out[8]: [0.30000000000000004, 0.2]
In [9]: target = [0.341, 0.2]
# 0.341 differs in more than 3 significant digits
In [10]: DeepDiff(numbers, target, significant_digits=3)
Out[10]:
{'values_changed': {'root[0]': {'new_value': 0.341,
'old_value': 0.30000000000000004}}}
Comparing if two lists without order (unordered lists) are equal
Lists in Python are unordered by default. Sometimes we want to compare two lists but treat them as the same as long as they have the same elements—regardless of their order.
There are two ways of doing this:
- sorting the lists and using the
==
operator - converting them to
set
s and using the==
operator - using
deepdiff
These first two methods assume the elements can be safely compared using the ==
operator. This approach doesn’t work for floating-point numbers, and other complex objects, but as we saw in the previous section, we can use deepdiff
.
Sorting the lists and using the ==
operator
You can sort lists in Python in two different ways:
- using the
list.sort()
method - using the
sorted()
function
The first method sorts a list in place, and that means your list will be modified. It's a good idea to not modify a list in place as it can introduce bugs that are hard to detect.
Using sorted
is better since it returns a new list and keep the original unmodified.
Let's see how it works.
In [6]: numbers = [10, 30, 20]
In [7]: target = [10, 20, 30]
In [8]: numbers == target
Out[8]: False
In [9]: sorted(numbers) == sorted(target)
Out[9]: True
In [10]: sorted(numbers)
Out[10]: [10, 20, 30]
In [11]: sorted(target)
Out[11]: [10, 20, 30]
As a consequence, by sorting the lists first we ensure that both lists will have the same order, and thus can be compared using the ==
operator.
Converting the list
s to a set
Contrary to lists, sets in Python don’t care about order. For example, a set {1, 2, 3}
is the same as {2, 3, 1}
. As such, we can use this feature to compare the two lists ignoring the elements’ order.
To do so, we convert each list into a set, then using the ==
to compare them.
In [12]: numbers = [10, 30, 20]
In [13]: target = [10, 20, 30]
In [14]: set(numbers) == set(target)
Out[14]: True
In [15]: set(numbers)
Out[15]: {10, 20, 30}
In [16]: set(target)
Out[16]: {10, 20, 30}
Using the deepdiff
library
This library also allows us to ignore the order in sequences such as list
s. By default, it will take the order in consideration, but if we set ignore_order
to True
, then we're all good. Let's see this in action.
In [11]: numbers = [10, 30, 20]
In [12]: target = [10, 20, 30]
In [13]: DeepDiff(numbers, target)
Out[13]:
{'values_changed': {'root[1]': {'new_value': 20, 'old_value': 30},
'root[2]': {'new_value': 30, 'old_value': 20}}}
In [14]: DeepDiff(numbers, target, ignore_order=True)
Out[14]: {}
Using deepdiff
has pros and cons. In the end, it is an external library you need to install, so if you can use a set
to compare the lists, then stick to it. However, if you have other use cases where it can shine, then I’d go with it.
How to compare two lists and return matches
In this section, we'll see how we can compare two lists and find their intersection. In other words, we want to find the values that appear in both.
To do that, we can once more use a set
and take their intersection.
In [1]: t1 = [2, 1, 0, 7, 4, 9, 3]
In [2]: t2 = [7, 6, 11, 12, 9, 23, 2]
In [3]: set(t1).intersection(set(t2))
Out[3]: {2, 7, 9}
# the & operator is a shorthand for the set.intersection() method
In [4]: set(t1) & set(t2)
Out[4]: {2, 7, 9}
How to compare two lists in python and return differences
We can the find difference between two lists in python in two different ways:
- using
set
- using the
deepdiff
library
Using set
Just like we did to determine the intersection, we can leverage the set
data structure to check difference between two lists in python.
If we want to get all the elements that are present in the first list but not in the second, we can use the set.difference()
.
On the other hand, if we want to find all the elements that are in either of the lists but not both, then we can use set.symmetric_difference()
.
In [8]: t1 = [2, 1, 0, 7, 4, 9, 3]
In [9]: t2 = [7, 6, 11, 12, 9, 23, 2]
In [10]: set(t1).difference(set(t2))
Out[10]: {0, 1, 3, 4}
In [11]: set(t2).difference(set(t1))
Out[11]: {6, 11, 12, 23}
In [12]: set(t1).symmetric_difference(set(t2))
Out[12]: {0, 1, 3, 4, 6, 11, 12, 23}
In [13]: set(t1) - set(t2)
Out[13]: {0, 1, 3, 4}
In [14]: set(t1) ^ set(t2)
Out[14]: {0, 1, 3, 4, 6, 11, 12, 23}
This method has a limitation: it groups what is different between the lists into one final result which is the set difference. What if we want to know which elements in that diff belong to what list?
Using deepdiff
As we've seen so far, this library is powerful and it returns a nice diff. Let's see what happens when we use deepdiff
to get the difference between two lists in Python.
In [15]: t1 = [2, 1, 0, 7, 4, 9, 3]
In [16]: t2 = [7, 6, 11, 12, 9, 23, 2]
In [17]: DeepDiff(t1, t2)
Out[17]:
{'values_changed': {'root[0]': {'new_value': 7, 'old_value': 2},
'root[1]': {'new_value': 6, 'old_value': 1},
'root[2]': {'new_value': 11, 'old_value': 0},
'root[3]': {'new_value': 12, 'old_value': 7},
'root[4]': {'new_value': 9, 'old_value': 4},
'root[5]': {'new_value': 23, 'old_value': 9},
'root[6]': {'new_value': 2, 'old_value': 3}}}
In [18]: DeepDiff(t1, t2, ignore_order=True)
Out[18]:
{'values_changed': {'root[4]': {'new_value': 6, 'old_value': 4},
'root[6]': {'new_value': 11, 'old_value': 3},
'root[1]': {'new_value': 12, 'old_value': 1}},
'iterable_item_added': {'root[5]': 23},
'iterable_item_removed': {'root[2]': 0}}
Accordingly, deepdiff
returns what changed from one list to the other. The right approach then will depend on your use case. If you want a detailed diff, then use DeepDiff
. Otherwise, just use a set
.
How to compare two lists of strings
Comparing two lists of string in Python depends largely on what type of comparison you want to make. That's because we can compare a string in a handful of ways.
In this section, we'll see 3 different ways of doing that.
The simplest one is using a ==
operator, like we saw in the beginning. This method is suitable if you want a strict comparison between each string.
In [1]: names = ['jack', 'josh', 'james']
In [2]: target = ['jack', 'josh', 'james']
In [3]: names == target
Out[3]: True
Things start to get messy if you want to compare the list of strings but ignoring the case. Using the ==
for that just doesn't work.
In [4]: names = ['Jack', 'Josh', 'James']
In [2]: target = ['jack', 'josh', 'james']
In [5]: names == target
Out[5]: False
The best tool for that is again deepdiff
. It allows us to ignore the string by passing a boolean flag to it.
In [1]: import deepdiff
In [2]: names = ['Jack', 'Josh', 'James']
In [3]: target = ['jack', 'josh', 'james']
# ignoring string case
In [4]: deepdiff.DeepDiff(names, target, ignore_string_case=True)
Out[4]: {}
# considering the case
In [5]: deepdiff.DeepDiff(names, target)
Out[5]:
{'values_changed': {'root[0]': {'new_value': 'jack', 'old_value': 'Jack'},
'root[1]': {'new_value': 'josh', 'old_value': 'Josh'},
'root[2]': {'new_value': 'james', 'old_value': 'James'}}}
We can also ignore the order in which the strings appear in the lists.
In [6]: names = ['Jack', 'James', 'Josh']
In [7]: target = ['jack', 'josh', 'james']
# ignoring the order and string case
In [8]: deepdiff.DeepDiff(names, target, ignore_string_case=True, ignore_order=T
...: rue)
Out[8]: {}
# considering the order but ignoring the case
In [9]: deepdiff.DeepDiff(names, target, ignore_string_case=True)
Out[9]:
{'values_changed': {'root[1]': {'new_value': 'josh', 'old_value': 'james'},
'root[2]': {'new_value': 'james', 'old_value': 'josh'}}}
You can also go further and perform advanced comparisons by passing a custom operator to DeepDiff
.
For example, suppose you want to compare the strings but ignoring any whitespace they may have.
Or perhaps you want to perform a fuzzy matching using an edit distance metric.
To do that, we can write the comparison logic in the operator class and pass it to DeepDiff
.
In this first example, we'll ignore any whitespace by trimming the strings before comparing them.
class IgnoreWhitespaceOperator:
def match(self, level) -> bool:
return True
def give_up_diffing(self, level, diff_instance) -> bool:
if isinstance(level.t1, str) and isinstance(level.t2, str):
return level.t1.strip() == level.t2.strip()
return False
Then we can just plug into DeepDiff
by adding it to the list of custom_operators
, like so custom_operators=[IgnoreWhitespaceOperator()]
.
In [6]: from deepdiff import DeepDiff
In [13]: names = ['Jack', 'James ', ' Josh ']
In [14]: target = ['Jack', 'James', 'Josh',]
# the operator will ignore the spaces in both lists
In [15]: DeepDiff(names, target, custom_operators=[IgnoreWhitespaceOperator()])
Out[15]: {}
In [16]: target = ['Jack', 'James', 'Josh', 'Jelly']
# if one of the list has an additional member, this will be flagged
In [17]: DeepDiff(names, target, custom_operators=[IgnoreWhitespaceOperator()])
Out[17]: {'iterable_item_added': {'root[3]': 'Jelly'}}
In [18]: target = ['Jack', 'Josh', 'James']
# by default, the library doesn't ignore order
In [19]: DeepDiff(names, target, custom_operators=[IgnoreWhitespaceOperator()])
Out[19]:
{'values_changed': {'root[1]': {'new_value': 'Josh', 'old_value': 'James '},
'root[2]': {'new_value': 'James', 'old_value': ' Josh '}}}
# if you don't care about order, be explicit
In [20]: DeepDiff(names, target, ignore_order=True, custom_operators=[IgnoreWhitespaceOperator()])
Out[20]: {}
How to compare two lists of dictionaries
Comparing two lists of dictionaries in Python is definitely intricate without the help of an external library. As we've seen so far, deepdiff
is versatile enough and we can use it to compare deep complex objects such as lists of dictionaries.
Let's see what happens when we pass two lists of dictionaries.
In [1]: from deepdiff import DeepDiff
In [2]: first_list = [
...: {
...: 'number': 1,
...: 'list': ['one', 'two']
...: },
...: {
...: 'number': 2,
...: 'list': ['one', 'two']
...: },
...: ]
In [3]: target_list = [
...: {
...: 'number': 3,
...: 'list': ['one', 'two']
...: },
...: {
...: 'number': 2,
...: 'list': ['one', 'two']
...: },
...: ]
In [4]: DeepDiff(first_list, target_list)
Out[4]: {'values_changed': {"root[0]['number']": {'new_value': 3, 'old_value': 1}}}
It outputs the exact location where the elements differ and what the difference is!
Let's see another example where a list has a missing element.
In [2]: first_list = [
...: {
...: 'number': 1,
...: 'list': ['one', 'two']
...: },
...: {
...: 'number': 2,
...: 'list': ['one', 'two']
...: },
...: ]
In [5]: target = [
...: {
...: 'number': 3,
...: 'list': ['one', 'two']
...: },
...: ]
In [6]:
In [6]: DeepDiff(first_list, target)
Out[6]:
{'values_changed': {"root[0]['number']": {'new_value': 3, 'old_value': 1}},
'iterable_item_removed': {'root[1]': {'number': 2, 'list': ['one', 'two']}}}
It says the the second dictionary has been removed, which is the case for this example.
How to compare two list of lists
Comparing multidimensional lists—a.k.a list of lists—is easy for deepdiff
. It works just like a list of dict
s.
In the example below, we have two multidimensional lists that we want to compare. When passed to DeepDiff
, it returns the exact location in which the elements differ.
For example, for the position [1][0]
, the new value is 8, and the old is 3. Another interesting aspect is that it works for deeply nested structures, for instance, deepdiff
also highlights the difference in the [2][0][0]
position.
In [1]: from deepdiff import DeepDiff
In [2]: first_list = [[1, 2], [3, 4], [[5]]]
In [3]: target_list = [[1, 2], [8, 4], [[7]]]
In [4]: DeepDiff(first_list, target_list)
Out[4]:
{'values_changed': {'root[1][0]': {'new_value': 8, 'old_value': 3},
'root[2][0][0]': {'new_value': 7, 'old_value': 5}}}
When feeding the library with two identical multidimensional lists, it returns an empty response.
In [3]: target_list = [[1, 2], [8, 4], [[7]]]
In [5]: second_list = [[1, 2], [8, 4], [[7]]]
In [7]: DeepDiff(second_list, target_list)
Out[7]: {}
How to compare two lists of objects
Sometimes we have a list of custom objects that we want to compare. Maybe we want to get a diff, or just check if they contain the same elements. The solution for this problem couldn't be different: use deepdiff
.
The following example demonstrates the power of this library. We're going to compare two lists containing a custom objects, and we'll be able to assert if they are equal or not and what are the differences.
In the example below, we have two lists of Person
objects. The only difference between the two is that in the last position Person
object has a different age. deepdiff
not only finds the right position - [1]
- but also finds that age
field is different as well.
In [9]: from deepdiff import DeepDiff
In [10]: first = [Person('Jack', 34), Person('Janine', 23)]
In [11]: target = [Person('Jack', 34), Person('Janine', 24)]
In [12]: DeepDiff(first, target)
Out[12]: {'values_changed': {'root[1].age': {'new_value': 24, 'old_value': 23}}}
In [14]: second = [Person('Jack', 34), Person('Janine', 24)]
In [15]: DeepDiff(second, target)
Out[15]: {}
How to compare two lists of numpy arrays
In this section, we'll see how to compare two lists of numpy
arrays. This is a fairly common task for those who work with data science and/or machine learning.
We saw in the first section that using the ==
operator doesn't work well with lists of numpy
arrays. Luckily we can use... guess what!? Yes, we can use deepdiff
.
The example below shows two lists with different numpy
arrays and the library can detect the exact position in which they differ. How cool is that?
In [16]: import numpy as np
In [17]: from deepdiff import DeepDiff
In [18]: first = [np.ones(3), np.array([1, 2, 3])]
In [19]: target = [np.zeros(4), np.array([1, 2, 3, 4])]
In [20]: DeepDiff(first, target)
Out[20]:
{'values_changed': {'root[0][0]': {'new_value': 0.0, 'old_value': 1.0},
'root[0][1]': {'new_value': 0.0, 'old_value': 1.0},
'root[0][2]': {'new_value': 0.0, 'old_value': 1.0}},
'iterable_item_added': {'root[0][3]': 0.0, 'root[1][3]': 4}}
Conclusion
In this post, we saw many ways to compare two lists in Python. The best method depends on what kind of elements we have and how we want to compare. Hopefully, you now know how to:
- check if two lists are equal in python
- compare two lists without order (unordered lists)
- compare two lists in python and return matches
- compare two lists in python and return differences
- compare two lists of strings
- compare two lists of dictionaries
- compare two list of lists
- compare two lists of objects
- compare two lists of numpy arrays
Other posts you may like:
See you next time!
This post was originally published at https://miguendes.me