The way we've been taught to compare two objects in Python is a bit misleading. Most books and tutorials teach object comparison by using either the == or the is operator. In reality, these two operators cover just a small fraction of the most frequent use cases.

For example:

• what if we want to compare a list of floating-point numbers considering a certain tolerance?
• what if we wish to contrast two lists but ignoring the order in which the elements appear?
• maybe we need to compare two lists and return the elements that intersect both
• sometimes we might want to get the difference between two lists
• what if we have two lists of strings and need to compare them by ignoring the string cases?
• what if we're given a list of numpy arrays to compare each other, what can we do?
• or maybe we have a list of custom objects, or a list of dictionaries.

The list goes on and on, and for all of these use cases using == doesn't help.

That's what we are going to see in this article. We’ll learn the best ways of comparing two lists in Python for several use cases where the == operator is not enough.

Ready? Let's go!

Comparing if two lists are equal in python

The easiest way to compare two lists for equality is to use the == operator. This comparison method works well for simple cases, but as we'll see later, it doesn't work with advanced comparisons.

An example of a simple case would be a list of int or str objects.

>>> numbers = [1, 2, 3]
>>> target = [1, 2, 3]
>>> numbers == target
True
>>> [1, 2, 3] == [1, 3, 2]
False
>>> ['name', 'lastname'] == ['name', 'lastname']
True
>>> ['name', 'lastname'] == ['name', 'last name']   
False

Pretty simple, right? Unfortunately, the world is complex, and so is production grade code. In the real world, things get complicated really fast. As an illustration, consider the following cases.

Suppose you have a list of floating points that is built dynamically. You can add single elements, or elements derived from a mathematical operation such as 0.1 + 0.1.

>>> numbers = []
>>> numbers.append(0.1 + 0.1 + 0.1)  # derive the element based on a summation
>>> numbers.append(0.2) # add a single element
>>> target = [0.3, 0.2]
>>> numbers == target  # compares the lists
False
>>> numbers  # Ooopppssss....
[0.30000000000000004, 0.2]
>>> target
[0.3, 0.2]

Clearly, floating point arithmetic has its limitations, and sometimes we want to compare two lists but ignore precision errors, or even define some tolerance. For cases like this, the == operator won’t suffice.

Things can get more complicated if the lists have custom objects or objects from other libraries, such as numpy.

In [1]: import numpy as np

In [2]: numbers = [np.ones(3), np.zeros(2)]

In [3]: numbers
Out[3]: [array([1., 1., 1.]), array([0., 0.])]

In [4]: target = [np.ones(3), np.zeros(2)]

In [5]: numbers == target
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-b832db4b039d> in <module>
----> 1 numbers == target

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

You might also like to compare the lists and return the matches. Or maybe compare the two lists and return the differences. Or perhaps you want to compare two lists ignoring the duplicates, or compare a list of dictionaries in Python.

In every single case, using == is not the answer, and that's what we are going to see next: how to perform complex comparison operations between two lists in Python.

Comparing two lists of float numbers

In the previous section, we saw that floating point arithmetic can cause precision errors. If we have a list of floats and want to compare it with another list, chances are that the == operator won't help.

Let's revisit the example from the previous section and see what is the best way of comparing two lists of floats.

>>> numbers = []
>>> numbers.append(0.1 + 0.1 + 0.1)  # derive the element based on a summation
>>> numbers.append(0.2) # add a single element
>>> target = [0.3, 0.2]
>>> numbers == target  # compares the lists
False
>>> numbers  # Ooopppssss....
[0.30000000000000004, 0.2]
>>> target
[0.3, 0.2]

As you see, 0.1 + 0.1 + 0.1 = 0.30000000000000004, which causes the comparison to fail. Now, how can we do better? Is it even possible?

There are a few ways of doing approaching this task. One would be to create our own custom function, that iterates over the elements and compare it one by one using the math.isclose() function.

Fortunately we don't have to reinvent the wheel. As I showed in the "how to compare two dicts" article, we can use a library called deepdiff for that. This library supports different types of objects and lists are one of them.

The example below starts off by setting up the two lists we want to compare. We then pass it to the deepdiff.DeepDiff constructor which returns the difference. That's great, the returned value is much more informative than a simple boolean.

Since we want to ignore the precision error, we can set the number of digits AFTER the decimal point to be used in the comparison.

The result is an empty dict, which means the lists are equal. If we try comparing a list with a float number that differs in more than 3 significant digits, the library will return that diff.

For reproducibility, in this article I used the latest version of deepdiff which is 5.6.0.

In [1]: from deepdiff import DeepDiff

In [2]: numbers = []

In [3]: numbers.append(0.1 + 0.1 + 0.1)  # derive the element based on a summation

In [4]: numbers.append(0.2) # add a single element

In [5]: target = [0.3, 0.2]

# if we don't specify the number of significant digits, the comparison will use ==
In [6]: DeepDiff(numbers, target)
Out[6]: 
{'values_changed': {'root[0]': {'new_value': 0.3,
   'old_value': 0.30000000000000004}}}

# 0.30000000000000004 and 0.3 are equal if we only look at the first 3 significant digits
In [7]: DeepDiff(numbers, target, significant_digits=3)
Out[7]: {}

In [8]: numbers
Out[8]: [0.30000000000000004, 0.2]

In [9]: target = [0.341, 0.2]

# 0.341 differs in more than 3 significant digits
In [10]: DeepDiff(numbers, target, significant_digits=3)
Out[10]: 
{'values_changed': {'root[0]': {'new_value': 0.341,
   'old_value': 0.30000000000000004}}}

Comparing if two lists without order (unordered lists) are equal

Lists in Python are unordered by default. Sometimes we want to compare two lists but treat them as the same as long as they have the same elements—regardless of their order.

There are two ways of doing this:

• sorting the lists and using the == operator
• converting them to sets and using the == operator
• using deepdiff

These first two methods assume the elements can be safely compared using the == operator. This approach doesn’t work for floating-point numbers, and other complex objects, but as we saw in the previous section, we can use deepdiff.

Sorting the lists and using the == operator

You can sort lists in Python in two different ways:

• using the list.sort() method
• using the sorted() function

The first method sorts a list in place, and that means your list will be modified. It's a good idea to not modify a list in place as it can introduce bugs that are hard to detect.

Using sorted is better since it returns a new list and keep the original unmodified.

Let's see how it works.

In [6]: numbers = [10, 30, 20]

In [7]: target = [10, 20, 30]

In [8]: numbers == target
Out[8]: False

In [9]: sorted(numbers) == sorted(target)
Out[9]: True

In [10]: sorted(numbers)
Out[10]: [10, 20, 30]

In [11]: sorted(target)
Out[11]: [10, 20, 30]

As a consequence, by sorting the lists first we ensure that both lists will have the same order, and thus can be compared using the == operator.

Converting the lists to a set

Contrary to lists, sets in Python don’t care about order. For example, a set {1, 2, 3} is the same as {2, 3, 1}. As such, we can use this feature to compare the two lists ignoring the elements’ order.

To do so, we convert each list into a set, then using the == to compare them.

In [12]: numbers = [10, 30, 20]

In [13]: target = [10, 20, 30]

In [14]: set(numbers) == set(target)
Out[14]: True

In [15]: set(numbers)
Out[15]: {10, 20, 30}

In [16]: set(target)
Out[16]: {10, 20, 30}

Using the deepdiff library

This library also allows us to ignore the order in sequences such as lists. By default, it will take the order in consideration, but if we set ignore_order to True, then we're all good. Let's see this in action.

In [11]: numbers = [10, 30, 20]

In [12]: target = [10, 20, 30]

In [13]: DeepDiff(numbers, target)
Out[13]: 
{'values_changed': {'root[1]': {'new_value': 20, 'old_value': 30},
  'root[2]': {'new_value': 30, 'old_value': 20}}}

In [14]: DeepDiff(numbers, target, ignore_order=True)
Out[14]: {}

Using deepdiff has pros and cons. In the end, it is an external library you need to install, so if you can use a set to compare the lists, then stick to it. However, if you have other use cases where it can shine, then I’d go with it.

How to compare two lists and return matches

In this section, we'll see how we can compare two lists and find their intersection. In other words, we want to find the values that appear in both.

To do that, we can once more use a set and take their intersection.

In [1]: t1 = [2, 1, 0, 7, 4, 9, 3]

In [2]: t2 = [7, 6, 11, 12, 9, 23, 2]

In [3]: set(t1).intersection(set(t2))
Out[3]: {2, 7, 9}

# the & operator is a shorthand for the set.intersection() method 
In [4]: set(t1) & set(t2)
Out[4]: {2, 7, 9}

How to compare two lists in python and return differences

We can the find difference between two lists in python in two different ways:

• using set
• using thedeepdiff library

Using set

Just like we did to determine the intersection, we can leverage the set data structure to check difference between two lists in python.

If we want to get all the elements that are present in the first list but not in the second, we can use the set.difference().

On the other hand, if we want to find all the elements that are in either of the lists but not both, then we can use set.symmetric_difference().

In [8]: t1 = [2, 1, 0, 7, 4, 9, 3]

In [9]: t2 = [7, 6, 11, 12, 9, 23, 2]

In [10]: set(t1).difference(set(t2))
Out[10]: {0, 1, 3, 4}

In [11]: set(t2).difference(set(t1))
Out[11]: {6, 11, 12, 23}

In [12]: set(t1).symmetric_difference(set(t2))
Out[12]: {0, 1, 3, 4, 6, 11, 12, 23}

In [13]: set(t1) - set(t2)
Out[13]: {0, 1, 3, 4}

In [14]: set(t1) ^ set(t2)
Out[14]: {0, 1, 3, 4, 6, 11, 12, 23}

This method has a limitation: it groups what is different between the lists into one final result which is the set difference. What if we want to know which elements in that diff belong to what list?

Using deepdiff

As we've seen so far, this library is powerful and it returns a nice diff. Let's see what happens when we use deepdiff to get the difference between two lists in Python.

In [15]: t1 = [2, 1, 0, 7, 4, 9, 3]

In [16]: t2 = [7, 6, 11, 12, 9, 23, 2]

In [17]: DeepDiff(t1, t2)
Out[17]: 
{'values_changed': {'root[0]': {'new_value': 7, 'old_value': 2},
  'root[1]': {'new_value': 6, 'old_value': 1},
  'root[2]': {'new_value': 11, 'old_value': 0},
  'root[3]': {'new_value': 12, 'old_value': 7},
  'root[4]': {'new_value': 9, 'old_value': 4},
  'root[5]': {'new_value': 23, 'old_value': 9},
  'root[6]': {'new_value': 2, 'old_value': 3}}}

In [18]: DeepDiff(t1, t2, ignore_order=True)
Out[18]: 
{'values_changed': {'root[4]': {'new_value': 6, 'old_value': 4},
  'root[6]': {'new_value': 11, 'old_value': 3},
  'root[1]': {'new_value': 12, 'old_value': 1}},
 'iterable_item_added': {'root[5]': 23},
 'iterable_item_removed': {'root[2]': 0}}

Accordingly, deepdiff returns what changed from one list to the other. The right approach then will depend on your use case. If you want a detailed diff, then use DeepDiff. Otherwise, just use a set.

How to compare two lists of strings

Comparing two lists of string in Python depends largely on what type of comparison you want to make. That's because we can compare a string in a handful of ways.

In this section, we'll see 3 different ways of doing that.

The simplest one is using a == operator, like we saw in the beginning. This method is suitable if you want a strict comparison between each string.

In [1]: names = ['jack', 'josh', 'james']

In [2]: target = ['jack', 'josh', 'james']

In [3]: names == target
Out[3]: True

Things start to get messy if you want to compare the list of strings but ignoring the case. Using the == for that just doesn't work.

In [4]: names = ['Jack', 'Josh', 'James']

In [2]: target = ['jack', 'josh', 'james']

In [5]: names == target
Out[5]: False

The best tool for that is again deepdiff. It allows us to ignore the string by passing a boolean flag to it.

In [1]: import deepdiff

In [2]: names = ['Jack', 'Josh', 'James']

In [3]: target = ['jack', 'josh', 'james']

# ignoring string case
In [4]: deepdiff.DeepDiff(names, target, ignore_string_case=True)
Out[4]: {}

# considering the case
In [5]: deepdiff.DeepDiff(names, target)
Out[5]: 
{'values_changed': {'root[0]': {'new_value': 'jack', 'old_value': 'Jack'},
  'root[1]': {'new_value': 'josh', 'old_value': 'Josh'},
  'root[2]': {'new_value': 'james', 'old_value': 'James'}}}

We can also ignore the order in which the strings appear in the lists.

In [6]: names = ['Jack', 'James', 'Josh']

In [7]: target = ['jack', 'josh', 'james']

# ignoring the order and string case
In [8]: deepdiff.DeepDiff(names, target, ignore_string_case=True, ignore_order=T
   ...: rue)
Out[8]: {}

# considering the order but ignoring the case
In [9]: deepdiff.DeepDiff(names, target, ignore_string_case=True)
Out[9]: 
{'values_changed': {'root[1]': {'new_value': 'josh', 'old_value': 'james'},
  'root[2]': {'new_value': 'james', 'old_value': 'josh'}}}

You can also go further and perform advanced comparisons by passing a custom operator to DeepDiff.

For example, suppose you want to compare the strings but ignoring any whitespace they may have.

Or perhaps you want to perform a fuzzy matching using an edit distance metric.

To do that, we can write the comparison logic in the operator class and pass it to DeepDiff.

In this first example, we'll ignore any whitespace by trimming the strings before comparing them.

class IgnoreWhitespaceOperator:

    def match(self, level) -> bool:
        return True

    def give_up_diffing(self, level, diff_instance) -> bool:
        if isinstance(level.t1, str) and isinstance(level.t2, str):
            return level.t1.strip() == level.t2.strip()
        return False

Then we can just plug into DeepDiff by adding it to the list of custom_operators, like so custom_operators=[IgnoreWhitespaceOperator()].

In [6]: from deepdiff import DeepDiff

In [13]: names = ['Jack', 'James ', '  Josh ']

In [14]: target = ['Jack', 'James', 'Josh',]

# the operator will ignore the spaces in both lists
In [15]: DeepDiff(names, target, custom_operators=[IgnoreWhitespaceOperator()])
Out[15]: {}

In [16]: target = ['Jack', 'James', 'Josh', 'Jelly']

# if one of the list has an additional member, this will be flagged
In [17]: DeepDiff(names, target, custom_operators=[IgnoreWhitespaceOperator()])
Out[17]: {'iterable_item_added': {'root[3]': 'Jelly'}}

In [18]: target = ['Jack', 'Josh', 'James']

# by default, the library doesn't ignore order
In [19]: DeepDiff(names, target, custom_operators=[IgnoreWhitespaceOperator()])
Out[19]: 
{'values_changed': {'root[1]': {'new_value': 'Josh', 'old_value': 'James '},
  'root[2]': {'new_value': 'James', 'old_value': '  Josh '}}}

# if you don't care about order, be explicit
In [20]: DeepDiff(names, target, ignore_order=True, custom_operators=[IgnoreWhitespaceOperator()])
Out[20]: {}

How to compare two lists of dictionaries

Comparing two lists of dictionaries in Python is definitely intricate without the help of an external library. As we've seen so far, deepdiff is versatile enough and we can use it to compare deep complex objects such as lists of dictionaries.

Let's see what happens when we pass two lists of dictionaries.

In [1]: from deepdiff import DeepDiff

In [2]: first_list = [
   ...:     {
   ...:         'number': 1,
   ...:         'list': ['one', 'two']
   ...:     },
   ...:     {
   ...:         'number': 2,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [3]: target_list = [
   ...:     {
   ...:         'number': 3,
   ...:         'list': ['one', 'two']
   ...:     },
   ...:     {
   ...:         'number': 2,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [4]: DeepDiff(first_list, target_list)
Out[4]: {'values_changed': {"root[0]['number']": {'new_value': 3, 'old_value': 1}}}

It outputs the exact location where the elements differ and what the difference is!

Let's see another example where a list has a missing element.

In [2]: first_list = [
   ...:     {
   ...:         'number': 1,
   ...:         'list': ['one', 'two']
   ...:     },
   ...:     {
   ...:         'number': 2,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [5]: target = [
   ...:     {
   ...:         'number': 3,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [6]: 

In [6]: DeepDiff(first_list, target)
Out[6]: 
{'values_changed': {"root[0]['number']": {'new_value': 3, 'old_value': 1}},
 'iterable_item_removed': {'root[1]': {'number': 2, 'list': ['one', 'two']}}}

It says the the second dictionary has been removed, which is the case for this example.

How to compare two list of lists

Comparing multidimensional lists—a.k.a list of lists—is easy for deepdiff. It works just like a list of dicts.

In the example below, we have two multidimensional lists that we want to compare. When passed to DeepDiff, it returns the exact location in which the elements differ.

For example, for the position [1][0], the new value is 8, and the old is 3. Another interesting aspect is that it works for deeply nested structures, for instance, deepdiff also highlights the difference in the [2][0][0] position.

In [1]: from deepdiff import DeepDiff

In [2]: first_list = [[1, 2], [3, 4], [[5]]]

In [3]: target_list = [[1, 2], [8, 4], [[7]]]

In [4]: DeepDiff(first_list, target_list)
Out[4]: 
{'values_changed': {'root[1][0]': {'new_value': 8, 'old_value': 3},
  'root[2][0][0]': {'new_value': 7, 'old_value': 5}}}

When feeding the library with two identical multidimensional lists, it returns an empty response.

In [3]: target_list = [[1, 2], [8, 4], [[7]]]

In [5]: second_list = [[1, 2], [8, 4], [[7]]]

In [7]: DeepDiff(second_list, target_list)
Out[7]: {}

How to compare two lists of objects

Sometimes we have a list of custom objects that we want to compare. Maybe we want to get a diff, or just check if they contain the same elements. The solution for this problem couldn't be different: use deepdiff.

The following example demonstrates the power of this library. We're going to compare two lists containing a custom objects, and we'll be able to assert if they are equal or not and what are the differences.

In the example below, we have two lists of Person objects. The only difference between the two is that in the last position Person object has a different age. deepdiff not only finds the right position - [1] - but also finds that age field is different as well.

In [9]: from deepdiff import DeepDiff

In [10]: first = [Person('Jack', 34), Person('Janine', 23)]

In [11]: target = [Person('Jack', 34), Person('Janine', 24)]

In [12]: DeepDiff(first, target)
Out[12]: {'values_changed': {'root[1].age': {'new_value': 24, 'old_value': 23}}}

In [14]: second = [Person('Jack', 34), Person('Janine', 24)]

In [15]: DeepDiff(second, target)
Out[15]: {}

How to compare two lists of numpy arrays

In this section, we'll see how to compare two lists of numpy arrays. This is a fairly common task for those who work with data science and/or machine learning.

We saw in the first section that using the == operator doesn't work well with lists of numpyarrays. Luckily we can use... guess what!? Yes, we can use deepdiff.

The example below shows two lists with different numpy arrays and the library can detect the exact position in which they differ. How cool is that?

In [16]: import numpy as np

In [17]: from deepdiff import DeepDiff

In [18]: first = [np.ones(3), np.array([1, 2, 3])]

In [19]: target = [np.zeros(4), np.array([1, 2, 3, 4])]

In [20]: DeepDiff(first, target)
Out[20]: 
{'values_changed': {'root[0][0]': {'new_value': 0.0, 'old_value': 1.0},
  'root[0][1]': {'new_value': 0.0, 'old_value': 1.0},
  'root[0][2]': {'new_value': 0.0, 'old_value': 1.0}},
 'iterable_item_added': {'root[0][3]': 0.0, 'root[1][3]': 4}}

Conclusion

In this post, we saw many ways to compare two lists in Python. The best method depends on what kind of elements we have and how we want to compare. Hopefully, you now know how to:

• check if two lists are equal in python
• compare two lists without order (unordered lists)
• compare two lists in python and return matches
• compare two lists in python and return differences
• compare two lists of strings
• compare two lists of dictionaries
• compare two list of lists
• compare two lists of objects
• compare two lists of numpy arrays

Other posts you may like:

• The Best Way to Compare Two Dictionaries in Python

• How to Compare Two Strings in Python (in 8 Easy Ways)

• 7 Different Ways to Flatten a List of Lists in Python

See you next time!

This post was originally published at https://miguendes.me

When it comes to Python, there are several ways of doing it. The best one will always depend on the use case, but we can narrow them down to a few that best fit this goal.

In this article, we'll do exactly that.

By the end of this tutorial, you'll have learned:

• how to compare strings using the == and != operators
• how to use the is operator to compare two strings
• how to compare strings using the <, >, <=, and >= operators
• how to compare two string ignoring the case
• how to ignore whitespaces when performing string comparison
• how to determine if two strings are similar by doing fuzzy matching
• how to compare two strings and return the difference
• how to debug when the string comparison is not working

Let's go!

Comparing strings using the == and != operators

The simplest way to check if two strings are equal in Python is to use the == operator. And if you are looking for the opposite, then != is what you need. That's it!

== and != are boolean operators, meaning they return True or False. For example, == returns True if the two strings match, and False otherwise.

>>> name = 'Carl'

>>> another_name = 'Carl'

>>> name == another_name
True

>>> name != another_name
False

>>> yet_another_name = 'Josh'

>>> name == yet_another_name
False

These operators are also case sensitive, which means uppercase letters are treated differently. The example below shows just that, city starts with an uppercase L whereas capital starts with a lowercase l. As a result, Python returns False when comparing them with ==.

>>> name = 'Carl'

>>> yet_another_name = 'carl'

>>> name == yet_another_name
False

>>> name != yet_another_name
True

Comparing strings using the is operator

Another way of comparing if two strings are equal in Python is using the is operator. However, the kind of comparison it performs is different than ==. The is operator compare if the 2 string are the same instance.

In Python—and in many other languages—we say two objects are the same instance if they are the same object in memory.

>>> name = 'John Jabocs Howard'

>>> another_name = name

>>> name is another_name
True

>>> yet_another_name = 'John Jabocs Howard'

>>> name is yet_another_name
False

>>> id(name)
140142470447472

>>> id(another_name)
140142470447472

>>> id(yet_another_name)
140142459568816

The image below shows how this example would be represented in memory.

As you see, we're comparing identities, not content. Objects with the same identity usually have the same references, and share the same memory location. Keep that in mind when using the is operator.

Comparing strings using the <, >, <=, and >= operators

The third way of comparing strings is alphabetically. This is useful when we need to determine the lexicographical order of two strings.

Let's see an example.

>>> name = 'maria'

>>> another_name = 'marcus'

>>> name < another_name
False

>>> name > another_name
True

>>> name <= another_name
False

>>> name >= another_name
True

To determine the order, Python compares the strings char by char. In our example, the first three letters are the same mar, but the next one is not, c from marcus comes before i from maria.

It's important to have in mind that this comparisons are case-sensitive. Python treats upper-case and lower-case differently. For example, if we change "maria" to "Maria", then the result is different because M comes before m.

>>> name = 'Maria'

>>> another_name = 'marcus'

>>> name < another_name
True

>>> ord('M') < ord('m')
True

>>> ord('M')
77

>>> ord('m')
109

⚠️ WARNING ⚠️: Avoid comparing strings that represent numbers using these operators. The comparison is done based on alphabetical ordering, which causes "2" < "10" to evaluated to False.

>>> a = '2'

>>> b = '10'

>>> a < b
False

>>> a <= b
False

>>> a > b
True

>>> a >= b
True

Compare two strings by ignoring the case

Sometimes we may need to compare two strings—a list of strings, or even a dictionary of strings—regardless of the case.

Achieving that will depend on the alphabet we're dealing with. For ASCII strings, we can either convert both strings to lowercase using str.lower(), or uppercase with str.upper() and compare them.

For other alphabets, such as Greek or German, converting to lowercase to make the strings case insensitive doesn't always work. Let's see some examples.

Suppose we have a string in German named 'Straße', which means "Street". You can also write the same word without the ß, in this case, the word becomes Strasse. If we try to lowercase it, or uppercase it, see what happens.

>>> a = 'Atraße'

>>> a = 'Straße'

>>> b = 'strasse'

>>> a.lower() == b.lower()
False

>>> a.lower()
'straße'

>>> b.lower()
'strasse'

That happens because a simple call to str.lower() won't do anything to ß. Its lowercase form is equivalent to ss but ß itself has the same form and shape in lower or upper case.

The best way to ignore case and make effective case insensitive string comparisons is to use str.casefold. According to the docs:

Casefolding is similar to lowercasing but more aggressive because it is intended to remove all case distinctions in a string.

Let's see what happens when we use str.casefold instead.

>>> a = 'Straße'

>>> b = 'strasse'

>>> a.casefold() == b.casefold()
True

>>> a.casefold()
'strasse'

>>> b.casefold()
'strasse'

How to compare two strings and ignore whitespace

Sometimes you might want to compare two strings by ignoring space characters. The best solution for this problem depends on where the spaces are, whether there are multiple spaces in the string and so on.

The first example we'll see consider that the only difference between the strings is that one of them have leading and/or trailing spaces. In this case, we can trim both strings using the str.strip method and use the == operator to compare them.

>>> s1 = 'Hey, I really like this post.'

>>> s2 = '      Hey, I really like this post.   '

>>> s1.strip() == s2.strip()
True

However, sometimes you have a string with whitespaces all over it, including multiple spaces inside it. If that is the case, then str.strip is not enough.

>>> s2 = '      Hey, I really      like this post.   '

>>> s1 = 'Hey, I really like this post.'

>>> s1.strip() == s2.strip()
False

The alternative then is to remove the duplicate whitespaces using a regular expression. This method only returns duplicated chars, so we still need to strip the leading and trailing ones.

>>> s2 = '      Hey, I really      like this post.   '

>>> s1 = 'Hey, I really like this post.'

>>> re.sub('\s+', ' ', s1.strip())
'Hey, I really like this post.'

>>> re.sub('\s+', ' ', s2.strip())
'Hey, I really like this post.'

>>> re.sub('\s+', ' ', s1.strip()) == re.sub('\s+', ' ', s2.strip())
True

Or if you don't care about duplicates and want to remove everything, then just pass the empty string as the second argument to re.sub.

>>> s2 = '      Hey, I really      like this post.   '

>>> s1 = 'Hey, I really like this post.'

>>> re.sub('\s+', '', s1.strip())
'Hey,Ireallylikethispost.'

>>> re.sub('\s+', '', s2.strip())
'Hey,Ireallylikethispost.'

>>> re.sub('\s+', '', s1.strip()) == re.sub('\s+', '', s2.strip())
True

The last and final method is to use a translation table. This solution is an interesting alternative to regex.

>>> table = str.maketrans({' ': None})

>>> table
{32: None}

>>> s1.translate(table)
'Hey,Ireallylikethispost.'

>>> s2.translate(table)
'Hey,Ireallylikethispost.'

>>> s1.translate(table) == s2.translate(table)
True

A nice thing about this method is that it allows removing not only spaces but other chars such as punctuation as well.

>>> import string

>>> table = str.maketrans(dict.fromkeys(string.punctuation + ' '))

>>> s1.translate(table)
'HeyIreallylikethispost'

>>> s2.translate(table)
'HeyIreallylikethispost'

>>> s1.translate(table) == s2.translate(table)
True

How to compare two strings for similarity (fuzzy string matching)

Another popular string comparison use case is checking if two strings are almost equal. In this task, we're interested in knowing how similar they are instead of comparing their equality.

To make it easier to understand, consider a scenario when we have two strings and we are willing to ignore misspelling errors. Unfortunately, that's not possible with the == operator.

We can solve this problem in two different ways:

• using the difflib from the standard library
• using an external library such as jellysifh

Using difflib

The difflib in the standard library has a SequenceMatcher class that provides a ratio() method that returns a measure of the string's similarity as a percentage.

Suppose you have two similar strings, say a = "preview", and b = "previeu". The only difference between them is the final letter. Let's imagine that this difference is small enough for you and you want to ignore it.

By using SequenceMatcher.ratio() we can get the percentage in which they are similar and use that number to assert if the two strings are similar enough.

from difflib import SequenceMatcher

>>> a = "preview"

>>> b = "previeu"

>>> SequenceMatcher(a=a, b=b).ratio()
0.8571428571428571

In this example, SequenceMatcher tells us that the two strings are 85% similar. We can then use this number as a threshold and ignore the difference.

>>> def is_string_similar(s1: str, s2: str, threshold: float = 0.8) -> bool
    ...: :
    ...:     return SequenceMatcher(a=s1, b=s2).ratio() > threshold
    ...:

>>> is_string_similar(s1="preview", s2="previeu")
True

>>> is_string_similar(s1="preview", s2="preview")
True

>>> is_string_similar(s1="preview", s2="previewjajdj")
False

There's one problem, though. The threshold depends on the length of the string. For example, two very small strings, say a = "ab" and b = "ac" will be 50% different.

>>> SequenceMatcher(a="ab", b="ac").ratio()
0.5

So, setting up a decent threshold may be tricky. As an alternative, we can try another algorithm, one that the counts transpositions of letters in a string. And the good new is, such an algorithm exists, and that's what we'll see next.

Using Damerau-Levenshtein distance

The Damerau-Levenshtein algorithm counts the minimum number of operations needed to change one string into another.

In another words, it tells how many insertions, deletions or substitutions of a single character; or transposition of two adjacent characters we need to perform so that the two string become equal.

In Python, we can use the function damerau_levenshtein_distance from the jellysifh library.

Let's see what the Damerau-Levenshtein distance is for the last example from the previous section.

>>> import jellyfish

>>> jellyfish.damerau_levenshtein_distance('ab', 'ac')
1

It's 1! So that means to transform "ac" into "ab" we need 1 change. What about the first example?

>>> s1 = "preview"

>>> s2 = "previeu"

>>>  jellyfish.damerau_levenshtein_distance(s1, s2)
1

It's 1 too! And that makes lots of sense, after all we just need to edit the last letter to make them equal.

This way, we can set the threshold based on number of changes instead of ratio.

>>> def are_strings_similar(s1: str, s2: str, threshold: int = 2) -> bool:
    ...:     return jellyfish.damerau_levenshtein_distance(s1, s2) <= threshold
    ...: 

>>> are_strings_similar("ab", "ac")
True

>>> are_strings_similar("ab", "ackiol")
False

>>> are_strings_similar("ab", "cb")
True

>>> are_strings_similar("abcf", "abcd")
True

# this ones are not that similar, but we have a default threshold of 2
>>> are_strings_similar("abcf", "acfg")
True

>>> are_strings_similar("abcf", "acyg")
False

How to compare two strings and return the difference

Sometimes we know in advance that two strings are different and we want to know what makes them different. In other words, we want to obtain their "diff".

In the previous section, we used difflib as a way of telling if two strings were similar enough. This module is actually more powerful than that, and we can use it to compare the strings and show their differences.

The annoying thing is that it requires a list of strings instead of just a single string. Then it returns a generator that you can use to join into a single string and print the difference.

>>> import difflib

>>> d = difflib.Differ()

>>> diff = d.compare(['my string for test'], ['my str for test'])

>>> diff
<generator object Differ.compare at 0x7f27703250b0>

>>> list(diff)
['- my string for test', '?       ---\n', '+ my str for test']

>>> print('\n'.join(diff))
- my string for test
?       ---

+ my str for test

String comparison not working?

In this section, we'll discuss the reasons why your string comparison is not working and how to fix it. The two main reasons based on my experience are:

• using the wrong operator
• having a trailing space or newline

Comparing strings using is instead of ==

This one is very common amongst novice Python developers. It's easy to use the wrong operator, especially when comparing strings.

As we've discussed in this article, only use the is operator if you want to check if the two string are the same instances.

Having a trailing whitespace of newline (\n)

This one is very common when reading a string from the input function. Whenever we use this function to collect information, the user might accidentally add a trailing space.

If you store the result from the input in a variable, you won't easily see the problem.

>>> a = 'hello'

>>> b = input('Enter a word: ')
Enter a word: hello 

>>> a == b
False

>>> a
'hello'

>>> b
'hello '

>>> a == b.strip()
True

The solution here is to strip the whitespace from the string the user enters and then compare it. You can do it to whatever input source you don't trust.

Conclusion

In this guide, we saw 8 different ways of comparing strings in Python and two most common mistakes. We saw how we can leverage different operations to perform string comparison and how to use external libraries to do string fuzzy matching.

Key takeaways:

• Use the == and != operators to compare two strings for equality
• Use the is operator to check if two strings are the same instance
• Use the <, >, <=, and >= operators to compare strings alphabetically
• Use str.casefold() to compare two string ignoring the case
• Trim strings using native methods or regex to ignore whitespaces when performing string comparison
• Use difflib or jellyfish to check if two strings are almost equal (fuzzy matching)
• Use difflib to to compare two strings and return the difference
• String comparison is not working? Check for trailing or leading spaces, or understand if you are using the right operator for the job

That's it for today, and I hope you learned something new. See you next time!

Other posts you may like:

• How to Choose Between isdigit(), isdecimal() and isnumeric() in Python

• The Best Way to Compare Two Dictionaries in Python

• The Best Ways to Compare Two Lists in Python

• 15 Easy Ways to Trim a String in Python

• Pylint: How to fix "c0209: formatting a regular string which could be a f-string (consider-using-f-string)"

• How to Implement a Random String Generator With Python

• How to Check If a String Is a Valid URL in Python

• Python F-String: 73 Examples to Help You Master It

This post was originally published at https://miguendes.me

The error was:

script.py:7:8: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)

At first I found it very confusing; my code was the same and it'd been working fine before upgrading it. I decided to dig a little deeper and found this pull request on Pylint's github page.

It turns out, this is a new feature that landed on Pylint 2.11.0.

In this post, you will see 3 different ways to fix this "formatting a regular string which could be a f-string (consider-using-f-string)" error.

How to Fix "formatting a regular string which could be a f-string (consider-using-f-string)"

Before Python introduced f-strings, one could use % or the str.format method to format a string. Even though these methods are still valid, f-strings are slightly preferred now.

As a way of enforcing developers to make this migration, the new Pylint version raises this error when it detects the old way of formatting string.

To fix that you can either:

• replace the old formatting method with a f-string
• ignore the Pylint error

Replacing % or the str.format with a f-string

Let's consider this small script that uses both methods.

name = 'world'

a = 'my hello %s' % name

print(a)

b = 'again this name is {}'.format(name) 

print(b)

If we run Pylint 2.11.0+ on it, we get a few errors:

If it's OK for you to update to f-string, then that’s the recommended way. How you do that depends on how you're formatting your strings but in doubt you can check this article to learn the myriad ways you can use a f-string.

In my case, replacing % and str.format becomes:

name = 'world'

a = f'my hello {name}'

print(a)

b = f'again this name is {name}'

print(b)

If we re-run Pylint, we get:

In the next section, we'll use flags to disable this error.

Ignoring the error using flags

You can also ignore the error instead of converting it to f-strings. To do that, you can either add a disabling flag at the top of the python file, or disable it line-by-line.

Ignoring all errors in the file

When you place this flag at the very top of your Python file, Pylint ignores that error across the whole file.

# pylint: disable=consider-using-f-string

name = 'world'

a = 'my hello %s' % name

print(a)

b = 'again this name is {}'.format(name)

print(b)

When we re-run the check, Pylint returns:

Another alternative is to add this flag to a .pylintrc file. This file should be placed at the root of your project, and by doing so, Pylint will ignore the error across the whole project.

A minimal example in this case would be:

# .pylintrc

[MASTER]

disable=consider-using-f-string

After this change, if we re-run Pylint we get:

Ignoring individual errors

To ignore each individual case, place a disabling flag next to the expression #pylint: disable=consider-using-f-stringyou want to ignore.

name = 'world'

a = 'my hello %s' % name #pylint: disable=consider-using-f-string

print(a)

b = 'again this name is {}'.format(name) #pylint: disable=consider-using-f-string

print(b)

The errors will now be suppressed:

Conclusion

That's it for today. I hope this article helped you understand and fix the infamous "c0209: formatting a regular string which could be a f-string (consider-using-f-string)" error.

See you next time!

I remember the struggle to manipulate paths as strings using the os module. I was constantly looking up error messages related to improper path manipulation.

The os module never felt intuitive and ergonomic to me, but my luck changed when pathlib landed in Python 3.4. It was a breath of fresh air, much easier to use, and felt more Pythonic to me.

The only problem was: finding examples on how to use it was hard; the documentation only covered a few use cases. And yes, Python's docs are good, but for newcomers, examples are a must.

Even though the docs are much better now, they don't showcase the module in a problem-solving fashion. That’s why I decided to create this cookbook.

This article is a brain dump of everything I know about pathlib. It's meant to be a reference rather than a linear guide. Feel free to jump around to sections that are more relevant to you.

In this guide, we'll go over dozens of use cases such as:

• how to create (touch) an empty file
• how to convert a path to string
• getting the home directory
• creating new directories, doing it recursively, and dealing with issues when they
• getting the current working directory
• get the file extension from a filename
• get the parent directory of a file or script
• read and write text or binary files
• how to delete files
• how create nested directories
• how to list all files and folders in a directory
• how to list all subdirectories recursively
• how to remove a directory along with its contents

I hope you enjoy!

Table of contents

• What is pathlib in Python?
• The anatomy of a pathlib.Path
• How to convert a path to string
• How to join a path by adding parts or other paths
• Working with directories using pathlib
  • How to get the current working directory (cwd) with pathlib
  • How to get the home directory with pathlib
  • How to expand the initial path component with Path.expanduser()
  • How to list all files and directories
  • Using isdir to list only the directories
  • Getting a list of all subdirectories in the current directory recursively
  • How to recursively iterate through all files
  • How to change directories with Python pathlib
  • How to delete directories with pathlib
  • How to remove a directory along with its contents with pathlib
• Working with files using pathlib
  • How to touch a file and create parent directories
  • How to get the filename from path
  • How to get the file extension from a filename using pathlib
  • How to open a file for reading with pathlib
  • How to read text files with pathlib
  • How to read JSON files from path with pathlib
  • How to write a text file with pathlib
  • How to copy files with pathlib
  • How to delete a file with pathlib
  • How to delete all files in a directory with pathlib
  • How to rename a file using pathlib
  • How to get the parent directory of a file with pathlib
• Conclusion

What is pathlib in Python?

pathlib is a Python module created to make it easier to work with paths in a file system. This module debuted in Python 3.4 and was proposed by the PEP 428.

Prior to Python 3.4, the os module from the standard library was the go to module to handle paths. os provides several functions that manipulate paths represented as plain Python strings. For example, to join two paths using os, one can use theos.path.join function.

>>> import os
>>> os.path.join('/home/user', 'projects')
'/home/user/projects'

>>> os.path.expanduser('~')
'C:\\Users\\Miguel'

>>> home = os.path.expanduser('~')

>>> os.path.join(home, 'projects')
'C:\\Users\\Miguel\\projects'

Representing paths as strings encourages inexperienced Python developers to perform common path operations using string method. For example, joining paths with + instead of using os.path.join(), which can lead to subtle bugs and make the code hard to reuse across multiple platforms.

Moreover, if you want the path operations to be platform agnostic, you will need multiple calls to various os functions such as os.path.dirname(), os.path.basename(), and others.

In an attempt to fix these issues, Python 3.4 incorporated the pathlib module. It provides a high-level abstraction that works well under POSIX systems, such as Linux as well as Windows. It abstracts way the path's representation and provides the operations as methods.

The anatomy of a pathlib.Path

To make it easier to understand the basics components of a Path, in this section we'll their basic components.

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/blog/config.tar.gz')

>>> path.drive
'/'

>>> path.root
'/'

>>> path.anchor
'/'

>>> path.parent
PosixPath('/home/miguel/projects/blog')

>>> path.name
'config.tar.gz'

>>> path.stem
'config.tar'

>>> path.suffix
'.gz'

>>> path.suffixes
['.tar', '.gz']

>>> from pathlib import Path

>>> path = Path(r'C:/Users/Miguel/projects/blog/config.tar.gz')

>>> path.drive
'C:'

>>> path.root
'/'

>>> path.anchor
'C:/'

>>> path.parent
WindowsPath('C:/Users/Miguel/projects/blog')

>>> path.name
'config.tar.gz'

>>> path.stem
'config.tar'

>>> path.suffix
'.gz'

>>> path.suffixes
['.tar', '.gz']

How to convert a path to string

pathlib implements the magic __str__ method, and we can use it convert a path to string. Having this method implemented means you can get its string representation by passing it to the str constructor, like in the example below.

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/tutorial')

>>> str(path)
'/home/miguel/projects/tutorial'

>>> repr(path)
"PosixPath('/home/miguel/projects/blog/config.tar.gz')"

The example above illustrates a PosixPath, but you can also convert a WindowsPath to string using the same mechanism.

>>> from pathlib import Path

>>> path = Path(r'C:/Users/Miguel/projects/blog/config.tar.gz')

# when we convert a WindowsPath to string, Python adds backslashes
>>> str(path)
'C:\\Users\\Miguel\\projects\\blog\\config.tar.gz'

# whereas repr returns the path with forward slashes as it is represented on Windows
>>> repr(path)
"WindowsPath('C:/Users/Miguel/projects/blog/config.tar.gz')"

How to join a path by adding parts or other paths

One of the things I like the most about pathlib is how easy it is to join two or more paths, or parts. There are three main ways you can do that:

• you can pass all the individual parts of a path to the constructor
• use the .joinpath method
• use the / operator

>>> from pathlib import Path

# pass all the parts to the constructor
>>> Path('.', 'projects', 'python', 'source')
PosixPath('projects/python/source')

# Using the / operator to join another path object
>>> Path('.', 'projects', 'python') / Path('source')
PosixPath('projects/python/source')

# Using the / operator to join another a string
>>> Path('.', 'projects', 'python') / 'source'
PosixPath('projects/python/source')

# Using the joinpath method
>>> Path('.', 'projects', 'python').joinpath('source')
PosixPath('projects/python/source')

On Windows, Path returns a WindowsPath instead, but it works the same way as in Linux.

>>> Path('.', 'projects', 'python', 'source')
WindowsPath('projects/python/source')

>>> Path('.', 'projects', 'python') / Path('source')
WindowsPath('projects/python/source')

>>> Path('.', 'projects', 'python') / 'source'
WindowsPath('projects/python/source')

>>> Path('.', 'projects', 'python').joinpath('source')
WindowsPath('projects/python/source')

Working with directories using pathlib

In this section, we'll see how we can traverse, or walk, through directories with pathlib. And when it comes to navigating folders, there many things we can do, such as:

• getting the current working directory
• getting the home directory
• expanding the home directory
• creating new directories, doing it recursively, and dealing with issues when they already exist
• how create nested directories
• listing all files and folders in a directory
• listing only folders in a directory
• listing only the files in a directory
• getting the number of files in a directory
• listing all subdirectories recursively
• listing all files in a directory and subdirectories recursively
• recursively listing all files with a given extension or pattern
• changing current working directories
• removing an empty directory
• removing a directory along with its contents

How to get the current working directory (cwd) with pathlib

The pathlib module provides a classmethod Path.cwd() to get the current working directory in Python. It returns a PosixPath instance on Linux, or other Unix systems such as macOS or OpenBSD. Under the hood, Path.cwd() is just a wrapper for the classic os.getcwd().

>>> from pathlib import Path

>>> Path.cwd()
PosixPath('/home/miguel/Desktop/pathlib')

On Windows, it returns a WindowsPath.

>>> from pathlib import Path

>>> Path.cwd()
>>> WindowsPath('C:/Users/Miguel/pathlib')

You can also print it by converting it to string using a f-string, for example.

>>> from pathlib import Path

>>> print(f'This is the current directory: {Path.cwd()}')
This is the current directory: /home/miguel/Desktop/pathlib

PS: If you

How to get the home directory with pathlib

When pathlib arrived in Python 3.4, a Path had no method for navigating to the home directory. This changed on Python 3.5, with the inclusion of the Path.home() method.

In Python 3.4, one has to use os.path.expanduser, which is awkward and unintuitive.

# In python 3.4
>>> import pathlib, os
>>> pathlib.Path(os.path.expanduser("~"))
PosixPath('/home/miguel')

From Python 3.5 onwards, you just call Path.home().

# In Python 3.5+
>>> import pathlib

>>> pathlib.Path.home()
PosixPath('/home/miguel')

Path.home() also works well on Windows.

>>> import pathlib

>>> pathlib.Path.home()
WindowsPath('C:/Users/Miguel')

How to expand the initial path component with Path.expanduser()

In Unix systems, the home directory can be expanded using ~ ( tilde symbol). For example, this allows us to represent full paths like this: /home/miguel/Desktop as just: ~/Desktop/.

>>> from pathlib import Path

>>> path = Path('~/Desktop/')
>>> path.expanduser()
PosixPath('/home/miguel/Desktop')

Despite being more popular on Unix systems, this representation also works on Windows.

>>> path = Path('~/projects')

>>> path.expanduser()
WindowsPath('C:/Users/Miguel/projects')

>>> path.expanduser().exists()
True

What's the opposite of os.path.expanduser()?

Unfortunately, the pathlib module doesn't have any method to do the inverse operation. If you want to condense the expanded path back to its shorter version, you need to get the path relative to your home directory using Path.relative_to, and place the ~ in front of it.

>>> from pathlib import Path

>>> path = Path('~/Desktop/')
>>> expanded_path = path.expanduser()
>>> expanded_path
PosixPath('/home/miguel/Desktop')
>>> '~' / expanded_path.relative_to(Path.home())
PosixPath('~/Desktop')

Creating directories with pathlib

A directory is nothing more than a location for storing files and other directories, also called folders. pathlib.Path comes with a method to create new directories named Path.mkdir().

This method takes three arguments:

• mode: Used to determine the file mode and access flags
• parents: Similar to the mkdir -p command in Unix systems. Default to False which means it raises errors if there's the parent is missing, or if the directory is already created. When it's True, pathlib.mkdir creates the missing parent directories.
• exist_ok: Defaults to False and raises FileExistsError if the directory being created already exists. When you set it to True, pathlib ignores the error if the last part of the path is not an existing non-directory file.

>>> from pathlib import Path

# lists all files and directories in the current folder
>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
 PosixPath('/home/miguel/path/reports')]

# create a new path instance
>>> path = Path('new_directory')

# only the path instance has been created, but it doesn't exist on disk yet
>>> path.exists()
False

# create path on disk
>>> path.mkdir()

# now it exsists
>>> path.exists()
True

# indeed, it shows up
>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
 PosixPath('/home/miguel/path/reports'),
 PosixPath('/home/miguel/path/new_directory')]

Creating a directory that already exists

When you have a directory path and it already exists, Python raises FileExistsError if you call Path.mkdir() on it. In the previous section, we briefly mentioned that this happens because by default the exist_ok argument is set to False.

>>> from pathlib import Path

>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
 PosixPath('/home/miguel/path/reports'),
 PosixPath('/home/miguel/path/new_directory')]

>>> path = Path('new_directory')

>>> path.exists()
True

>>> path.mkdir()
---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
<ipython-input-25-4b7d1fa6f6eb> in <module>
----> 1 path.mkdir()

~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in mkdir(self, mode, parents, exist_ok)
   1311         try:
-> 1312             self._accessor.mkdir(self, mode)
   1313         except FileNotFoundError:
   1314             if not parents or self.parent == self:

FileExistsError: [Errno 17] File exists: 'new_directory'

To create a folder that already exists, you need to set exist_ok to True. This is useful if you don't want to check using if's or deal with exceptions, for example. Another benefit is that is the directory is not empty, pathlib won't override it.

>>> path = Path('new_directory')

>>> path.exists()
True

>>> path.mkdir(exist_ok=True)

>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
 PosixPath('/home/miguel/path/reports'),
 PosixPath('/home/miguel/path/new_directory')]

>>> (path / 'new_file.txt').touch()

>>> list(path.iterdir())
[PosixPath('new_directory/new_file.txt')]

>>> path.mkdir(exist_ok=True)

# the file is still there, pathlib didn't overwrote it
>>> list(path.iterdir())
[PosixPath('new_directory/new_file.txt')]

How to create parent directories recursively if not exists

Sometimes you might want to create not only a single directory but also a parent and a subdirectory in one go.

The good news is that Path.mkdir() can handle situations like this well thanks to its parents argument. When parents is set to True, pathlib.mkdir creates the missing parent directories; this behavior is similar to the mkdir -p command in Unix systems.

>>> from pathlib import Path

>>> path = Path('new_parent_dir/sub_dir')

>>> path.mkdir()
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-35-4b7d1fa6f6eb> in <module>
----> 1 path.mkdir()

~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in mkdir(self, mode, parents, exist_ok)
   1311         try:
-> 1312             self._accessor.mkdir(self, mode)
   1313         except FileNotFoundError:
   1314             if not parents or self.parent == self:

FileNotFoundError: [Errno 2] No such file or directory: 'new_parent_dir/sub_dir'

>>> path.mkdir(parents=True)

>>> path.exists()
True

>>> path.parent
PosixPath('new_parent_dir')

>>> path
PosixPath('new_parent_dir/sub_dir')

How to list all files and directories

There are many ways you can list files in a directory with Python's pathlib. We'll see each one in this section.

To list all files in a directory, including other directories, you can use the Path.iterdir() method. For performance reasons, it returns a generator that you can either use to iterate over it, or just convert to a list for convenience.

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/pathlib')
>>> list(path.iterdir())
[PosixPath('/home/miguel/projects/pathlib/script.py'),
 PosixPath('/home/miguel/projects/pathlib/README.md'),
 PosixPath('/home/miguel/projects/pathlib/tests'),
 PosixPath('/home/miguel/projects/pathlib/src')]

Using isdir to list only the directories

We've seen that iterdir returns a list of Paths. To list only the directories in a folder, you can use the Path.is_dir() method. The example below will get all the folder names inside the directory.

⚠️ WARNING: This example only lists the immediate subdirectories in Python. In the next subsection, we'll see how to list all subdirectories.

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/pathlib')

>>> [p for p in path.iterdir() if p.is_dir()]
[PosixPath('/home/miguel/projects/pathlib/tests'),
 PosixPath('/home/miguel/projects/pathlib/src')]

Getting a list of all subdirectories in the current directory recursively

In this section, we'll see how to navigate in directory and subdirectories. This time we'll use another method from pathlib.Path named glob.

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/pathlib')

>>> [p for p in path.glob('**/*') if p.is_dir()]
[PosixPath('/home/miguel/projects/pathlib/tests'),
 PosixPath('/home/miguel/projects/pathlib/src'),
 PosixPath('/home/miguel/projects/pathlib/src/dir')]

As you see, Path.glob will also print the subdirectory src/dir.

Remembering to pass '**/ to glob() is a bit annoying, but there's a way to simplify this by using Path.rglob().

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/pathlib')

>>> [p for p in path.rglob('*') if p.is_dir()]
[PosixPath('/home/miguel/projects/pathlib/tests'),
 PosixPath('/home/miguel/projects/pathlib/src'),
 PosixPath('/home/miguel/projects/pathlib/src/dir')]

How to list only the files with is_file

Just as pathlib provides a method to check if a path is a directory, it also provides one to check if a path is a file. This method is called Path.is_file(), and you can use to filter out the directories and print all file names in a folder.

>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')

>>> [p for p in path.iterdir() if p.is_file()]
[PosixPath('/home/miguel/projects/pathlib/script.py'),
 PosixPath('/home/miguel/projects/pathlib/README.md')]

⚠️ WARNING: This example only lists the files inside the current directory. In the next subsection, we'll see how to list all files inside the subdirectories as well.

Another nice use case is using Path.iterdir() to count the number of files inside a folder.

>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')

>>> len([p for p in path.iterdir() if p.is_file()])
2

How to recursively iterate through all files

In previous sections, we used Path.rglob() to list all directories recursively, we can do the same for files by filtering the paths using the Path.is_file() method.

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/pathlib')

>>> [p for p in path.rglob('*') if p.is_file()]
[PosixPath('/home/miguel/projects/pathlib/script.py'),
 PosixPath('/home/miguel/projects/pathlib/README.md'),
 PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
 PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]

How to recursively list all files with a given extension or pattern

In the previous example, we list all files in a directory, but what if we want to filter by extension? For that, pathlib.Path has a method named match(), which returns True if matching is successful, and False otherwise.

In the example below, we list all .py files recursively.

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/pathlib')

>>> [p for p in path.rglob('*') if p.is_file() and p.match('*.py')]
[PosixPath('/home/miguel/projects/pathlib/script.py'),
 PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
 PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]

We can use the same trick for other kinds of files. For example, we might want to list all images in a directory or subdirectories.

>>> from pathlib import Path
>>> path = Path('/home/miguel/pictures')

>>> [p for p in path.rglob('*')
         if p.match('*.jpeg') or p.match('*.jpg') or p.match('*.png')
]
[PosixPath('/home/miguel/pictures/dog.png'),
 PosixPath('/home/miguel/pictures/london/sunshine.jpg'),
 PosixPath('/home/miguel/pictures/london/building.jpeg')]

We can actually simplify it even further, we can use only Path.glob and Path.rglob to matching. (Thanks to u/laundmo and u/SquareRootsi for pointing out!)

>>> from pathlib import Path

>>> path = Path('/home/miguel/projects/pathlib')

>>> list(path.rglob('*.py'))
[PosixPath('/home/miguel/projects/pathlib/script.py'),
 PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
 PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]

>>> list(path.glob('*.py'))
[PosixPath('/home/miguel/projects/pathlib/script.py')]

>>> list(path.glob('**/*.py'))
[PosixPath('/home/miguel/projects/pathlib/script.py'),
 PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
 PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]

How to change directories with Python pathlib

Unfortunately, pathlib has no built-in method to change directories. However, it is possible to combine it with the os.chdir() function, and use it to change the current directory to a different one.

⚠️ WARNING: For versions prior to 3.6, os.chdir only accepts paths as string.

>>> import pathlib

>>> pathlib.Path.cwd()
PosixPath('/home/miguel')

>>> target_dir = '/home'

>>> os.chdir(target_dir)

>>> pathlib.Path.cwd()
PosixPath('/home')

How to delete directories with pathlib

Deleting directories using pathlib depends on if the folder is empty or not. To delete an empty directory, we can use the Path.rmdir()method.

>>> from pathlib import Path

>>> path = Path('new_empty_dir')

>>> path.mkdir()

>>> path.exists()
True

>>> path.rmdir()

>>> path.exists()
False

If we put some file or other directory inside and try to delete, Path.rmdir() raises an error.

>>> from pathlib import Path

>>> path = Path('non_empty_dir')

>>> path.mkdir()

>>> (path / 'file.txt').touch()

>>> path
PosixPath('non_empty_dir')

>>> path.exists()
True

>>> path.rmdir()
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-64-00bf20b27a59> in <module>
----> 1 path.rmdir()

~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in rmdir(self)
   1350         Remove this directory.  The directory must be empty.
                      ...
-> 1352         self._accessor.rmdir(self)
   1353
   1354     def lstat(self):

OSError: [Errno 39] Directory not empty: 'non_empty_dir'

Now, the question is: how to delete non-empty directories with pathlib?

This is what we'll see next.

How to remove a directory along with its contents with pathlib

To delete a non-empty directory, we need to remove its contents, everything.

To do that with pathlib, we need to create a function that uses Path.iterdir() to walk or traverse the directory and:

• if the path is a file, we call Path.unlink()
• otherwise, we call the function recursively. When there are no more files, that is, when the folder is empty, just call Path.rmdir()

Let's use the following example of a non empty directory with nested folder and files in it.

$ tree /home/miguel/Desktop/blog/pathlib/sandbox/
/home/miguel/Desktop/blog/pathlib/sandbox/
├── article.txt
└── reports
    ├── another_nested
    │   └── some_file.png
    └── article.txt

2 directories, 3 files

To remove it we can use the following recursive function.

>>> from pathlib import Path

>>> def remove_all(root: Path):
         for path in root.iterdir():
             if path.is_file():
                 print(f'Deleting the file: {path}')
                 path.unlink()
             else:
                 remove_all(path)
         print(f'Deleting the empty dir: {root}')
         root.rmdir()

Then, we invoke it for the root directory, inclusive.

>>> from pathlib import Path

>>> root = Path('/home/miguel/Desktop/blog/pathlib/sandbox')
>>> root
PosixPath('/home/miguel/Desktop/blog/pathlib/sandbox')

>>> root.exists()
True

>>> remove_all(root)
Deleting the file: /home/miguel/Desktop/blog/pathlib/sandbox/reports/another_nested/some_file.png
Deleting the empty dir: /home/miguel/Desktop/blog/pathlib/sandbox/reports/another_nested
Deleting the file: /home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt
Deleting the empty dir: /home/miguel/Desktop/blog/pathlib/sandbox/reports
Deleting the file: /home/miguel/Desktop/blog/pathlib/sandbox/article.txt
Deleting the empty dir: /home/miguel/Desktop/blog/pathlib/sandbox

>>> root
PosixPath('/home/miguel/Desktop/blog/pathlib/sandbox')

>>> root.exists()
False

I need to be honest, this solution works fine but it's not the most appropriate one. pathlib is not suitable for these kind of operations.

As suggested by u/Rawing7 from reddit, a better approach is to use shutil.rmtree.

>>> from pathlib import Path

>>> import shutil

>>> root = Path('/home/miguel/Desktop/blog/pathlib/sandbox')

>>> root.exists()
True

>>> shutil.rmtree(root)

>>> root.exists()
False

Working with files

In this section, we'll use pathlib to perform operations on a file, for example, we'll see how we can:

• create new files
• copy existing files
• delete files with pathlib
• read and write files with pathlib

Specifically, we'll learn how to:

• create (touch) an empty file
• touch a file with timestamp
• touch a new file and create the parent directories if they don't exist
• get the file name
• get the file extension from a filename
• open a file for reading
• read a text file
• read a JSON file
• read a binary file
• opening all the files in a folder
• write a text file
• write a JSON file
• write bytes data file
• copy an existing file to another directory
• delete a single file
• delete all files in a directory
• rename a file by changing its name, or by adding a new extension
• get the parent directory of a file or script

How to touch (create an empty) a file

pathlib provides a method to create an empty file named Path.touch(). This method is very handy when you need to create a placeholder file if it does not exist.

>>> from pathlib import Path

>>> Path('empty.txt').exists()
False

>>> Path('empty.txt').touch()

>>> Path('empty.txt').exists()
True

Touch a file with timestamp

To create a timestamped empty file, we first need to determine the timestamp format.

One way to do that is to use the time and datetime. First we define a date format, then we use the datetime module to create the datetime object. Then, we use the time.mktime to get back the timestamp.

Once we have the timestamp, we can just use f-strings to build the filename.

>>> import time, datetime

>>> s = '02/03/2021'

>>> d = datetime.datetime.strptime(s, "%d/%m/%Y")

>>> d
datetime.datetime(2021, 3, 2, 0, 0)

>>> d.timetuple()
time.struct_time(tm_year=2021, tm_mon=3, tm_mday=2, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=1, tm_yday=61, tm_isdst=-1)

>>> time.mktime(d.timetuple())
1614643200.0

>>> int(time.mktime(d.timetuple()))
1614643200

>>> from pathlib import Path

>>> Path(f'empty_{int(time.mktime(d.timetuple()))}.txt').exists()
False

>>> Path(f'empty_{int(time.mktime(d.timetuple()))}.txt').touch()

>>> Path(f'empty_{int(time.mktime(d.timetuple()))}.txt').exists()
True

>>> str(Path(f'empty_{int(time.mktime(d.timetuple()))}.txt'))
'empty_1614643200.txt'

How to touch a file and create parent directories

Another common problem when creating empty files is to place them in a directory that doesn't exist yet. The reason is that path.touch() only works if the directory exists. To illustrate that, let's see an example.

>>> from pathlib import Path

>>> Path('path/not_created_yet/empty.txt')
PosixPath('path/not_created_yet/empty.txt')

>>> Path('path/not_created_yet/empty.txt').exists()
False

>>> Path('path/not_created_yet/empty.txt').touch()
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-24-177d43b041e9> in <module>
----> 1 Path('path/not_created_yet/empty.txt').touch()

~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in touch(self, mode, exist_ok)
   1302         if not exist_ok:
   1303             flags |= os.O_EXCL
-> 1304         fd = self._raw_open(flags, mode)
   1305         os.close(fd)
   1306

~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in _raw_open(self, flags, mode)
   1114         as os.open() does.
                      ...
-> 1116         return self._accessor.open(self, flags, mode)
   1117
   1118     # Public API

FileNotFoundError: [Errno 2] No such file or directory: 'path/not_created_yet/empty.txt'

If the target directory does not exist, pathlib raises FileNotFoundError. To fix that we need to create the directory first, the simplest way, as described in the "creating directories" section, is to use the Path.mkdir(parents=True, exist_ok=True). This method creates an empty directory including all parent directories.

>>> from pathlib import Path

>>> Path('path/not_created_yet/empty.txt').exists()
False

# let's create the empty folder first
>>> folder = Path('path/not_created_yet/')

# it doesn't exist yet
>>> folder.exists()
False

# create it
>>> folder.mkdir(parents=True, exist_ok=True)

>>> folder.exists()
True

# the folder exists, but we still need to create the empty file
>>> Path('path/not_created_yet/empty.txt').exists()
False

# create it as usual using pathlib touch
>>> Path('path/not_created_yet/empty.txt').touch()

# verify it exists
>>> Path('path/not_created_yet/empty.txt').exists()
True

How to get the filename from path

A Path comes with not only method but also properties. One of them is the Path.name, which as the name implies, returns the filename of the path. This property ignores the parent directories, and return only the file name including the extension.

>>> from pathlib import Path

>>> picture = Path('/home/miguel/Desktop/profile.png')

>>> picture.name
'profile.png'

How to get the filename without the extension

Sometimes, you might need to retrieve the file name without the extension. A natural way of doing this would be splitting the string on the dot. However, pathlib.Path comes with another helper property named Path.stem, which returns the final component of the path, without the extension.

>>> from pathlib import Path

>>> picture = Path('/home/miguel/Desktop/profile.png')

>>> picture.stem
'profile'

How to get the file extension from a filename using pathlib

If the Path.stem property returns the filename excluding the extension, how can we do the opposite? How to retrieve only the extension?

We can do that using the Path.suffix property.

>>> from pathlib import Path

>>> picture = Path('/home/miguel/Desktop/profile.png')

>>> picture.suffix
'.png'

Some files, such as .tar.gz has two parts as extension, and Path.suffix will return only the last part. To get the whole extension, you need the property Path.suffixes.

This property returns a list of all suffixes for that path. We can then use it to join the list into a single string.

>>> backup = Path('/home/miguel/Desktop/photos.tar.gz')

>>> backup.suffix
'.gz'

>>> backup.suffixes
['.tar', '.gz']

>>> ''.join(backup.suffixes)
'.tar.gz'

How to open a file for reading with pathlib

Another great feature from pathlib is the ability to open a file pointed to by the path. The behavior is similar to the built-in open() function. In fact, it accepts pretty much the same parameters.

>>> from pathlib import Path

>>> p = Path('/home/miguel/Desktop/blog/pathlib/recipe.txt')

# open the file
>>> f = p.open()

# read it
>>> lines = f.readlines()

>>> print(lines)
['1. Boil water. \n', '2. Warm up teapot. ...\n', '3. Put tea into teapot and add hot water.\n', '4. Cover teapot and steep tea for 5 minutes.\n', '5. Strain tea solids and pour hot tea into tea cups.\n']

# then make sure to close the file descriptor
>>> f.close()

# or use a context manager, and read the file in one go
>>> with p.open() as f:
             lines = f.readlines()

>>> print(lines)
['1. Boil water. \n', '2. Warm up teapot. ...\n', '3. Put tea into teapot and add hot water.\n', '4. Cover teapot and steep tea for 5 minutes.\n', '5. Strain tea solids and pour hot tea into tea cups.\n']

# you can also read the whole content as string
>>> with p.open() as f:
             content = f.read()


>>> print(content)
1. Boil water.
2. Warm up teapot. ...
3. Put tea into teapot and add hot water.
4. Cover teapot and steep tea for 5 minutes.
5. Strain tea solids and pour hot tea into tea cups.

How to read text files with pathlib

In the previous section, we used the Path.open() method and file.read() function to read the contents of the text file as a string. Even though it works just fine, you still need to close the file or using the with keyword to close it automatically.

pathlib comes with a .read_text() method that does that for you, which is much more convenient.

>>> from pathlib import Path

# just call '.read_text()', no need to close the file
>>> content = p.read_text()

>>> print(content)
1. Boil water.
2. Warm up teapot. ...
3. Put tea into teapot and add hot water.
4. Cover teapot and steep tea for 5 minutes.
5. Strain tea solids and pour hot tea into tea cups.

The file is opened and then closed. The optional parameters have the same meaning as in open(). pathlib docs

How to read JSON files from path with pathlib

A JSON file a nothing more than a text file structured according to the JSON specification. To read a JSON, we can open the path for reading—as we do for text files—and use json.loads() function from the the json module.

>>> import json
>>> from pathlib import Path

>>> response = Path('./jsons/response.json')

>>> with response.open() as f:
        resp = json.load(f)

>>> resp
{'name': 'remi', 'age': 28}

How to read binary files with pathlib

At this point, if you know how to read a text file, then you reading binary files will be easy. We can do this two ways:

• with the Path.open() method passing the flags rb
• with the Path.read_bytes() method

Let's start with the first method.

>>> from pathlib import Path

>>> picture = Path('/home/miguel/Desktop/profile.png')

# open the file
>>> f = picture.open()

# read it
>>> image_bytes = f.read()

>>> print(image_bytes)
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01R\x00\x00\x01p\x08\x02\x00\x00\x00e\xd3d\x85\x00\x00\x00\x03sBIT\x08\x08\x08\xdb\xe1O\xe0\x00\x00\x00\x10tEXtSoftware\x00Shutterc\x82\xd0\t\x00\x00 \x00IDATx\xda\xd4\xbdkw\x1cY\x92\x1ch\xe6~#2\x13\xe0\xa3\xaa\xbbg
...  [OMITTED] ....
0e\xe5\x88\xfc\x7fa\x1a\xc2p\x17\xf0N\xad\x00\x00\x00\x00IEND\xaeB`\x82'

# then make sure to close the file descriptor
>>> f.close()

# or use a context manager, and read the file in one go
>>> with p.open('rb') as f:
            image_bytes = f.read()

>>> print(image_bytes)
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01R\x00\x00\x01p\x08\x02\x00\x00\x00e\xd3d\x85\x00\x00\x00\x03sBIT\x08\x08\x08\xdb\xe1O\xe0\x00\x00\x00\x10tEXtSoftware\x00Shutterc\x82\xd0\t\x00\x00 \x00IDATx\xda\xd4\xbdkw\x1cY\x92\x1ch\xe6~#2\x13\xe0\xa3\xaa\xbbg
...  [OMITTED] ....
0e\xe5\x88\xfc\x7fa\x1a\xc2p\x17\xf0N\xad\x00\x00\x00\x00IEND\xaeB`\x82'

And just like Path.read_text(), pathlib comes with a .read_bytes() method that can open and close the file for you.

>>> from pathlib import Path

# just call '.read_bytes()', no need to close the file
>>> picture = Path('/home/miguel/Desktop/profile.png')

>>> picture.read_bytes()
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01R\x00\x00\x01p\x08\x02\x00\x00\x00e\xd3d\x85\x00\x00\x00\x03sBIT\x08\x08\x08\xdb\xe1O\xe0\x00\x00\x00\x10tEXtSoftware\x00Shutterc\x82\xd0\t\x00\x00 \x00IDATx\xda\xd4\xbdkw\x1cY\x92\x1ch\xe6~#2\x13\xe0\xa3\xaa\xbbg
...  [OMITTED] ....
0e\xe5\x88\xfc\x7fa\x1a\xc2p\x17\xf0N\xad\x00\x00\x00\x00IEND\xaeB`\x82'

How to open all files in a directory in Python

Let's image you need a Python script to search all files in a directory and open them all. Maybe you want to filter by extension, or you want to do it recursively. If you've been following this guide from the beginning, you now know how to use the Path.iterdir() method.

To open all files in a directory, we can combine Path.iterdir() with Path.is_file().

>>> import pathlib
>>> for i in range(2):
        print(i)
# we can use iterdir to traverse all paths in a directory
>>> for path in pathlib.Path("my_images").iterdir():
        # if the path is a file, then we open it
        if path.is_file():
            with path.open(path, "rb") as f:
                image_bytes = f.read()
                load_image_from_bytes(image_bytes)

If you need to do it recursively, we can use Path.rglob() instead of Path.iterdir().

>>> import pathlib
# we can use rglob to walk nested directories
>>> for path in pathlib.Path("my_images").rglob('*'):
        # if the path is a file, then we open it
        if path.is_file():
            with path.open(path, "rb") as f:
                image_bytes = f.read()
                load_image_from_bytes(image_bytes)

How to write a text file with pathlib

In previous sections, we saw how to read text files using Path.read_text().

To write a text file to disk, pathlib comes with a Path.write_text(). The benefits of using this method is that it writes the data and close the file for you, and the optional parameters have the same meaning as in open().

⚠️ WARNING: If you open an existing file, Path.write_text() will overwrite it.

>>> import pathlib

>>> file_path = pathlib.Path('/home/miguel/Desktop/blog/recipe.txt')

>>> recipe_txt = '''
    1. Boil water.
    2. Warm up teapot. ...
    3. Put tea into teapot and add hot water.
    4. Cover teapot and steep tea for 5 minutes.
    5. Strain tea solids and pour hot tea into tea cups.
    '''

>>> file_path.exists()
False

>>> file_path.write_text(recipe_txt)
180

>>> content = file_path.read_text()

>>> print(content)

1. Boil water.
2. Warm up teapot. ...
3. Put tea into teapot and add hot water.
4. Cover teapot and steep tea for 5 minutes.
5. Strain tea solids and pour hot tea into tea cups.

How to write JSON files to path with pathlib

Python represents JSON objects as plain dictionaries, to write them to a file as JSON using pathlib, we need to combine the json.dump function and Path.open(), the same way we did to read a JSON from disk.

>>> import json

>>> import pathlib

>>> resp = {'name': 'remi', 'age': 28}

>>> response = pathlib.Path('./response.json')

>>> response.exists()
False

>>> with response.open('w') as f:
         json.dump(resp, f)


>>> response.read_text()
'{"name": "remi", "age": 28}'

How to write bytes data to a file

To write bytes to a file, we can use either Path.open() method passing the flags wb or Path.write_bytes() method.

>>> from pathlib import Path

>>> image_path_1 = Path('./profile.png')

>>> image_bytes = b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00 [OMITTED] \x00I
     END\xaeB`\x82'

>>> with image_path_1.open('wb') as f:
         f.write(image_bytes)


>>> image_path_1.read_bytes()
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00 [OMITTED] \x00IEND\xaeB`\x82'

>>> image_path_2 = Path('./profile_2.png')

>>> image_path_2.exists()
False

>>> image_path_2.write_bytes(image_bytes)
37

>>> image_path_2.read_bytes()
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00 [OMITTED] \x00IEND\xaeB`\x82'

How to copy files with pathlib

pathlib cannot copy files. However, if we have a file represented by a path that doesn't mean we can't copy it. There are two different ways of doing that:

• using the shutil module
• using the Path.read_bytes() and Path.write_bytes() methods

For the first alternative, we use the shutil.copyfile(src, dst) function and pass the source and destination path.

>>> import pathlib, shutil

>>> src = Path('/home/miguel/Desktop/blog/pathlib/sandbox/article.txt')

>>> src.exists()
True

>>> dst = Path('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')

>>> dst.exists()
>>> False

>>> shutil.copyfile(src, dst)
PosixPath('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')

>>> dst.exists()
True

>>> dst.read_text()
'This is \n\nan \n\ninteresting article.\n'

>>> dst.read_text() == src.read_text()
True

⚠️ WARNING: shutil prior to Python 3.6 cannot handle Path instances. You need to convert the path to string first.

The second method involves copying the whole file, then writing it to another destination.

>>> import pathlib, shutil

>>> src = Path('/home/miguel/Desktop/blog/pathlib/sandbox/article.txt')

>>> src.exists()
True

>>> dst = Path('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')

>>> dst.exists()
False

>>> dst.write_bytes(src.read_bytes())
36

>>> dst.exists()
True

>>> dst.read_text()
'This is \n\nan \n\ninteresting article.\n'

>>> dst.read_text() == src.read_text()
True

⚠️ WARNING: This method will overwrite the destination path. If that's a concern, it's advisable either to check if the file exists first, or to open the file in writing mode using the x flag. This flag will open the file exclusive creation, thus failing with FileExistsError if the file already exists.

Another downside of this approach is that it loads the file to memory. If the file is big, prefer shutil.copyfileobj. It supports buffering and can read the file in chunks, thus avoiding uncontrolled memory consumption.

>>> import pathlib, shutil

>>> src = Path('/home/miguel/Desktop/blog/pathlib/sandbox/article.txt')
>>> dst = Path('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')

>>> if not dst.exists():
         dst.write_bytes(src.read_bytes())
     else:
         print('File already exists, aborting...')

File already exists, aborting...

>>> with dst.open('xb') as f:
         f.write(src.read_bytes())

---------------------------------------------------------------------------
FileExistsError                           Traceback (most recent call last)
<ipython-input-25-1974c5808b1a> in <module>
----> 1 with dst.open('xb') as f:
      2     f.write(src.read_bytes())
      3

How to delete a file with pathlib

You can remove a file or symbolic link with the Path.unlink() method.

>>> from pathlib import Path

>>> Path('path/reports/report.csv').touch()

>>> path = Path('path/reports/report.csv')

>>> path.exists()
True

>>> path.unlink()

>>> path.exists()
False

As of Python 3.8, this method takes one argument named missing_ok. By default, missing_ok is set to False, which means it will raise an FileNotFoundError error if the file doesn't exist.

>>> path = Path('path/reports/report.csv')

>>> path.exists()
False

>>> path.unlink()
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-8eea53121d7f> in <module>
----> 1 path.unlink()

~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in unlink(self, missing_ok)
   1342         try:
-> 1343             self._accessor.unlink(self)
   1344         except FileNotFoundError:
   1345             if not missing_ok:

FileNotFoundError: [Errno 2] No such file or directory: 'path/reports/report.csv'

# when missing_ok is True, no error is raised
>>> path.unlink(missing_ok=True)

How to delete all files in a directory with pathlib

To remove all files in a folder, we need to traverse it and check if the path is a file, and if so, call Path.unlink() on it as we saw in the previous section.

To walk over the contents of a directory, we can use Path.iterdir(). Let's consider the following directory.

$ tree /home/miguel/path/
/home/miguel/path/
├── jsons
│   └── response.json
├── new_parent_dir
│   └── sub_dir
├── non_empty_dir
│   └── file.txt
├── not_created_yet
│   └── empty.txt
├── number.csv
├── photo_1.png
├── report.md
└── reports

This method only deletes the immediate files under the current directory, so it is not recursive.

>>> import pathlib

>>> path = pathlib.Path('/home/miguel/path')

>>> list(path.iterdir())
Out[5]:
[PosixPath('/home/miguel/path/jsons'),
 PosixPath('/home/miguel/path/non_empty_dir'),
 PosixPath('/home/miguel/path/not_created_yet'),
 PosixPath('/home/miguel/path/reports'),
 PosixPath('/home/miguel/path/photo_1.png'),
 PosixPath('/home/miguel/path/number.csv'),
 PosixPath('/home/miguel/path/new_parent_dir'),
 PosixPath('/home/miguel/path/report.md')]

>>> for p in path.iterdir():
        if p.is_file():
            p.unlink()


>>> list(path.iterdir())
[PosixPath('/home/miguel/path/jsons'),
 PosixPath('/home/miguel/path/non_empty_dir'),
 PosixPath('/home/miguel/path/not_created_yet'),
 PosixPath('/home/miguel/path/reports'),
 PosixPath('/home/miguel/path/new_parent_dir')]

How to rename a file using pathlib

pathlib also comes with a method to rename files called Path.rename(target). It takes a target file path and renames the source to the target. As of Python 3.8, Path.rename() returns the new Path instance.

>>> from pathlib import Path

>>> src_file = Path('recipe.txt')

>>> src_file.open('w').write('An delicious recipe')
19
>>> src_file.read_text()
'An delicious recipe'

>>> target = Path('new_recipe.txt')

>>> src_file.rename(target)
PosixPath('new_recipe.txt')

>>> src_file
PosixPath('recipe.txt')

>>> src_file.exists()
False

>>> target.read_text()
'An delicious recipe'

Renaming only file extension

If all you want is to change the file extension to something else, for example, change from .txt to .md, you can use Path.rename(target) in conjunction with Path.with_suffix(suffix) method, which does the following:

• appends a new suffix, if the original path doesn’t have one
• removes the suffix, if the supplied suffix is an empty string

Let's see an example where we change our recipe file from plain text .txt to markdown .md.

>>> from pathlib import Path

>>> src_file = Path('recipe.txt')

>>> src_file.open('w').write('An delicious recipe')
19

>>> new_src_file = src_file.rename(src_file.with_suffix('.md'))

>>> new_src_file
PosixPath('recipe.md')

>>> src_file.exists()
False

>>> new_src_file.exists()
True

>>> new_src_file.read_text()
'An delicious recipe'

>>> removed_extension_file = new_src_file.rename(src_file.with_suffix(''))

>>> removed_extension_file
PosixPath('recipe')

>>> removed_extension_file.read_text()
'An delicious recipe'

How to get the parent directory of a file with pathlib

Sometimes we want to get the name of the directory a file belongs to. You can get that through a Path property named parent. This property represents the logical parent of the path, which means it returns the parent of a file or directory.

>>> from pathlib import Path

>>> path = Path('path/reports/report.csv')

>>> path.exists()
False

>>> parent_dir = path.parent

>>> parent_dir
PosixPath('path/reports')

>>> parent_dir.parent
PosixPath('path')

Conclusion

That was a lot to learn, and I hope you enjoyed it just as I enjoyed writing it.

pathlib has been part of the standard library since Python 3.4 and it's a great solution when it comes to handling paths.

In this guide, we covered the most important use cases in which pathlib shines through tons of examples.

I hope this cookbook is useful to you, and see you next time.

Other posts you may like:

• Find the Current Working Directory in Python

• The Best Ways to Compare Two Lists in Python

• Python F-String: 73 Examples to Help You Master It

See you next time!

This article was originally published at https://miguendes.me

One such feature is the autouse fixtures, a.k.a xUnit setup on steroids. They are a special type of fixture that gets invoked automatically, and its main use case is to act as a setup/teardown function.

Another use case is to perform some task, like mocking an external dependency, that must happen before every test.

For example, suppose you have a set of functions that execute HTTP calls. For each one, you provide a test. To ensure your test doesn't call the real API, we can mock the call using a library such responses.

However, if you want one of the tests to call the API, as in an integration test, then you'll have to disable the autouse fixture. And that's what we're going to see today.

In this post, we'll learn a simple technique to disable autouse fixtures for one or more tests.

Table of Contents

1. pytest's autouse fixture - example
2. Disabling an autouse fixture
3. Conclusion

pytest Fixture Autouse - Example

In this section, we'll build an example to illustrate the usage of autouse fixtures and how to we can disable them when necessary.

For this example, we'll write some tests that mock the random module.

Consider the following case where we'll be building a random password generator. The function takes a password length and returns a random string of size length. And to do that, it uses random.choices to randomly pick k chars from a seed string called all_chars.

# file: autouse/__init__.py

import random
import string


def get_random_password(length: int = 20) -> str:
    """
    Generates a random password with up to length chars.
    """

    all_chars = string.ascii_letters + string.digits + string.punctuation
    return ''.join(random.choices(all_chars, k=length))

Since we don't control how random.choices picks, we cannot test it in a deterministic way. To make that happen, we can patch random.choices and make it return a fixed list of chars.

You can also set the random.seed to a fixed number before every test run by making it an autouse fixture.

# file: tests/test_random.py

import random

import pytest

from autouse import get_random_password


@pytest.fixture(autouse=True)
def patch_random():
    with unittest.mock.patch('autouse.random.choices') as mocked_choices:
        mocked_choices.return_value = ['a', 'B', 'c', '2']
        yield


def test_mocked_random_char():
    assert get_random_password() == 'aBc2'

The benefits of the autouse fixture are that we don't need to pass it to every test that needs it. And by using yield, you undo the patching after the test finishes, which is great for cleaning up.

If we run this test, it passes just fine.

============================= test session starts ==============================
collecting ... collected 1 item

test_random.py::test_mocked_random_char PASSED                           [100%]

========================= 1 passed, 1 warning in 0.05s =========================

Disabling an autouse fixture

Now, let's say that we want to test the robustness of our random number generator and we want to test that it never generates the same string in a row.

To do that, we need to call the real function, and not patch it. Let's create this test and see what it does.

# file: tests/test_random.py

def test_random_char_does_not_duplicate():
    password_one = get_random_password()
    password_two = get_random_password()
    assert password_one != password_two

But when we run this test, it fails:

test_random.py::test_random_char_does_not_duplicate FAILED               [100%]
test_random.py:18 (test_random_char_does_not_duplicate)
def test_random_char_does_not_duplicate():
        password_one = get_random_password()
        password_two = get_random_password()
>       assert password_one != password_two
E       AssertionError: assert 'aBc2' != 'aBc2'

test_random.py:22: AssertionError

The reason is that pytest injects the autouse fixture to every test case within the scope you specified.

Now the question is, how can we disable an autouse fixture for one or more tests in pytest?

One way to do that is to create a custom pytest mark and annotate the test with it. For example:

@pytest.fixture(autouse=True)
def patch_random(request):
    if 'disable_autouse' in request.keywords:
        yield 
    else:
        with unittest.mock.patch('autouse.random.choices') as mocked_choices:
            mocked_choices.return_value = ['a', 'B', 'c', '2']
            yield

In this example, we created a pytest mark called disable_autouse and we annotated the test_random_char_does_not_duplicate test with it.

This mark becomes available in the request fixture. We can pass this request fixture to the autouse one and check if the keyword disable_autouse is in the list of keywords.

When that's the case, we don't mock, just yield, which gives back the control to test_random_char_does_not_duplicate, thus avoiding mocking the random.choices function.

Let's see what happens when we run the test with this mark...

@pytest.mark.disable_autouse
def test_random_char_does_not_duplicate():
    password_one = get_random_password()
    password_two = get_random_password()
    assert password_one != password_two

The test passes, since it's not mocked anymore.

============================= test session starts ==============================
collecting ... collected 1 item

test_random.py::test_random_char_does_not_duplicate PASSED               [100%]

========================= 1 passed, 1 warning in 0.03s =========================

Conclusion

pytest has some great features such as autouse fixture. They make it easier to set up and teardown unit tests but if we ever want to disable it, then things get trickier.

In this post, we learned how to disable autouse fixture in pytest by marking the tests with a custom pytest mark. I hope you enjoyed this article and see you next time.

Other posts you may like:

• Learn how to unit test REST APIs in Python with Pytest by example.

• 7 pytest Features and Plugins That Will Save You Tons of Time

• How to Use Fixtures as Arguments in pytest.mark.parametrize

• How to Check if an Exception Is Raised (or Not) With pytest

• 7 pytest Plugins You Must Definitely Use

• How to Unit Test Complex Data Like Numpy Arrays in Python

References:

Disable autouse fixtures on specific pytest marks

pytest - is there a way to ignore an autouse fixture?

This article was originally published at https://miguendes.me

But... the truth is, you don't need to know every one of them.

In this article, you'll see only the most important techniques, such as stripping leading and trailing spaces (as well as the ones inside the string). You'll also learn how to remove tabs, newlines, carriage return (CRLF), and other characters. And we'll be using nothing more than native methods and regex—no external libraries required!

By the end of this article, you'll have mastered:

• How to trim a string

  • by stripping leading whitespace from the beginning

  • by stripping trailing whitespace from the end

  • by removing spaces the start and end of a string

• How trim newlines

• How trim carriage return (CRLF)

• How trim tabs

• How to trim a combination of characters from a string

• How to remove multiple spaces inside a string

  • by removing only duplicates

  • by removing all spaces

• How to strip a list of strings

• How to strip an (Numpy) array of strings

How to Trim Characters From a String

Trimming a string means deleting certain chars from the start, the end, or both sides of a string. Removing unwanted chars makes it easier to compare strings and can prevent hard to debug issues.

You can remove any kind o character, but usually what we're interested in is deleting blank spaces, new lines, carriage return (CRLF), tabs and other special symbols.

In this section, we're going to see how to remove leading or trailing spaces, blank spaces, newline character, carriage return (CRLF), and tabs.

Stripping Leading Whitespace From Beginning of a String

The str class has a very convenient method to trim leading spaces named str.lstrip, a shorthand for "left-strip", since it trims a string from the left-hand side. You can think of it as a left trim.

>>> '   hello   '.lstrip()
'hello   '

When calling str.lstrip with no arguments, it removes all whitespaces from left to right. But if all you want is to strip the first char, then there are two ways of doing this. The first one assumes that there will always be at least one whitespace in the beginning of the string. If that's the case, then you can just slice it.

>>> s = '  hello'
>>> s = s[1:]
>>> s
' hello'

If there's no guarantee of that, we'll need to check first if the string starts with space.

>>> def strip_first(s: str, ch: str = ' ') -> str:
     if s and s[0] == ch:
         return s[1:]
     return s

>>> strip_first('hello')
'hello'

>>> strip_first('   hello')
 '  hello'

Stripping Trailing Whitespace From End of a String

The way to remove trailing spaces from the end of the string is to use str.rstrip.

This method expects a list of chars and trims the string from the right. It removes all chars that match one of those you passed, and stop as soon as it cannot match anymore. By default, str.rstrip() removes blanks if you don't pass anything to it. You can think of it as a right trim.

>>> '   hello   '.rstrip()
'   hello'
>>> '***hello***'.rstrip('*')
'***hello'

Sometimes you might want to trim only the last character of a string. And we can use the same logic from the previous example. Check if the last char is a space, and use slice to remove it.

>>> def strip_last(s: str, ch: str = ' ') -> str:
     if s and s[-1] == ch:
         return s[:-1]
     return s


>>> strip_last('hello')
'hello'

>>> strip_last('hello ')
'hello'

>>> strip_last('')
''

Removing Spaces From From Start and End of a String

If all you want is to remove whitespaces from start and end of string, str.strip will serve you better.

This method trims both sides of the string. And just like str.lstrip and str.rstrip, if you can pass any combination of chars as argument, it removes them from both ends.

⚠️ WARNING ⚠️: A common misconception is to think that there's a trim() function in Python.

# by default, strip removes whitespaces
>>> '   hello   '.strip()
'hello'
# but you can also strip other character
>>> '***hello***'.strip('*')
'hello'

How to Trim Newlines

We've seen how str.strip can remove blank spaces from both sides of a string. I've also mentioned that this method takes a chars argument that you can use pass a combination of character you want to trim.

To trim line breaks, you can pass \n and it will strip all newlines from both sides of the string.

>>> s = """
... 
... 
...  hello
... 
... 
... """
>>> s
'\n\n\n hello\n\n\n'
>>> s.strip('\n')
' hello'

How to Trim Carriage Return (CRLF)

The Carriage Return (CR), and Line Feed (LF) are nothing more than a newline character. They are represented by the concatenation of \r and \n forming \r\n. This is how Microsoft Windows, Symbian OS and other non-Unix operating systems represent a new line [source].

Removing them from a string is the same as removing the single newline. You feed str.strip with \r\n and method does its job!

>>> s = "  hello world\r\n\r\n"
>>> print(s)
  hello world


>>> s.strip('\r\n')
'  hello world'

How to Trim Tabs

If you are following this guide from the beginning you might already know how to do this. Trimming tabs from a string in Python is the same as other characters, you use str.strip and pass the '\t' string to it.

>>> s = "\t\t\t  hello  world \t"       
>>> s
'\t\t\t  hello  world \t'
>>> print(s)
              hello  world     
>>> s.strip('\t')
'  hello  world '

And that's it!

How to Trim a Combination of Characters From a String

As I mentioned before, str.strip takes as argument a string, not just a single char. This sequence of chars is a combination of all chars you want to remove from the beginning and end of your string.

>>> s = "  \ns hello world \n    s"
>>> s    
'  \ns hello world \n    s'
>>> print(s)

s hello world 
    s
>>> s.strip('\n s')
'hello world'

How to Remove Multiple Spaces Inside a String

Sometimes you want to do more than trimming, let's say you want to remove chars inside the string. There are two ways of doing this: one is to remove only the duplicates; the other is to remove all extra spaces.

Removing Only Duplicates

To remove only the duplicated characters, you can use the regex module re

>>> import re
>>> s = "   Python   is really   a    great language.    "
>>> re.sub("\s+" , " ", s)
' Python is really a great language. '

This method gets rid of all consecutive spaces. What if you want to do not only that, but also trim the string by removing the leading and trailing blanks?

One way is to split the string and then joining then like so:

>>> s = "   Python   is really   a    great language.    "
>>> " ".join(s.split())
'Python is really a great language.'
>>> # This is the same as using regex then stripping the whitespaces
>>> re.sub("\s+" , " ", s).strip()
'Python is really a great language.'

Removing All Spaces

Now, if you want to strip all whitespace in your string, either use regex or call the str.replace method.

Using re (regex module)

>>> import re
>>> s = "   Python   is really   a    great language.    "
>>> re.sub("\s+" , "", s) 
'Pythonisreallyagreatlanguage.'

Using replace

>>> s = "   Python   is really   a    great language.    "
>>> s.replace(' ', '')
'Pythonisreallyagreatlanguage.'

How to Strip a List of Strings

Trimming a list of strings is almost the same as trimming an individual one. The only difference is that you have to iterate over the list, and call str.strip method on each one. You do so by using a list comprehension, for example, to return a new list with all strings trimmed.

>>> lst = ["string1\n", "string2\n", "string3\n"]
>>> [s.strip('\n') for s in lst]
['string1', 'string2', 'string3']

How to Strip an (Numpy) Array of Strings

It's very common to use Numpy for data science tasks due to its performance and ease to use.

If you have a array of strings and want to trim each one of them, Numpy comes with an efficient vectorized implementation of strip.

In fact, it also has .lstrip, .rstrip, .replace, and many other string operations.

The vectorized versions work slightly differently, they are not a method but a function in the numpy.char module. So you must pass the array and the list of chars you want to trim.

>>> import numpy as np
>>> arr = np.array([' helloworld   ', ' hello'])
array([' helloworld   ', ' hello'], dtype='<U7')
>>> np.char.strip(arr, ' ')
array(['helloworld', 'hello'], dtype='<U7')

Conclusion

In this post, you learned several ways of trimming a string in Python, including array of strings. Python allows us to strip leading and trailing characters easily. And if instead of removing the extra chars on each side you want to remove the ones internally, you can count on the regex module. I hope you've found this article helpful and see you next time!

References:

https://stackoverflow.com/questions/761804/how-do-i-trim-whitespace-from-a-string

https://stackoverflow.com/questions/8270092/remove-all-whitespace-in-a-string

https://stackoverflow.com/questions/1546226/is-there-a-simple-way-to-remove-multiple-spaces-in-a-string

Other posts you may like:

• How to Choose Between isdigit(), isdecimal() and isnumeric() in Python

• How to Compare Two Strings in Python (in 8 Easy Ways)

• Pylint: How to fix "c0209: formatting a regular string which could be a f-string (consider-using-f-string)"

• How to Implement a Random String Generator With Python

• How to Check If a String Is a Valid URL in Python

• Python F-String: 73 Examples to Help You Master It

See you next time!

This post was originally published at https://miguendes.me

When processing strings, usually by reading them from some source, you might want to check if the given string is a number. The string class (str) comes with 3 different methods that you can use for that purpose.

Each of them has pros and cons, and distinguishing the difference between them will save you tons of development and debugging time.

In this article, you will:

• learn what str.isdigit(), str.isdecimal(), and str.isnumeric() do, their limitations, how to use them, and when you should use them
• understand the difference difference between isdigit vs isnumeric vs isdecimal
• understand why isdigit,isnumeric, or isdecimal is not working for you
• how to solve common problems that cannot be easily solved with them, such as:
  • how to make sure a float number string is digit
  • how to use isdigit,isnumeric, or isdecimal with negative numbers

Table of Contents

1. How isdigit() Works and When to Use It
2. How isdecimal() Works and When to Use It
3. How isnumeric() Works and When to Use It

4. Solving Common Problems

   4.1. How to Check if Float Numbers Are Digits?

   4.2. How to Check if Negative Numbers Are Digits?

   4.3. Why isdigit Is Not Working for Me?

5. Conclusion

How isdigit() Works and When to Use It

str.isdigit() is the most obvious choice if you want to determine if a string - or a character - is a digit in Python.

According to its documentation, this method returns True if all characters in the string are digits and it's not empty, otherwise it will return False. Let's see some examples:

# all characters in the string are digits
>>> '102030'.isdigit()
True

# 'a' is not a digit
>>> '102030a'.isdigit()
False

# isdigit fails if there's whitespace
>>> ' 102030'.isdigit()
False

# it must be at least one char long
>>> ''.isdigit()
False

# dots '.' are also not digit
>>> '12.5'.isdigit()
False

Unlike many people think, isdigit is not a function but a method in the str, bytes, and bytearray classes.

This works well for these simpler cases, but what happens if a string has a space?

# ' ' (space) is not a digit
In [8]: ' 102030'.isdigit()
Out[8]: False

This fails because the string contains a space in the beginning. As a result, we cannot use it as it is to read from unprocessed sources such as the input() function. You must always remember to preprocess the input before checking with isdigit(). That might be one of the reasons isdigit is not working for you.

>>> a = input('Enter a number: ')
Enter a number: 56 

>>> a
'56 '

>>> a.isdigit()
False

>>> a.strip()
'56'

>>> a.strip().isdigit()
True

Despite this strict behavior, isdigit has some gotchas. If we read the documentation carefully, it says that the method can also handle "superscript digits".

Digits include decimal characters and digits that need special handling, such as the compatibility superscript digits.

But how does that work? Will it return True for strings with superscripts such as 2⁷?

>>> d = '2' + '\u2077'

>>> d
'2⁷'

>>> d.isdigit()
True

# it accepts superscripts only
>>> '⁵'.isdigit()
True

# and superscripts first followed by a number
>>> '⁵5'.isdigit()
True

It turns out it does! You can actually use it with input():

>>> a = input('Enter a number:')
Enter a number:2⁷

>>> a
'2⁷'

>>> a.isdigit()
True

Even though it works well with superscripts, it doesn't handle fractions chars. This method is really about single digits.

# fractions in Unicode are not digits
>>> '⅕'.isdigit()
False

As we can see, str.isdigit() works really well with Unicode characters. If we take a look at the unit test suite for this method, we can see some interesting test cases.

# https://github.com/python/cpython/blob/3.10/Lib/test/test_unicode.py#L704
    def test_isdigit(self):
        super().test_isdigit()
        self.checkequalnofix(True, '\u2460', 'isdigit')
        self.checkequalnofix(False, '\xbc', 'isdigit')
        self.checkequalnofix(True, '\u0660', 'isdigit')

        for ch in ['\U00010401', '\U00010427', '\U00010429', '\U0001044E',
                   '\U0001F40D', '\U0001F46F', '\U00011065']:
            self.assertFalse(ch.isdigit(), '{!a} is not a digit.'.format(ch))
        for ch in ['\U0001D7F6', '\U00011066', '\U000104A0', '\U0001F107']:
            self.assertTrue(ch.isdigit(), '{!a} is a digit.'.format(ch))

The image below shows some of these test cases.

str.isdigit() works really well with numeric Unicode

Unicode characters that don't represent digits are not accepted

Summary of What isdigit Cannot Do

Can it handle whitespace?

No

Can it handle hexadecimal?

No

Does it raise exception?

No

Does it accept negative digits (with minus sign)?

No

When to Use it?

Use str.isdigit when you want to verify that each and every character in a string is a single digit, that is, not punctuation, not a letter, and not negative.

How isdecimal() Works and When to Use It

The str.isdecimal() method is very similar, it returns True if all chars are decimal characters and the string is not empty. This means that superscripts are NOT decimal numbers, thus they'll return False.

>>> '5'.isdecimal()
True

>>> '⁵'.isdecimal()
False

>>> '5⁵'.isdecimal()
False

>>> '-4'.isdecimal()
False

>>> '4.5'.isdecimal()
False

Superscripts are not decimal numbers

isdecimal also accepts Unicode characters that are used to form numbers in base 10 in other languages. For example, the Arabic-Indic digit zero is considered a decimal, as a result '٠'.isdecimal() returns true.

>>> '٠'.isdecimal()
True

Arabic-Indic such as '٠' decimal in base 10

When to Use it?

Use str.isdecimal when you want to verify that each and every character in a string can form a base 10 number. Since punctuation, superscripts, letters, and minus sign are not decimals, they'll return False.

How isnumeric() Works and When to Use It

This one overlaps significantly with isdigit and isdecimal. According to the documentation, isnumeric returns True if all characters string are numeric and must not be empty.

The key difference here is the word numeric. What is the difference between a numeric character and a digit character?

The difference is that a digit is a single Unicode value whereas a numeric character is any Unicode symbol that represents a numeric value, and that includes fractions!

Not only that, isnumeric works well with roman numerals!

Let's see some examples in action.

>>> '⅕'.isdigit()
False

>>> '⅕'.isnumeric()
True

>>> '⁵'.isnumeric()
True

'5⁵'.isnumeric()
True

>>> '-4'.isnumeric()
False

>>> '4.5'.isnumeric()
False

>>> '5 '.isnumeric()
False

# ⅮⅪ in roman numerals in unicode and represent 511 in base10
>>> 'ⅮⅪ'.isnumeric()
True

# Roman numerals are not digits
>>> 'ⅮⅪ'.isdigit()
False

# Ascii letters 'D', 'X', and 'I' are not numeric
>>> 'DXI'.isnumeric()
False

When to Use it?

Use str.isnumeric when you want to verify that each and every character in a string is a valid numeric char, including fractions, superscripts and roman numbers. Since punctuation, letters, and minus sign are not numeric values, they'll evaluate to False.

Solving Common Problems

In this section, we'll see how to fix the most common problems when using isdigit, isnumeric, and isdecimal.

How to Check if Float Numbers Are Digits?

The best way to check that is to try to cast it to float.

It the float constructor doesn't raise any exceptions, then the string is a valid float. This is a pythonic idiom called EAFP (Easier to ask for forgiveness than permission).

def is_float_digit(n: str) -> bool:
     try:
         float(n)
         return True
     except ValueError:
         return False

>>> is_float_digit('23.45')
True

>>> is_float_digit('23.45a')
False

CAUTION: This string method does not work with superscript! The only way to verify that is to replace the '.' and then calling `isdigit()' on it.

>>> is_float_digit('23.45⁵')
False

def is_float_digit_v2(n: str) -> bool:
     return n.replace('.', '', 1).isdigit()

>>> is_float_digit_v2('23.45⁵')
True

How to Check if Negative Numbers Are Digits?

Checking numbers starting with minus sign depend on the target type.

Since we're talking about digits here, it makes sense to assert if the string can be converted to int. This is very similar to the EAFP approach discussed for floats.

However, just like the previous approach, it doesn't handle superscripts.

def is_negative_number_digit(n: str) -> bool:
     try:
         int(n)
         return True
     except ValueError:
         return False

>>> is_negative_number_digit('-2345')
True

>>> is_negative_number_digit('-2345⁵')
False

To fix that, the best way is to strip the minus sign.

def is_negative_number_digit_v2(n: str) -> bool:
     return n.lstrip('-').isdigit()

>>> is_negative_number_digit_v2('-2345')
True

>>> is_negative_number_digit_v2('-2345⁵')
True

Why isdigit Is Not Working for Me?

The most common issue that prevents isdigit / isnumeric / isdecimal to work properly is having a leading or trailing whitespace in the string. Before using them it's imperative to remove any leading or trailing whitespace, or other character such as newline (\n)).

>>> ' 54'.isdigit()
False

>>> ' 54'.strip().isdigit()
True

>>> ' 54'.isnumeric()
False

>>> ' 54'.strip().isnumeric()
True

>>> ' 54'.isdecimal()
False

>>> ' 54'.strip().isdecimal()
True

>>> '65\n'.isdigit()
False

>>> '65\n'.strip().isdigit()
True

Conclusion

In this post, we saw the subtle difference between isdigit vs isdecimal vs isnumeric and how to choose the most appropriate string method for your use case. We also saw cases that cannot be dealt with them alone and how to overcome those limitations.

That's it for today and I hope you've enjoyed this post!

References:

https://docs.python.org/3/library/stdtypes.html

https://www.fileformat.info/info/unicode/char/0660/browsertest.htm

https://stackoverflow.com/a/7643705

https://stackoverflow.com/a/28279773

The Motivation

I started blogging with two goals in mind:

• to share things I find interesting

• to improve my writing skills in English

Sharing Interesting Stuff

The definition of “interesting” is in the eye of beholder. What I find cool might not catch your attention, and some things will be considered boring. If it’s boring, why sharing?

Had I held this mindset, I’d never get started. So I promised myself: I’d write what I considered interesting and if it helps other people, then great! If not, that’s fine!

Adopting this demeanor was crucial to help me get started. Going from 0 to something requires more energy than keeping consistency. So once we overcome this initial barrier, it’s just a matter of putting a bit more energy from time to time.

After producing the first few articles, I realized that what was interesting to me was also interesting to other people. Each “thank you” would put more gas into my motivation tank, so I kept going.

Improving My Writing Skills

You might have noticed that English is not my native language. In fact, my English is pretty much self-taught. I started learning it primarily to consume English content and never had the intention to produce anything.

But... what is the point of learning a language and not being able to use it fully?

Even though I could write shorter messages, e-mails and things like that, expressing complex ideas was a challenge to me.

Writing is considerably harder than passive reading, and doing that in a second language is even harder. So I asked myself, “What can I do to improve my writing skills?”.

The answer could not be more obvious: write more!

The Downs - part. 1

I spent quite a long time trying to set up a blog platform. I chose hugo because it's fast and has many features, making it very flexible. The only problem is that this flexibility comes with a cost: it's a sea of distraction.

I think I spent almost a month trying to tweak it, testing themes and whatnot. I then realized that I was focusing on the wrong thing. The focus should be on the content, not on the appearance.

Luckily, I bumped by a tweet from Catalin Pit recommending hashnode. It was exactly what I was looking for, I setup my blog in less than an hour and wrote my first article in a few days.

The Downs - part. 2

This end of August was the end of summer in Europe and a second wave of Covid had been looming over. A new lockdown was imminent, and I’d have to stay inside as much as I could. Not only that, I also didn’t have to commute every day, which would consume about 1h of my day.

With this “free time” available, I set a pretty impressive streak and wrote one article per week. My goal was to publish every Saturday morning.

And did that until... I went back to Brazil for Christmas and I decided that I’d not touch a computer during that time. I had some signs of burnout and thought that would be the way to revert it.

After the break, I returned to the UK and had little to no motivation of writing again. Maybe I was still burned out, I don’t know. The truth is that I wrote only 4 articles after the break and I used to write that many each month.

I’m not sure if I can do 1 post a week ever again. My focus has shifted from quantity to quality. It’s not that my first few articles lacked quality, but keeping this standard requires too much time and energy.

The Ups - The Stats

Now it’s time for some interesting stuff: the stats! Up to this point, not considering this article, I’ve written 22 posts. Out of those, 5 of them brings most of the organic traffic.

Traffic By Channel

In 1 year, people have viewed this blog around ~129K times, according to Google Analytics. This number might be higher if we consider that tech-savvy people block GA. Taking only organic search into account, 46% of the views come from search engines such as Google, Bing and Duck Duck Go. It’s actually not bad considering that I'm not an SEO expert.

Organic Visits

In this chart, we can view the organic traffic every month. I started investing in SEO after subscribing to Monica Lent's Blogging for Devs newsletter by the end of 2020. The things I learned from it really paid off, as you can see in the graph!

The views had been growing steady until July. The decline has presumably something to do with Google’s release of an update to its core algorithm.

I experienced the same thing in December, and the traffic will perhaps recover like it did back them.

ps: If you want to learn more about SEO, I definitely recommend Monica's Blogging for Devs and her new course SEO for Devs

The Most Popular Ones - Top 5 Overall

Here’s a list of my popular posts according to GA, and backed by hashnode analytics.

73 Examples to Help You Master Python’s f-strings

The Best Way to Compare Two Dictionaries in Python

5 Hidden Python Features You Probably Never Heard Of

How to Shoot Yourself in the Foot With Python. Part 1.

How to Use datetime.timedelta in Python With Examples

The Most Searched Ones - Top 5 Overall

And here's a list of my popular articles based on organic searches only.

The Best Way to Compare Two Dictionaries in Python

How to Use datetime.timedelta in Python With Examples

How to Pass Multiple Arguments to a map Function in Python

How to Check if an Exception Is Raised (or Not) With pytest

How to Use Fixtures as Arguments in pytest.mark.parametrize

Plans for the Future

I write mostly about Python and its ecosystem. I’ll probably continue writing about it but I’m also considering adding a bit more of Machine Learning to it. I’ve been working with ML for a good four years and I think I have some cool stuff to share.

For now, my plan is to find a schedule that works for me and that allows me to keep consistent. I’m not sure if I can write one article per week, but one every two weeks seems very doable. Still, the focus will be always on quality first.

Conclusion

That’s it, folks! I hope you liked this post and see you next time! I'd like to thank the awesome Chris Bongers for proofreading this article and suggest a few improvements, thanks mate!

Why?

The idea of having an else-less if expression in Python came to my mind when I had to work with a Ruby service at my past job. Ruby, contrary to Python, makes lots of things implicit [Citation needed], and this kind of if expression is one of them. I say it's implicit because it returns nil if the expression evaluates to false. This is also called conditional modifier.

$ irb
irb(main):001:0> RUBY_VERSION
=> "2.7.1"
irb(main):002:0> a = 42 if true
=> 42
irb(main):003:0> b = 21 if false
=> nil
irb(main):004:0> b
=> nil
irb(main):005:0> a
=> 42

In Python, one cannot do that without explicitly adding an else to the expression. In fact, as of this PR, the interpreter will tell right away that the else is mandatory in the SyntaxError message.

$ ./python
Python 3.11.0a0 (heads/main:938e84b4fa, Aug  6 2021, 08:59:36) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 42 if True
  File "<stdin>", line 1
    a = 42 if True
        ^^^^^^^^^^
SyntaxError: expected 'else' after 'if' expression

However, I find Ruby's if actually very convenient. This convenience became more evident when I had to go back to Python and write things like this:

>>> my_var = 42 if some_cond else None

So I thought to myself, what would be like if Python had similar feature? Could I do it myself? How hard would that be?

How?

Me trying to make sense of CPython's source code.

Digging into CPython's code and changing the language's syntax sounded not trivial to me.

Luckily, during the same week, I found out on Twitter that Anthony Shaw had just written a book on CPython Internals and it was available for pre-release. I didn't think twice and bought the book.

I've got to be honest, I'm the kind of person who buys things and doesn't use them immediately. As I had other plans in mind, I let it "getting dust" in my home folder for a while.

Until... I had to work with that Ruby service again. It reminded me of the CPython Internals book and how challenging hacking the guts of Python would be.

First thing was to go though the book from the very start and try to follow each step. The book focus on Python 3.9, so in order to follow though it, one needs to checkout the 3.9 tag, and that's what I did.

I learned about how the code is structured and then how to compile it. The next chapters show how to extend the grammar and add new things, such as a new operator.

As I got familiar with the code base and how to tweak the grammar, I decided to give it a spin and make my own changes to it.

The First (Failed) Attempt

As I started finding my way around CPython's code from the latest main branch, I noticed that lots of things had changed since Python 3.9, yet some fundamental concepts didn't.

My first shot was to dig into the grammar definition and find the if expression rule. The file is currently named Grammar/python.gram. Locating it was not difficult, an ordinary CTRL+F for the 'else' keyword was enough.

file: Grammar/python.gram
...
expression[expr_ty] (memo):
    | invalid_expression
    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }
    | disjunction
    | lambdef
....

Now with the rule in hand, my idea was to add one more option to the current if expression where it would match a=disjunction 'if' b=disjunction and c expression would be NULL.

This new rule should be placed immediately after the complete one, otherwise the parser would match a=disjunction 'if' b=disjunction always, returning a SyntaxError.

expression[expr_ty] (memo):
    | invalid_expression
    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }
    | a=disjunction 'if' b=disjunction  { _PyAST_IfExp(b, a, NULL, EXTRA) }
    | disjunction
    | lambdef
....

Regenerating the Parser and Compiling Python From Source

CPython comes with a Makefile containing lots of useful commands. One of them is the regen-pegen command which converts Grammar/python.gram into Parser/parser.c.

Besides changing the grammar, I had to modify the AST for the if expression. AST stands for Abstract Syntax Tree and it is a way of representing the syntactic structure of the grammar as a tree. For a more information about ASTs, I highly recommend the Crafting Interpreters book by Robert Nystrom.

Moving on, if you observe the rule for if expression goes like this:

    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }

The means, when the parser finds this rule, it calls the _PyAST_IfExp which gives back a expr_ty data structure. So this gave me a clue, in order to implement the behavior of the new rule, I'd need to change _PyAST_IfExp.

To find where is located, I used my rip-grep skills and searched for it inside the source root.

$ rg _PyAST_IfExp -C2 .

[OMITTED]
Python/Python-ast.c
2686-
2687-expr_ty
2688:_PyAST_IfExp(expr_ty test, expr_ty body, expr_ty orelse, int lineno, int
2689-             col_offset, int end_lineno, int end_col_offset, PyArena *arena)
2690-{
[OMITTED]

... And the implementation goes like this:

expr_ty
_PyAST_IfExp(expr_ty test, expr_ty body, expr_ty orelse, int lineno, int
             col_offset, int end_lineno, int end_col_offset, PyArena *arena)
{
    expr_ty p;
    if (!test) {
        PyErr_SetString(PyExc_ValueError,
                        "field 'test' is required for IfExp");
        return NULL;
    }
    if (!body) {
        PyErr_SetString(PyExc_ValueError,
                        "field 'body' is required for IfExp");
        return NULL;
    }
    if (!orelse) {
        PyErr_SetString(PyExc_ValueError,
                        "field 'orelse' is required for IfExp");
        return NULL;
    }
    p = (expr_ty)_PyArena_Malloc(arena, sizeof(*p));
    if (!p)
        return NULL;
    p->kind = IfExp_kind;
    p->v.IfExp.test = test;
    p->v.IfExp.body = body;
    p->v.IfExp.orelse = orelse;
    p->lineno = lineno;
    p->col_offset = col_offset;
    p->end_lineno = end_lineno;
    p->end_col_offset = end_col_offset;
    return p;
}

Since I pass orelse as NULL, I thought it was just a matter of changing the body of if (!orelse) and assign None to orelse.

    if (!orelse) {
-       PyErr_SetString(PyExc_ValueError,
-                       "field 'orelse' is required for IfExp");
-       return NULL;
+       orelse = Py_None;
    }

Now time to test it, I compile the code with make -j8 -s and fire up the interpreter.

$ make -j8 -s                                                                                                                                                            
Python/Python-ast.c: In function ‘_PyAST_IfExp’:
Python/Python-ast.c:2703:16: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types]
         orelse = Py_None;

Despite the glaring obvious warnings, I decided to ignore it just to see what happens. 😅

$ ./python
Python 3.11.0a0 (heads/ruby-if-new-dirty:f92b9133ef, Aug  2 2021, 09:13:02) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 42 if True
>>> a
42
>>> b = 21 if False
[1]    16805 segmentation fault (core dumped)  ./python

Ouch! It works for the if True case, but assigning Py_None to expr_ty orelse causes a segfault.

Time to go back to see what is wrong.

The Second Attempt

It wasn't too difficult to figure out where I messed up. orelse is a expr_ty and I'm assigning to it a Py_None which is a PyObject *. Again, thanks to rip-grep I found its definition.

$ rg constant -tc -C2

Include/internal/pycore_asdl.h
14-typedef PyObject * string;
15-typedef PyObject * object;
16:typedef PyObject * constant;

Now, how did I find out Py_None was a constant?

Whilst reviewing Grammar/python.gram file, I found that one of the rules for the new pattern matching syntax is defined like this:

# Literal patterns are used for equality and identity constraints
literal_pattern[pattern_ty]:
    | value=signed_number !('+' | '-') { _PyAST_MatchValue(value, EXTRA) }
    | value=complex_number { _PyAST_MatchValue(value, EXTRA) }
    | value=strings { _PyAST_MatchValue(value, EXTRA) }
    | 'None' { _PyAST_MatchSingleton(Py_None, EXTRA) }

However, this rule is a pattern_ty not an expr_ty. But that's fine. What really matters is to understand what _PyAST_MatchSingleton actually is. Then, I searched for it in Python/Python-ast.c.

file: Python/Python-ast.c
...
pattern_ty
_PyAST_MatchSingleton(constant value, int lineno, int col_offset, int
                      end_lineno, int end_col_offset, PyArena *arena)
...

Now back to the "drawing board", I look for the definition of a None node in the grammar. To my great relief, I find it!

atom[expr_ty]:
    | NAME
    | 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) }
    | 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) }
    | 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) }
....

At this point, I had all the information I needed. To return a expr_ty representing None I need to create a node in the AST which is constant by using the _PyAST_Constant function.

    | a=disjunction 'if' b=disjunction 'else' c=expression { _PyAST_IfExp(b, a, c, EXTRA) }
-   | a=disjunction 'if' b=disjunction { _PyAST_IfExp(b, a, NULL, EXTRA) }
+   | a=disjunction 'if' b=disjunction { _PyAST_IfExp(b, a, _PyAST_Constant(Py_None, NULL, EXTRA), EXTRA) }
    | disjunction

Now I must revert Python/Python-ast.c as well. Since I'm feeding it a valid expr_ty, it will never be NULL.

file: Python/Python-ast.c
...
     if (!orelse) {
-        orelse = Py_None;
+        PyErr_SetString(PyExc_ValueError,
+                        "field 'orelse' is required for IfExp");
+        return NULL;
     }
...

Let's compile it again and see what happens!

$ make -j8 -s && ./python                             
Python 3.11.0a0 (heads/ruby-if-new-dirty:25c439ebef, Aug  2 2021, 09:25:18) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> c = 42 if True
>>> c
42
>>> b = 21 if False
>>> type(b)
<class 'NoneType'>
>>>

WOT!? It works! 🎉🎉🎉

Now, we need to do one more test. Ruby functions allow returning a value if a condition matches and if not, the rest of the function body gets executed. Like this 👇

At this point I wonder if that would work out-of-the-box. I rush to the interpreter again and write the same function.

>>> def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... 
>>> f(False)
>>> f(True)
42
>>>

Ooopss...

The function returns None if test is False... To help me debug this, I summoned the ast module. The official docs define it like so:

The ast module helps Python applications to process trees of the Python abstract syntax grammar. The abstract syntax itself might change with each Python release; this module helps to find out programmatically what the current grammar looks like.

Let's print the AST for this function...

>>> fc = '''
... def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... '''
>>> print(ast.dump(ast.parse(fc), indent=4))
Module(
    body=[
        FunctionDef(
            name='f',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(arg='test')],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                Return(
                    value=IfExp(
                        test=Name(id='test', ctx=Load()),
                        body=Constant(value=42),
                        orelse=Constant(value=None))),
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Constant(value='missed return')],
                        keywords=[])),
                Return(
                    value=Constant(value=21))],
            decorator_list=[])],
    type_ignores=[])

Now things make more sense, my change to the grammar was just a syntax sugar. It turns an expression like this a if b into this a if b else None. The problem here is that Python will return no matter what, so the rest of the function is ignored.

We can also look at the bytecode generated to understand what exactly is executed by the interpreter. And for that, we can use the dis module. According to the docs:

The dis module supports the analysis of CPython bytecode by disassembling it.

>>> import dis
>>> dis.dis(f)
  2           0 LOAD_FAST                0 (test)
              2 POP_JUMP_IF_FALSE        4 (to 8)
              4 LOAD_CONST               1 (42)
              6 RETURN_VALUE
        >>    8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

What this basically means is that in case the test is false, the execution jumps to 8, which loads the None into the top of the stack and returns it.

Supporting "return-if"

To support the same Ruby feature, I can turn the expression return 42 if test into a regular if statement that returns if test is true.

To do that, I needed to add one more rule. This time, it would be a rule that matches the return <value> if <test> piece of code. Not only that, we need a _PyAST_ function that creates the node for us. Let's then call it _PyAST_ReturnIfExpr.

file: Grammar/python.gram

return_stmt[stmt_ty]:
+   | 'return' a=star_expressions 'if' b=disjunction { _PyAST_ReturnIfExpr(a, b, EXTRA) }
    | 'return' a=[star_expressions] { _PyAST_Return(a, EXTRA) }

As mentioned previously, the implementation for all these functions reside in Python/Python-ast.c, and the their definition in Include/internal/pycore_ast.h, so I put _PyAST_ReturnIfExpr there.

file: Include/internal/pycore_ast.h

 stmt_ty _PyAST_Return(expr_ty value, int lineno, int col_offset, int
                       end_lineno, int end_col_offset, PyArena *arena);
+stmt_ty _PyAST_ReturnIfExpr(expr_ty value, expr_ty test, int lineno, int col_of
fset, int
+                      end_lineno, int end_col_offset, PyArena *arena);
 stmt_ty _PyAST_Delete(asdl_expr_seq * targets, int lineno, int col_offset, int
                       end_lineno, int end_col_offset, PyArena *arena);

file: Python/Python-ast.c

 }

+stmt_ty
+_PyAST_ReturnIfExpr(expr_ty value, expr_ty test, int lineno, int col_offset, int end_lineno, int
+              end_col_offset, PyArena *arena)
+{
+    stmt_ty ret, p;
+    ret = _PyAST_Return(value, lineno, col_offset, end_lineno, end_col_offset, arena);
+
+    asdl_stmt_seq *body;  
+    body = _Py_asdl_stmt_seq_new(1, arena); 
+    asdl_seq_SET(body, 0, ret);
+
+    p = _PyAST_If(test, body, NULL, lineno, col_offset, end_lineno, end_col_offset, arena);
+
+    return p;
+}
+
 stmt_ty

Let's pause for a bit to examine the implementation of _PyAST_ReturnIfExpr. Like I mentioned previously, I want to turn return <value> if <test> into if <test>: return <value>.

Both return and the regular if are statements, so in CPython they're represented as stmt_ty. The _PyAST_If expectes a expr_ty test and a body, which is a sequence of statements. In this case, body is asdl_stmt_seq *body.

As a result, what we really want here is a if statement with a body where the only statement is a return <value> one.

CPython disposes of some convenient functions to build asdl_stmt_seq * and one of them is _Py_asdl_stmt_seq_new. So I used it to create the body and add the return statement I created a few lines before with _PyAST_Return.

Once that's done, the last step is to pass the test as well as the body to _PyAST_If.

And before I forget, you may be wondering what on earth is the PyArena *arena. Arena is a CPython abstraction used for memory allocation. It allows efficient memory usage by using memory mapping mmap() and placing them in contiguous chunks of memory [reference].

Now it's time to regenerate the parser and test it one more time.

>>> def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... 
>>> import dis
>>> f(False)
>>> f(True)
42

Oh no! It doesn't work...

Let's check the bytecodes...

>>> dis.dis(f)
  2           0 LOAD_FAST                0 (test)
              2 POP_JUMP_IF_FALSE        4 (to 8)
              4 LOAD_CONST               1 (42)
              6 RETURN_VALUE
        >>    8 LOAD_CONST               0 (None)
             10 RETURN_VALUE
>>>

... the same bloody bytecode instructions again!

Going Back to the Compilers Class

At that point, I was clueless. I had no idea what was going on until... I decided to go down the rabbit hole of expanding the grammar rules.

The new rule I added went like this 'return' a=star_expressions 'if' b=disjunction { _PyAST_ReturnIfExpr(a, b, EXTRA) }.

My only hypothesis is that a=star_expressions 'if' b=disjunction is being resolved to the else-less rule I added in the beginning.

By going over the grammar one more time, I figure that my theory holds. star_expressions will match a=disjunction 'if' b=disjunction { _PyAST_IfExp(b, a, NULL, EXTRA) }.

The only way to fix this is by getting rid of the star_expressions. So I change the rule to:

 return_stmt[stmt_ty]:
-    | 'return' a=star_expressions 'if' b=disjunction { _PyAST_ReturnIfExpr(a, b, EXTRA) }
+    | 'return' a=disjunction guard=guard !'else' { _PyAST_ReturnIfExpr(a, guard, EXTRA) }
     | 'return' a=[star_expressions] { _PyAST_Return(a, EXTRA) }

You might be wondering, what is guard and what is !else and what is star_expressions?

This 'guard' is a rule that is part of the pattern matching rules. The new pattern matching feature added in Python 3.10 allows things like this:

match point:
    case Point(x, y) if x == y:
        print(f"Y=X at {x}")
    case Point(x, y):
        print(f"Not on the diagonal")

And the rule goes by this:

guard[expr_ty]: 'if' guard=named_expression { guard }

With that, I added one more check. To avoid it failing with SyntaxError, we need to make sure the rule matches only code like this: return value if cond. Thus, to prevent code such as return an if cond else b being matched prematurely, I added a !'else' to the rule.

Last, but not least, the star_expressions allow us to return to return destructured iterables. For example:

>>> def f():
   ...:     a = [1, 2]
   ...:     return 0, *a
   ...: 

>>> f()
(0, 1, 2)

In this case, 0, *a is a tuple, which falls under the category of star_expressions. The regular if-expression doesn't allow using star_expressions with it AFAIK, so changing our new return rule won't be an issue.

Does it work yet?

After fixing the return rule, I regenerate the grammar one more time and compile it.

>>> def f(test):
...     return 42 if test
...     print('missed return')
...     return 21
... 
>>> f(False)
missed return
21
>>> f(True)
42

And... IT WORKS!!

Let's check the bytecode then...

>>> import dis
>>> dis.dis(f)
  2           0 LOAD_FAST                0 (test)
              2 POP_JUMP_IF_FALSE        4 (to 8)
              4 LOAD_CONST               1 (42)
              6 RETURN_VALUE

  3     >>    8 LOAD_GLOBAL              0 (print)
             10 LOAD_CONST               2 ('missed return')
             12 CALL_FUNCTION            1
             14 POP_TOP

  4          16 LOAD_CONST               3 (21)
             18 RETURN_VALUE
>>>

That's precisely what I wanted. In fact, to make sure, let also see if the AST is the same as the one with regular if.

>>> import ast
>>> print(ast.dump(ast.parse(fc), indent=4))
Module(
    body=[
        FunctionDef(
            name='f',
            args=arguments(
                posonlyargs=[],
                args=[
                    arg(arg='test')],
                kwonlyargs=[],
                kw_defaults=[],
                defaults=[]),
            body=[
                If(
                    test=Name(id='test', ctx=Load()),
                    body=[
                        Return(
                            value=Constant(value=42))],
                    orelse=[]),
                Expr(
                    value=Call(
                        func=Name(id='print', ctx=Load()),
                        args=[
                            Constant(value='missed return')],
                        keywords=[])),
                Return(
                    value=Constant(value=21))],
            decorator_list=[])],
    type_ignores=[])
>>>

And indeed it is!

If(
    test=Name(id='test', ctx=Load()),
    body=[
        Return(
            value=Constant(value=42))],
     orelse=[]),

This node is the same as the one that would be generated by

if test: return 42

If It's Not Tested, It's Broken?

To conclude this journey, I thought it'd be a good idea to add some unit tests as well. Before writing anything new, I wanted to get an idea of what I had broken.

With the code tested manually, I run all tests using the 'test' module, python -m test -j8. The -j8 means we'll use 8 processes to run the tests in parallel.

$ ./python -m test -j8

To my surprise, only one test fail! 😱

== Tests result: FAILURE ==

406 tests OK.

1 test failed:
    test_grammar

Since I ran all tests, it's hard to navigate on the output so I can run only this one again in isolation.

======================================================================
FAIL: test_listcomps (test.test_grammar.GrammarTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/miguel/projects/cpython/Lib/test/test_grammar.py", line 1732, in test_listcomps
    check_syntax_error(self, "[x if y]")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/projects/cpython/Lib/test/support/__init__.py", line 497, in check_syntax_error
    with testcase.assertRaisesRegex(SyntaxError, errtext) as cm:
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: SyntaxError not raised

----------------------------------------------------------------------

Ran 76 tests in 0.038s

FAILED (failures=1)
test test_grammar failed
test_grammar failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_grammar

1 re-run test:
    test_grammar

Total duration: 82 ms
Tests result: FAILURE

And there it is! It expects a syntax error when running a [x if y] expression. We can safely remove it and re-run the tests again.

== Tests result: SUCCESS ==

1 test OK.

Total duration: 112 ms
Tests result: SUCCESS

Now that everything is OK, it's time to add a few more tests. It's important to test not only the new "else-less if" but also the new return statement.

By navigating though the test_grammar.py file we can find a test for pretty much every grammar rule. The first one I look for is test_if_else_expr. This test doesn't fail, so it only tests for the happy case. To make it more robust we need to add two new tests to check if True and if False case.

        self.assertEqual((6 < 4 if 0), None)
        self.assertEqual((6 < 4 if 1), False)

I run everything again, all tests pass this time.

ps: bool in Python is subclass of integer, so you can use 1 to denote True and 0 for False

Ran 76 tests in 0.087s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 174 ms
Tests result: SUCCESS

Lastly, we need the tests for the return rule. They're defined in the test_return test. Just like the if expression one, this test pass with no modification.

To test this new use case, I create a function that receives a bool argument and returns if the argument is true, when it's false, it skips the return, just like the manual tests I have been doing up to this point.

        def g4(test):
            a = 1
            return a if test
            a += 1
            return a

        self.assertEqual(g4(False), 2)
        self.assertEqual(g4(True), 1)

Now, save the file and re-run test_grammar one more time.

----------------------------------------------------------------------

Ran 76 tests in 0.087s

OK

== Tests result: SUCCESS ==

1 test OK.

Total duration: 174 ms
Tests result: SUCCESS

All good in the hood! test_grammar passes with flying colors and the last thing, just in case, is to re-run the full test suite.

$ ./python -m test -j8

After a while, all tests pass and I'm very happy with the result.

Limitations

If you know Ruby well, by this point you've probably noticed that what I did here is not 100% the same as a conditional modifier. For example, in Ruby you can run actual expressions in these modifiers.

irb(main):002:0> a = 42
irb(main):003:0> a += 1 if false
=> nil
irb(main):004:0> a
=> 42
irb(main):005:0> a += 1 if true
=> 43

I cannot do the same with my implementation.

>>> a = 42
>>> a += 1 if False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'
>>> a += 1 if True
>>> a
43

What this reveals is that the return rule I created is just a workaround. If I want to make it as close as possible to Ruby's conditional modifier, I'll need to make it work with other statements as well, not just return.

Nevertheless, this is fine. My goal with this experiment was just to learn more about Python internals and see how would I navigate a little-known code base written in C and make the appropriate changes to it. And I have to admit that I'm pretty happy with the results!

Conclusion

Adding a new syntax inspired by Ruby is a really nice exercise to learn more about the internals of Python. Of course, if I had to convert this as a PR, the core developers would probably find a few shortcomings, as I have already found and described in the previous section. However, since I did this just for fun, I'm very happy with the results.

The source code with all my changes is on my CPython fork under the branch ruby-if-new.

I hope you've found this post cool and let's see what comes next!

Other posts you may like:

• 11 Useful Resources To Learn Python's Internals From Scratch

See you next time!

I have been asking myself that question for the past few months and now it seems that I'm starting to grasp, slowly...

During this time, I have grown a strong interest in learning more about the internal working of python. I find the CPython implementation so fascinating that I even started contributing to the language.

The CPython runtime is the most popular one but there are a few others like pypy. Unlike pypy, the core language is written in C whereas the standard library is a blend of Python and C.

For newcomers, navigating through the code can be a daunting task. Fortunately, there are some nice resources out there that can help to smooth the learning curve.

In this post, I'll show you my favorite resources to start learning more about the inner workings of Python, a.k.a CPython Internals.

By the end of this tutorial, you should be able to:

• choose the best books that will help you understand Python's source code
• learn more about the CPython internals via public talks
• find the best blogs and other resources that cover the Python internals

Table of Contents

1. Books About CPython Internals
2. Videos Related to CPython Internals
3. Blog Posts
4. Other Resources About CPython Internals
5. Conclusion

Books About CPython Internals

Python has grown immensely during the past years and the more people learning it, the more the demand for learning materials discussing advanced topics.

A few years ago, I bet just a few curious developers would be interested in learning more about CPython.

These days it’s not unusual to find comprehensive blog posts, videos and books going over the inner guts of Python. Speaking of books, I can surely vouch for two:

• CPython Internals: Your Guide to the Python 3 Interpreter by Anthony Shaw
• Inside The Python Virtual Machine by Obi Ike-Nwosu

CPython Internals: Your Guide to the Python 3 Interpreter - A Brief Review

This one is the most newly published book about CPython. Amongst all the things it covers, you will find information about:

• How to build and compile Python from source on MacOS, Linux and Windows
• How to set up a development environment
• The Python’s grammar and language specification
• The eval loop
• How Python manages memory
• How to run the test suite

You can find the full table of contents in the Real Python website, it comes in digital format and paperback. The eBook versions are DRM-Free and available in three different formats: epub, mobi and PDF. The paperback, on the other hand, can be found on amazon .

The nicest thing to me is that Anthony demonstrates how to add a new operator to the language: the almost-equal. This operator is represented by “~=” and can determine if two numbers are close enough to each other but not exactly equal. It walks through all the fundamental changes required to make this happen, including extending the grammar.

Thanks to this book, I started contributing to CPython and have already a handful PRs merged, including enhancements, documentation and bug fixes.

Lastly, the code samples are available on github.

Inside The Python Virtual Machine - A Brief Review

Another good book covering Python's internals, and if I'm not mistaken it's the first one to explore CPython in detail. Inside The Python Virtual Machine is much shorter than CPython Internals but covers some parts of the language in more detail, such as Python objects, Code and Frame objects. It's available for free as PDF, ePub and Kindle (mobi) on leanpub but I definitely encourage you to buy it.

What I loved the most is the in-depth examination of Python objects. It does a brilliant job dissecting the types, goes over the internals of objects and their attributes and concludes with an explanation of the Method Resolution Order (MRO). I haven’t read it in full yet but I like it so far.

Videos Related to CPython Internals

When it comes to video contents, there isn’t much structured content out there. The first one I learned about was P. Guo’s series covering Python 2.7. Unfortunately, he’s taken out the series from his website but you can still find it via an unlisted playlist.

CPython internals: A ten-hour codewalk through the Python interpreter source code

Check it out the first video of the series.

The full playlist can be found here .

Pablo Salgado - The soul of the beast

In this talk, Pablo Galindo, who is a Python’s core developer, looks at the former Python’s grammar and its limitations. This presentation is fantastic for those who want to understand the general structure of the compiler. By the end, Pablo shows how you can add a new operator to Python, the 'arrow operator' ->.

Eric Snow - to GIL or not to GIL: the Future of Multi-Core (C)Python - PyCon 2019

In this presentation, Eric Snow talks about Python’s GIL (Global Interpreter Lock) and the future developments to circumvent its impact on performance and unlock multi-core capability in Python. It won’t dive into the source code but it’s a good intro to one of the most controversial topics in Python.

Pablo Galindo Salgado - Time to take out the rubbish: garbage collector - PyCon 2019

In this talk, Pablo presents the "magic" behind Python’s memory management by detailing the inner works of the garbage collector and why it matters. He illustrates some gotchas such as the reason you cannot rely on __del__ method and describes in detail the reference counter.

CPython Full Course - Dev Internals

I’ve recently found this resource, and it looks very neat. The author examines the implementation of NoneType and also demonstrates how to include a __len__ method to ints.

What caught my eye was that the author uses gbd to debug the C portion of the code. I find it nice because it’s not so easy for find videos demonstrating how to debug the CPython code.

Blog Posts

My favorite blog post series on this topic is from Ten thousand meters by Victor. I believe it’s the most complete and up-to-date coverage into the internals of the Python interpreter in the form blog posts.

Another exceptional series, albeit slightly outdated, is the Python internals series from Eli Bendersky. There are lots of interesting stuff including adding new keywords and a great coverage on symbol tables.

Other Resources About CPython Internals

A Python Interpreter Written in Python by Allison Kaptur

This article / mini book is a great resource to understand Python’s bytecode. In only 500 lines of code Allison implements Byterun, a Python interpreter written in pure Python.

As impressive as it may sound, Byterun can actually run a variety of simple Python programs. After reading it you’ll have a much better understanding of the Python interpreter.

The booklet can be found one https://www.aosabook.org.

Python's Official Docs

The official documentation is also an excellent place to go, the text can be particularly dry but that's what you usually expect from a reference. A nice section in particular is the exploring guide. It goes through the source code’s structure and also links to other resources. The official website also shows the C-API in considerable detail.

Conclusion

Learning about CPython can be disheartening, but thanks to the increasing demand, more and more materials have been developed which softens the learning curve considerably. In this article I presented my preferred resources to learn about the internals of Python ranging from books, to videos and blog posts. I hope this can be useful to you as it is to me.

Other posts you may like:

• How I Patched Python to Include This Ruby Feature

See you next time!

By the end of this article, you should be able to:

• use the choice function to generate a random string from string.ascii_letters, string.digits + string.punctuation characters in Python 3
• generate a secure random string, useful to create random passwords

Generating a Random String With Upper Case, Lower Case, Digits and Punctuation

The string module comes with a nice set of constants that we can combine with the random module to create our random string.

Using Python's string.ascii_letters + string.digits + string.punctuation Characters

By concatenating string.ascii_letters + string.digits + string.punctuation, we will have our pool of characters that we can pick at random using the random.choices() method. This method returns a k sized list of elements chosen from the population with replacement. In our case, k will be the size of our random string.

>>> import string

>>> string.ascii_letters
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'

>>> string.digits
'0123456789'

>>> string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

>>> all_chars = string.ascii_letters + string.digits + string.punctuation

>>> import random

>>> ''.join(random.choices(all_chars, k=10))
'4`<WJ."=$r'

This works well, but there is one limitation: random.choices is available only on Python 3.6+. To make that work in older versions, you'll need to call the random.choice function and iterate over k times.

>>> ''.join((random.choice(all_chars) for _ in range(10)))
'd&6Bx5PX(R'

How to Generate a Cryptographically Secure Random String

The previous method works well if all we want is a random string for simple use cases. If what we want is a random string to be used as a password, then we need a more secure method. The reason is that the random module does not use a secure pseudo-number generator.

As an alternative we must resort to the Operating System’s pseudo-random number generator. The good news is that Python can access that using the random.SystemRandom class, which ensures that sequences are not reproducible.

We can re-use the previous example and change just one minor detail.

From: random.choices to random.SystemRandom()

>>> all_chars = string.ascii_letters + string.digits + string.punctuation

>>> import random

>>> ''.join(random.SystemRandom().choices(all_chars, k=10))
'T$WoW.sdQc'

Conclusion

In this post we saw 2 different ways of creating random strings Python. I hope you find it useful.

Other posts you may like:

• Design Patterns That Make Sense in Python: Simple Factory
• How to Pass Multiple Arguments to a map Function in Python
• 73 Examples to Help You Master Python's f-strings
• 3 Ways to Unit Test REST APIs in Python
• Everything You Need to Know About Python's Namedtuples
• The Best Way to Compare Two Dictionaries in Python
• 5 Hidden Python Features You Probably Never Heard Of

See you next time!

This post was originally published at https://miguendes.me

Say that you have the following dictionary containing your grades associated with a subject. You want to sort the values, in this case the grades, in a descending manner - the highest grade will appear first and the lowest last.

For example, you have this:

grades = {"Math": 34, "Science": 12, "English": 89, "Physics": 8}

... And you want this:

{'English': 89, 'Math': 34:, 'Science': 12:, 'Physics': 8}

You can do this is at least 3 different ways.

Sorting a dict by value descending using list comprehension

The quickest way is to iterate over the key-value pairs of your current dict and call sorted passing the dictionary values and setting reversed=True.

If you are using Python 3.7, regular dicts are ordered by default. So let's use it!

>>> grades = {"Math": 34, "Science": 12, "English": 89, "Physics": 8}
>>> grades
{'Math': 34, 'Science': 12, 'English': 89, 'Physics': 8}
>>> value_key_pairs = ((value, key) for (key,value) in grades.items())
>>> sorted_value_key_pairs = sorted(value_key_pairs, reverse=True)
>>> sorted_value_key_pairs
[(89, 'English'), (34, 'Math'), (12, 'Science'), (8, 'Physics')]
>>> {k: v for v, k in sorted_value_key_pairs}
 {'English': 89, 'Math': 34, 'Science': 12, 'Physics': 8}

And Voila! You have your sorted grades dict in a descending fashion.

What if I have Python 3.6 or lower?

In this case, you can use OrderedDict from the collections module.

>>> from collections import OrderedDict

>>> OrderedDict((k, v) for v, k in sorted_value_key_pairs)
OrderedDict([('English', 89), ('Math', 34), ('Science', 12), ('Physics', 8)])

Sorting a dictionary in descending order using the operator module

The operator module provides a functional interface to built-in operators like <, >, == and so on.

This module has many useful functions and one of them is the itemgetter. This function returns a callable object that will fetch the item using the __getitem__() method.

In a nutshell, if you do callable = operator.itemgetter(1), and pass a subscriptable object, say ('Physics', 8) to this callable, it will return the equivalent of ('Physics', 8)[1].

For example,

>>> subject_grade_pair = ('Physics', 8)

>>> get_grade = operator.itemgetter(1)

>>> get_grade(subject_grade_pair)
8

>>> subject_grade_pair[1]
8

Now, the question is, how can we use this to sort the values?

The sorted built-in function expects not only the iterable you want to sort but also a key. This key argument is nothing more than a function of one argument that you can feed each item of the list. And that’s exactly what we need to sort our list!

>>> import operator

# remember, the grade is the second item in the subject - grade pair
>>> sort_by_grade = operator.itemgetter(1)

>>> grades = {"Math": 34, "Science": 12, "English": 89, "Physics": 8}
>>> grades
{'Math': 34, 'Science': 12, 'English': 89, 'Physics': 8}

>>> sorted_value_key_pairs = sorted(grades.items(), key=sort_by_grade, reverse=True)
>>> {k: v for v, k in sorted_value_key_pairs}
 {'English': 89, 'Math': 34, 'Science': 12, 'Physics': 8}

At this point you might wonder:

This sort_by_grade looks like a glorified lambda...

Well, good shout, this brings us to the last section.

Using lambda to sort the dictionary in descending order

So, as we saw in the previous section, operator.itemgetter returns a callable that is equivalent to calling the __getitem__() method on a subscriptable object.

This is remarkably similar to pass a lambda as a key which takes a tuple, say (“Math”, 34), and returns the second item which is the grade.

>>> sort_by_grade_lambda = lambda subject_grade_pair: subject_grade_pair[1]

>>> sorted_value_key_pairs = sorted(grades.items(), key=sort_by_grade_lambda, reverse=True)

>>> {k: v for v, k in sorted_value_key_pairs}
 {'English': 89, 'Math': 34, 'Science': 12, 'Physics': 8}

And this is how you use a lambda function to sort a dict items by value in descending order.

Sorting a dictionary with complex objects as values

So far we've only dealt with simple objects such as int. What happens if we have a complex object, such as a custom Grade object as a dictionary value?

Let's see how it works.

class Grade:
    def __init__(self, grade: int, cutoff: int = 70):
        self.grade = grade
        self.cutoff = cutoff
        self.passed = grade >= cutoff

    def __repr__(self):
        return f"<Grade(grade={self.grade}, cutoff={self.cutoff}, passed={self.passed}>"

Let's try to sort it...

>>> grades = {"Math": Grade(grade=34), "Science": Grade(grade=12), "English": Grade(grade=89), "Physics": Grade(grade=8)}
>>> grades
grades = {"Math": Grade(grade=34), "Science": Grade(grade=12), "English": Grade(grade=89), "Physics": Grade(grade=8)}

>>> value_key_pairs = ((value, key) for (key,value) in grades.items())
>>> sorted_value_key_pairs = sorted(value_key_pairs, reverse=True)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-0c94e26fda4a> in <module>
----> 1 sorted_value_key_pairs = sorted(value_key_pairs, reverse=True)

TypeError: '<' not supported between instances of 'Grade' and 'Grade'

Oops! It didn't work. The reason is that, as the error message says, Grade doesn't implement the __lt__ operator, which makes it impossible to compare them.

To fix that we can either implement the __lt__ method or use the lambda as we used before with an adaptation. Let's see the lambda approach first.

>>> sort_by_grade_lambda = lambda subject_grade_pair: subject_grade_pair[1].grade

>>> sorted_value_key_pairs = sorted(grades.items(), key=sort_by_grade_lambda, reverse=True)

>>> {k: v for v, k in sorted_value_key_pairs}
{<Grade(grade=89, cutoff=70, passed=True>: 'English',
 <Grade(grade=34, cutoff=70, passed=False>: 'Math',
 <Grade(grade=12, cutoff=70, passed=False>: 'Science',
 <Grade(grade=8, cutoff=70, passed=False>: 'Physics'}

Implementing __lt__

Let's see how it looks when we implement the < operator.

class Grade:
    def __init__(self, grade: int, cutoff: int = 70):
        self.grade = grade
        self.cutoff = cutoff
        self.passed = grade >= cutoff

    def __lt__(self, other: "Grade") -> bool:
        return self.grade < other.grade

    def __repr__(self):
        return f"<Grade(grade={self.grade}, cutoff={self.cutoff}, passed={self.passed}>"

>>> grades = {"Math": Grade(grade=34), "Science": Grade(grade=12), "English": Grade(grade=89), "Physics": Grade(grade=8)}

>>> grades
{'Math': <Grade(grade=34, cutoff=70, passed=False>,
 'Science': <Grade(grade=12, cutoff=70, passed=False>,
 'English': <Grade(grade=89, cutoff=70, passed=True>,
 'Physics': <Grade(grade=8, cutoff=70, passed=False>}

>>> value_key_pairs = ((value, key) for (key,value) in grades.items())

>>> sorted_value_key_pairs = sorted(value_key_pairs, reverse=True)

>>> sorted_value_key_pairs
[(<Grade(grade=89, cutoff=70, passed=True>, 'English'),
 (<Grade(grade=34, cutoff=70, passed=False>, 'Math'),
 (<Grade(grade=12, cutoff=70, passed=False>, 'Science'),
 (<Grade(grade=8, cutoff=70, passed=False>, 'Physics')]

>>> {k: v for v, k in sorted_value_key_pairs}
{'English': <Grade(grade=89, cutoff=70, passed=True>,
 'Math': <Grade(grade=34, cutoff=70, passed=False>,
 'Science': <Grade(grade=12, cutoff=70, passed=False>,
 'Physics': <Grade(grade=8, cutoff=70, passed=False>}

And Voila! You have your grades dictionary sorted by value in a descending way.

Conclusion

In this post we saw 3 different ways of sorting a dictionary in descending order with Python. I hope you find it useful.

Other posts you may like:

• Design Patterns That Make Sense in Python: Simple Factory
• How to Pass Multiple Arguments to a map Function in Python
• 73 Examples to Help You Master Python's f-strings
• 3 Ways to Test API Client Applications in Python
• Everything You Need to Know About Python's Namedtuples
• The Best Way to Compare Two Dictionaries in Python
• 5 Hidden Python Features You Probably Never Heard Of

See you next time!

In this article you'll learn how to determine if a string is a valid web address or not.

The good new is, you don't need to write your own URL validator. We'll see how we can leverage a third-party URL validator to do the job for us.

Using the validators package

The validators package is a tool that comes with a wide range of validation utilities. You can validate all sorts of inputs such as emails, IP addresses, bitcoin addresses and, of course, URLs.

The URL validation function is available in the root of the module and will return True if the string is a valid URL, otherwise it returns an instance of ValidationFailure, which is a bit weird but not a deal breaker.

>>> import validators

>>> validators.url("http://localhost:8000")
True

The function from previous section can be re-written as follows:

import validators
from validators import ValidationFailure

...


def is_string_an_url(url_string: str) -> bool:
    result = validators.url(url_string)

    if isinstance(result, ValidationFailure):
        return False

    return result

...

>>> is_string_an_url("http://localhost:8000")
True
>>> is_string_an_url("http://.www.foo.bar/")
False

⚠️ WARNING: You must trim all leading and trailing spaces from the URL string before calling validators.url.

# URL has a whitespace at the end
>>> url = "http://localhost:8000 "
>>> is_string_an_url(url)
False
# strip any leading or trailing spaces from the URL
>>> is_string_an_url(url.strip())
True

Using django's URLValidator

django is a great web framework that has many features. It bundles several utilities that makes web development easier. One such utility is the validators module, which contains, amongst other things, an URL validator.

You can validate if a string is, or not, an URL by creating an instance of URLValidator and calling it.

from django.core.validators import URLValidator
from django.core.exceptions import ValidationError

...


def is_string_an_url(url_string: str) -> bool:
    validate_url = URLValidator(verify_exists=True)

    try:
        validate_url(url_string)
    except ValidationError, e:
        return False

    return True

This works well, but adding Django as a dependency just to use its validator is a bit too much. Unless, of course, your project already has Django as a part of it. If not, we have another alternative.

Conclusion

In this post we saw 2 different ways of validating a URL in Python. I hope you find it useful.

Other posts you may like:

• Design Patterns That Make Sense in Python: Simple Factory
• How to Pass Multiple Arguments to a map Function in Python
• 73 Examples to Help You Master Python's f-strings
• Everything You Need to Know About Python's Namedtuples
• The Best Way to Compare Two Dictionaries in Python
• 5 Hidden Python Features You Probably Never Heard Of

See you next time!

Using the os Module to Get the Current Directory

First thing you need to do is to import the module.

>>> import os

Then, you just need call the getcwd function.

>>> import os
...
>>> os.getcwd()
'/home/miguel'

And that is it! As you can see, the function returns a string. This is not very flexible, what if you want to list all the files in that directory? You will have to call it on os functions again.

The os module is pretty ancient and I wouldn’t recommend using it nowadays. In the following section, I’m going to show you the modern way of getting the current working directory in Python.

Getting the Current Working Directory Through the pathlib Module

The pathlib module was proposed in 2012 and added to Python in the 3.4 version. The idea was to provide an object-oriented API for filesystem paths. This module provides classes that represent the filesystem paths with semantics appropriate for different operating systems. Also, Path objects are immutable and hashable, which helps prevent programming errors caused by mutability.

To get the current working directory using pathlib you can use the classmethod cwd from the Path class. But first, you need to import it.

from pathlib import Path

Them, you can call the method.

>>> from pathlib import Path
...
>>> Path.cwd()
PosixPath('/home/miguel')

As you can see, the output is different than the os.getcwd(). As I mentioned earlier, all paths follow the semantics of the underlying filesystem. In my case, I'm using Linux, so the output is a PosixPath. On Windows, cwd returns a WindowsPath.

Being an object allows many cool functionalities such as iterating over all files just by calling a method. However, If you still want to get the string representation, you can call str on the Path.cwd().

>>> str(Path.cwd())
'/home/miguel'

Path.cwd Under the Hood

How does Path know the current directory? The answer is: it calls the os and returns an instance of Path. The following snippet shows the actual implementation.

    @classmethod
    def cwd(cls):
        """Return a new path pointing to the current working directory
        (as returned by os.getcwd()).
        """
        return cls(os.getcwd())

Wait! On Linux it returns a PosixPath but the class method belongs to Path. How does it know?

Great question! Path does some magic behind the scenes before creating the object. It implements the __new__ magic method and calls os to determine the underlying operating system. Check the implementation.

    def __new__(cls, *args, **kwargs):
        if cls is Path:
            cls = WindowsPath if os.name == 'nt' else PosixPath
        self = cls._from_parts(args, init=False)
        if not self._flavour.is_supported:
            raise NotImplementedError("cannot instantiate %r on your system"
                                      % (cls.__name__,))
        self._init()
        return self

Conclusion

That's it for today, folks! I hope you enjoyed this brief article.

Other posts you may like:

• Python pathlib Cookbook: 57+ Examples to Master It (2021)

• 3 Ways to Test API Client Applications in Python

• 5 Hidden Python Features You Probably Never Heard Of

• 7 pytest Features and Plugins That Will Save You Tons of Time

• Everything You Need to Know About Python's Namedtuples

See you next time!

This post was originally published at https://miguendes.me

In another words, how to turn 2-D lists into 1D:

• [[1, 2], [3, 4], [5, 6, 7], [8]] -> [1, 2, 3, 4, 5, 6, 7, 8]?
• [[1, 2], [4, 5], [[[7]]], [[[[8]]]]] -> [1, 2, 3, 4, 5, 6, 7, 8]?
• [1, 2, 3, [4, 5, 6], [[7, 8]]] -> [1, 2, 3, 4, 5, 6, 7, 8]?

What about lists with mixed types such as list of strings, or list of tuples?

• [[1, 2], "three", ["four", "five"]] -> [1, 2, "three", "four", "five"]
• [[1, 2], (3, 4), (5, 6, 7), [8]] -> [1, 2, 3, 4, 5, 6, 7, 8]

In this post, we’ll see how we can unnest an arbitrarily nested list of lists in 7 different ways. Each method has pros and cons, and varies in performance. By going over each one, you’ll learn how to identify the most appropriate solution for your problem by creating your own flatten() function in Python.

For all examples, we'll use Python 3, and for the tests pytest.

By the end of this guide, you'll have learned:

• how to flatten / unnest a list of mixed types, including list of strings, list of tuples or ints
• the best way to flatten lists of lists with list comprehensions
• how to unfold a list and remove duplicates
• how to convert a nested list of lists using the built-in function sum from the standard library
• how to use numpy to flatten nested lists
• how to use itertools chain to create a flat list
• the best way to flatten a nested list using recursion or without recursion

Table of Contents

1. Flattening a list of lists with list comprehensions
2. How to flatten list of strings, tuples or mixed types
3. [How to flatten a nested list and remove duplicates](#how to flatten a list and remove duplicates)
4. Flattening a nested list of lists with the sum function
5. Flattening using itertools.chain
6. Flatten a regular list of lists with numpy

7. Flattening irregular lists

   • The recursive approach

   • The iterative approach

8. Which method is faster? A performance comparison

9. Conclusion

Flattening a list of lists with list comprehensions

Let’s imagine that we have a simple list of lists like this [[1, 3], [2, 5], [1]] and we want to flatten it.

In other words, we want to convert the original list into a flat list like this [1, 3, 2, 5, 1]. The first way of doing that is through list/generator comprehensions. We iterate through each sublist, then iterate over each one of them producing a single element each time.

The following function accepts any multidimensional lists as an argument and returns a generator. The reason for that is to avoid building a whole list in memory. We can then use the generators to create a single list.

To make sure everything works as expected, we can assert the behavior with the test_flatten unit test.

from typing import List, Any, Iterable


def flatten_gen_comp(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions."""
    return (item
            for sublist in lst
            for item in sublist)


def test_flatten():
    lst = [[1, 3], [2, 5], [1]]

    assert list(flatten_gen_comp(lst)) == [1, 3, 2, 5, 1]

This function returns a generator, to get a list back we need to convert the generator to list.

When we run the test we can see it passing...

============================= test session starts ==============================

flatten.py::test_flatten PASSED                                          [100%]

============================== 1 passed in 0.01s ===============================

Process finished with exit code 0

If you prefer you can make the code shorter by using a lambda function.

>>> flatten_lambda = lambda lst: (item for sublist in lst for item in sublist)

>>> lst = [[1, 3], [2, 5], [1]]

>>> list(flatten_lambda(lst))
[1, 3, 2, 5, 1]

How to flatten list of strings, tuples or mixed types

The technique we've seen assumes the items are not iterables. Otherwise, it flattens them as well, which is the case for strings. Let's see what happens if we plug a list of lists and strings.

from typing import List, Any, Iterable


def flatten_gen_comp(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions."""
    return (item
            for sublist in lst
            for item in sublist)


>>> lst = [['hello', 'world'], ['my', 'dear'], 'friend']
>>> list(flatten(lst))
['hello', 'world', 'my', 'dear', 'f', 'r', 'i', 'e', 'n', 'd']

Oops, that's not what we want! The reason is, since one of the items is iterable, the function will unfold them as well.

One way of preventing that is by checking if the item is a list of not. If not, we don't iterate over it.

>>> from typing import List, Any, Iterable


>>> def flatten(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions.
        Returns a flattened version of list lst.
    """

    for sublist in lst:
         if isinstance(sublist, list):
             for item in sublist:
                 yield item
         else:
             yield sublist

>>> lst = [['hello', 'world'], ['my', 'dear'], 'friend']

>>> list(flatten(l))
['hello', 'world', 'my', 'dear', 'f', 'r', 'i', 'e', 'n', 'd']

Since we check if sublist is a list of not, this works with list of tuples as well, any list of iterables, for that matter.

>>> lst = [[1, 2], 3, (4, 5)]
>>> list(flatten(lst))
[1, 2, 3, (4, 5)]

Lastly, this flatten function works for multidimensional list of mixed types.

>>> lst = [[1, 2], 3, (4, 5), ["string"], "hello"]

>>> list(flatten(lst))
[1, 2, 3, (4, 5), 'string', 'hello']

How to flatten a list and remove duplicates

To flatten a list of lists and return a list without duplicates, the best way is to convert the final output to a set.

The only downside is that if the list is big, there'll be a performance penalty since we need to create the set using the generator, then convert set to list.

>>> from typing import List, Any, Iterable


>>> def flatten(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using generators comprehensions.
        Returns a flattened version of list lst.
    """

    for sublist in lst:
         if isinstance(sublist, list):
             for item in sublist:
                 yield item
         else:
             yield sublist

>>> lst = [[1, 2], 3, (4, 5), ["string"], "hello", 3, 4, "hello"]

>>> list(set(flatten(lst)))
[1, 2, 3, 'hello', 4, (4, 5), 'string']

Flattening a list of lists with the sum function

The second strategy is a bit unconventional and, truth to be told, very "magical".

Did you know that we can use the built-in function sum to create flattened lists?

All we need to do is to pass the list as an argument along an empty list. The following code snippet illustrates that.

If you’re curious about this approach, I discuss it in more detail in another post.

from typing import List, Any, Iterable


def flatten_sum(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using sum."""
    return sum(lst, [])


def test_flatten():
    lst = [[1, 3], [2, 5], [1]]

    assert list(flatten_sum(lst)) == [1, 3, 2, 5, 1]

And the tests pass too...

flatten.py::test_flatten PASSED

Even though it seems clever, it's not a good idea to use it in production IMHO. As we'll see later, this is the worst way to flatten a list in terms of performance.

Flattening using itertools.chain

The third alternative is to use the chain function from the itertools module. In a nutshell, chain creates a single iterator from a sequence of other iterables. This function is a perfect match for our use case.

Equivalent implementation of chain taken from the official docs looks like this.

import itertools


def chain(*iterables):
    # chain('ABC', 'DEF') --> A B C D E F
    for it in iterables:
        for element in it:
            yield element

We can then flatten the multi-level lists like so:

def flatten_chain(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using chain."""
    return itertools.chain(*lst)


def test_flatten():
    lst = [[1, 3], [2, 5], [1]]

    assert list(flatten_chain(lst)) == [1, 3, 2, 5, 1]

And the test pass...

[OMITTED]
....
flatten.py::test_flatten PASSED   
...
[OMITTED]

Flatten a regular list of lists with numpy

Another option to create flat lists from nested ones is to use numpy. This library is mostly used to represent and perform operations on multidimensional arrays such as 2D and 3D arrays.

What most people don't know is that some of its function also work with multidimensional lists or other list of iterables. For example, we can use the numpy.concatenate function to flatten a regular list of lists.

>>> import numpy as np

>>> lst = [[1, 3], [2, 5], [1], [7, 8]]

>>> list(np.concatenate(lst))
[1, 3, 2, 5, 1, 7, 8]

Flattening irregular lists

So far we've been flattening regular lists, but what happens if we have a list like this [[1, 3], [2, 5], 1] or this [1, [2, 3], [[4]], [], [[[[[[[[[5]]]]]]]]]]?

Unfortunately, that ends up not so well if we try to apply any of those preceding approaches. In this section, we’ll see two unique solutions for that, one recursive and the other iterative.

The recursive approach

Solving the flatting problem recursively means iterating over each list element and deciding if the item is already flattened or not. If so, we return it, otherwise we can call flatten_recursive on it. But better than words is code, so let’s see some code.

from typing import List, Any, Iterable


def flatten_recursive(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using recursion."""
    for item in lst:
        if isinstance(item, list):
            yield from flatten_recursive(item)
        else:
            yield item

def test_flatten_recursive():
    lst = [[1, 3], [2, 5], 1]

    assert list(flatten_recursive(lst)) == [1, 3, 2, 5, 1]

Bare in mind that the if isinstance(item, list) means it only works with lists. On the other hand, it will flatten lists of mixed types with no trouble.

>>> list(flatten_recursive(lst))
[1, 2, 3, (4, 5), 'string', 'hello']

The iterative approach

The iterative approach is no doubt the most complex of all of them.

In this approach, we’ll use a deque to flatten the irregular list. Quoting the official docs:

"Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short for “double-ended queue”). Deques support thread-safe, memory efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction."

In other words, we can use deque to simulate the stacking operation of the recursive solution.

To do that, we’ll start just like we did in the recursion case, we’ll iterate through each element and if the element is not a list, we’ll append to the left of the deque. The appendleft method append the element to the leftmost position of the deque, for example:

>>> from collections import deque

>>> l = deque()

>>> l.appendleft(2)

>>> l
deque([2])

>>> l.appendleft(7)

>>> l
deque([7, 2])

If the element is a list, though, then we need to reverse it first to pass it to “extendleft” method, like this my_deque.extendleft(reversed(item)). Again, similar to a list, extendleft adds each item to the leftmost position of the deque in series. As a result, deque will add the elements in a reverse order. That’s exactly why we need to reverse the sub-list before extending left. To make things clearer, let’s see an example.

>>> l = deque()

>>> l.extendleft([1, 2, 3])

>>> l
deque([3, 2, 1])

The final step is to iterate over the deque removing the leftmost element and yielding it if it’s not a list. If the element is a list, then we need to extend it left. The full implement goes like this:

from typing import List, Any, Iterable


def flatten_deque(lst: List[Any]) -> Iterable[Any]:
    """Flatten a list using a deque."""
    q = deque()
    for item in lst:
        if isinstance(item, list):
            q.extendleft(reversed(item))
        else:
            q.appendleft(item)
        while q:
            elem = q.popleft()
            if isinstance(elem, list):
                q.extendleft(reversed(elem))
            else:
                yield elem

def test_flatten_super_irregular():
    lst = [1, [2, 3], [4], [], [[[[[[[[[5]]]]]]]]]]

    assert list(flatten_deque(lst)) == [1, 2, 3, 4, 5]

When we run the test, it happily pass...

============================= test session starts ==============================
...

flatten.py::test_flatten_super_irregular PASSED                          [100%]

============================== 1 passed in 0.01s ===============================

This approach can also flatten lists of mixed types with no trouble.

>>> list(flatten_deque(lst))
[1, 2, 3, (4, 5), 'string', 'hello']

Which method is faster? A performance comparison

As we’ve seen in the previous section, if our multi-level list is irregular we have little choice. But assuming that this is not a frequent use case, how these approaches compare in terms of performance?

In this last part, we’ll run a benchmark and compare which solutions perform best.

To do that, we can use the timeit module, we can invoke it in IPython by doing %timeit [code_to_measure]. The following list is the timings for each one of them. As we can see, flatten_chain is the fastest implementation of all. It flatted our list lst in 267 µs avg. The slowest implementation is flatten_sum, taking around 42 ms to flatten the same list.

PS: A special thanks to @hynekcer who pointed out a bug in this benchmark. Since most of the functions return a generator, we need to consume all elements in order to get a better assessment. We can either iterate over the generator or create a list out of it.

Flatten generator comprehension

In [1]: lst = [[1, 2, 3], [4, 5, 6], [7], [8, 9]] * 1_000

In [2]: %timeit list(flatten_gen_comp(lst))
615 µs ± 2.74 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Flatten sum

In [3]: In [19]: %timeit list(flatten_sum(lst))
42 ms ± 660 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Flatten chain

In [4]: %timeit list(flatten_chain(lst))
267 µs ± 517 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Flatten numpy

In [5]: %timeit list(flatten_numpy(lst))
4.65 ms ± 14.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Flatten recursive

In [6]: %timeit list(flatten_recursive(lst))
3.02 ms ± 174 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Flatten deque

In [7]: %timeit list(flatten_deque(lst))
2.97 ms ± 21.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Conclusion

We can flatten a multi-level list in several different ways with Python. Each approach has its pros and cons. In this post we looked at 5 different ways to create 1D lists from nested 2D lists, including:

• regular nested lists
• irregular lists
• list of mixed types
• list of strings
• list of tuples or ints
• recursive and iterative approach
• using the iterools module
• removing duplicates

Other posts you may like:

• Everything You Need to Know About Python's Namedtuples
• The Best Way to Compare Two Dictionaries in Python
• Python's f-strings: 73 Examples to Help You Master It
• Design Patterns That Make Sense in Python: Simple Factory
• How to Pass Multiple Arguments to a map Function in Python
• 3 Ways to Unit Test REST APIs in Python

See you next time!

via GIPHY

Introduction

Design Patterns has been a popular subject since the Design Patterns: Elements of Reusable Object-Oriented Software (a.k.a GoF) book was released back in 1994. GoF’s goals were to show techniques, a.k.a patterns, to improve an object-oriented design. In total, the book demonstrated 23 patterns, classified in 3 groups:

• Creational

• Behavioral

• Structural

Among the creational patterns, we have the Factory Method. According to the book, the goal of this pattern is to define an interface to create an object. The sub classes will then decide which class will be instantiated. There’s also another variation called Simple Factory. This pattern creates an instance of an object without exposing the details behind the construction. In this article, we’ll see how to do that in Python in an idiomatic way.

When this is useful? Can we just call the constructor directly?

This pattern is helpful when you need to perform an extra setup before calling a constructor. In the next section we’ll see several examples on how they are used in the Python standard library and also in third-party packages such as pandas.

Usage

In this part, we’ll see how this pattern is used in practice and how you can implement it yourself.

Python Standard Library

The datetime module is one of the most important ones in the standard library. It defines a few classes such as date, datetime, and timedelta. This module uses the simple factory pattern extensively. A real example is the date class. It has a method called fromtimestamp that creates date instances given a timestamp.

In [3]: from datetime import date

In [4]: date.fromtimestamp(time.time())
Out[4]: datetime.date(2020, 11, 10)

If we look at the implementation, we can see that it extracts the year, month and day from the time instance and the call the constructor (cls). This is the kind of setup that is abstracted away from the user.

    @classmethod
    def fromtimestamp(cls, t):
        "Construct a date from a POSIX timestamp (like time.time())."
        y, m, d, hh, mm, ss, weekday, jday, dst = _time.localtime(t)
        return cls(y, m, d)

Another great example is the fromisocalendar method, which performs an extensive setup. Instead of leaving it to the user, the class provides the functionality “for free” by hiding that from you.

# https://github.com/python/cpython/blob/c304c9a7efa8751b5bc7526fa95cd5f30aac2b92/Lib/datetime.py#L860-L893
...
    @classmethod
    def fromisocalendar(cls, year, week, day):
        """Construct a date from the ISO year, week number and weekday.
        This is the inverse of the date.isocalendar() function"""
        # Year is bounded this way because 9999-12-31 is (9999, 52, 5)
        if not MINYEAR <= year <= MAXYEAR:
            raise ValueError(f"Year is out of range: {year}")

        if not 0 < week < 53:
            out_of_range = True

            if week == 53:
                # ISO years have 53 weeks in them on years starting with a
                # Thursday and leap years starting on a Wednesday
                first_weekday = _ymd2ord(year, 1, 1) % 7
                if (first_weekday == 4 or (first_weekday == 3 and
                                           _is_leap(year))):
                    out_of_range = False

            if out_of_range:
                raise ValueError(f"Invalid week: {week}")

        if not 0 < day < 8:
            raise ValueError(f"Invalid weekday: {day} (range is [1, 7])")

        # Now compute the offset from (Y, 1, 1) in days:
        day_offset = (week - 1) * 7 + (day - 1)

        # Calculate the ordinal day for monday, week 1
        day_1 = _isoweek1monday(year)
        ord_day = day_1 + day_offset

        return cls(*_ord2ymd(ord_day))
....

Pandas

pandas is one of the most used Python packages thanks to the rise of Data Science and Machine Learning. Just like Python, pandas also makes use of factory methods. A classic example is the from_dict method that belongs to the DataFrame class.

        >>> data = {'row_1': [3, 2, 1, 0], 'row_2': ['a', 'b', 'c', 'd']}
        >>> pd.DataFrame.from_dict(data, orient='index')
               0  1  2  3
        row_1  3  2  1  0
        row_2  a  b  c  d

When we inspect the implementation we can also see a lot of setup and extra checks.

    @classmethod
    def from_dict(cls, data, orient="columns", dtype=None, columns=None) -> DataFrame:
        ...
        index = None
        orient = orient.lower()
        if orient == "index":
            if len(data) > 0:
                # TODO speed up Series case
                if isinstance(list(data.values())[0], (Series, dict)):
                    data = _from_nested_dict(data)
                else:
                    data, index = list(data.values()), list(data.keys())
        elif orient == "columns":
            if columns is not None:
                raise ValueError("cannot use columns parameter with orient='columns'")
        else:  # pragma: no cover
            raise ValueError("only recognize index or columns for orient")

        return cls(data, index=index, columns=columns, dtype=dtype)

How to Implement It

The most idiomatic way of implementing factory methods in Python is by decorating them as classmethod. In Python, regular methods are attached to an object instance. We can access the objects’ fields via the self argument. classmethod, on the other hand, are bound not to an instance but to a class. That means when we call MyClass.factory_method we are passing MyClass as the first argument, called cls. This property makes them an excellent alternative for factory methods since calling cls(args) inside a classmethod is the same as MyClass(args).

To design your own factory methods, it’s sufficient to decorate it as a classmethod and return a new instance built with the cls argument. For example, presume that we want to implement a Point class and we want it also be constructed from Polar coordinates. The extra setup to convert from Polar to Cartesian is kept inside the method. Not simply it’s more readable, but also simplifies the constructor.

class Point:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y

    @classmethod
    def from_polar(cls, r: float, theta: float) -> "Point":
        """
        Converts a polar coordinate into cartesian point.

        >>> Point.from_polar(r=-2**0.5, theta=math.pi / 4)
        Point(x=-1.00, y=-1.00)
        """
        return cls(r * math.cos(theta), r * math.sin(theta))

    def __repr__(self):
        return f"{self.__class__.__name__}(x={self.x:.2f}, y={self.y:.2f})"

>>> Point.from_polar(r=-2**0.5, theta=math.pi / 4)
Point(x=-1.00, y=-1.00)

Conclusion

That's pretty much it! I hope you’ve learned something different and useful. Simple Factory methods are very cool and can abstract a lot of boilerplate. Not to mention that it makes your code clean and readable. In this post I showed how this pattern is used in the standard library and in other packages such as pandas.

Other posts you may like:

• How to Pass Multiple Arguments to a map Function in Python
• 73 Examples to Help You Master Python's f-strings
• How to Check if an Exception Is Raised (or Not) With pytest
• 3 Ways to Test API Client Applications in Python
• Everything You Need to Know About Python's Namedtuples
• The Best Way to Compare Two Dictionaries in Python
• 5 Hidden Python Features You Probably Never Heard Of
• 7 pytest Features and Plugins That Will Save You Tons of Time

See you next time!

The map() function is everywhere in Python. It's a built in, it's part of the concurrent.futures.Executor, and also multiprocessing.Pool; but... it's limited!

It's limited because you cannot pass multiple arguments to it. However, what if I told you that there's some easy ways you can do that?

In this post, I’m going to show what you can do to map a function that expects multiple arguments. By the end of this article, you'll know:

• what is a map function and the problem with it
• how to map two or more arguments with itertools.starmap()
• how to use functools.partial to "freeze" and pass multiple arguments to map
• the way to map multiple arguments by "repeating" them
• how to pass multiple args to multiprocessing pool.map
• how to pass multiple arguments to a concurrent futures ProcessPoolExecutor (or ThreadPoolExecutor)?

Let's go!

What Is a Map Function and the Problem With It

A map() is a function that expects one or more iterables and a function as arguments.

For each item in these iterables, map applies the function passed as argument. The result is an iterator where each element is produced by the function you provided as argument. If you pass multiple iterables, you must pass a function that accepts that many arguments.

The Problem

Let’s imagine that you have a function called sum_four that takes 4 arguments and returns their sum.

>>> def sum_four(a, b, c, d):
        return a + b + c + d

Let’s also suppose that you are solving a very specific problem that requires the first 3 arguments to be fixed. In this problem, you want to compare how the function behaves when you vary only the last parameter.

>>> a, b, c = 1, 2, 3

>>> sum_four(a=a, b=b, c=c, d=1)
 7

>>> sum_four(a=a, b=b, c=c, d=2)
 8

>>> sum_four(a=a, b=b, c=c, d=3)
 9

>>> sum_four(a=a, b=b, c=c, d=4)
 10

Now, say that you want to use map, because you like functional programming, or maybe because you come from a language that encourages this paradigm.

Since only d varies, we could store all potential values for d we want to test in a list like this all_d_values = [1, 2, 3, 4].

The issue is, given a function and a list of single elements, if you want to pass that list to a map function and it takes only one element, what can you do?

Solution 1 - Mapping Multiple Arguments with itertools.starmap()

The first solution is to not adopt the map function but use itertools.starmap instead. This function will take a function as arguments and an iterable of tuples. Then, starmap will iterate over each tuple t and call the function by unpacking the arguments, like this for t in tuples: function(*t).

To make things more clear, consider the following example.

>>> import itertools

>>> all_d_values = [1, 2, 3, 4]

>>> items = ((a, b, c, d) for d in all_d_values)

>>> list(items)
 [(1, 2, 3, 1), (1, 2, 3, 2), (1, 2, 3, 3), (1, 2, 3, 4)]

>>> list(itertools.starmap(sum_four, items))
 [7, 8, 9, 10]

As you can see, there’s a lot of repetition, which may inevitably consume a lot of memory if the list is big. To improve that I made items as a generator, this way we only hold in memory the element we’ll be processing.

Solution 2 - Using functools.partial to “Freeze” the Arguments

The second solution is to use currying and create a new partial function. According to the docs, partial() will "freeze" some portion of a function’s arguments and/or keywords resulting in a new function with a simplified signature.

>>> import functools

>>> partial_sum_four = functools.partial(sum_four, a, b, c)

>>> partial_sum_four(3)
9

>>> list(map(partial_sum_four, all_d_values))
[7, 8, 9, 10]

Solution 3 - Mapping Multiple Arguments by "Repeating" Them

The third alternative is to use the itertools.repeat().

This function produces an iterator that returns object over and over again. It will run indefinitely if you don’t specify the times argument.

If we take a closer look at map()'s signature, it accepts a function and multiple iterables, map(function, iterable, ...).

According to its description,

If additional iterable arguments are passed, function must take that many arguments and is applied to the items from all iterables in parallel. With multiple iterables, the iterator stops when the shortest iterable is exhausted.

Bingo! We can make a, b and c infitnite iterables by using itertools.repeat(). As soon as all_d_values is exhausted, which is the shortest iterable, map() will stop.

>>> import itertools
>>> list(map(sum_four, itertools.repeat(a), itertools.repeat(b), itertools.repeat(c), all_d_values))
 [7, 8, 9, 10]

To put it another way, using repeat() is roughly equivalent to:

>>> list(map(sum_four, [1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3], all_d_values))
 [7, 8, 9, 10]

You don't need to worry too much about memory as repeat produces the elements on the go. In fact, it returns a repeatobject, not list [ref] .

Problem 2: Passing Multiple Parameters to multiprocessing Pool.map

This problem is very similar to using the regular map(). The only difference is that we need to pass multiple arguments to the multiprocessing's pool map.

Suppose that we want to speed up our code and run sum_four in parallel using processes.

The good news is, you can use the solutions above, with one exception: Pool.map only accepts one iterable. This means we cannot use repeat() here. Let's see the alternatives.

Using pool.starmap

The Pool class from multiprocessing module implements a starmap function that works the same way as its counterpart from the itertools module.

>>> from multiprocessing import Pool

>>> import itertools

>>> def sum_four(a, b, c, d):
                return a + b + c + d

>>> a, b, c = 1, 2, 3

>>> all_d_values = [1, 2, 3, 4]

>>> items = [(a, b, c, d) for d in all_d_values]

>>> items
 [(1, 2, 3, 1), (1, 2, 3, 2), (1, 2, 3, 3), (1, 2, 3, 4)]

>>> with Pool(processes=4) as pool:
         res = pool.starmap(sum_four, items)

>>> res
 [7, 8, 9, 10]

Using partial()

As alternative, we can also rely on the good partial function.

>>> import functools

>>> partial_sum_four = functools.partial(sum_four, a, b, c)

>>> with Pool(processes=4) as pool:
         res = pool.map(partial_sum_four, all_d_values)

>>> res
 [7, 8, 9, 10]

Problem 3: How to Pass Multiple Arguments to concurrent futures Executor.map?

The concurrent.futures module provides a high-level interface called Executor to run callables asynchronously.

There are two different implementations available, a ThreadPoolExecutor and a ProcessPoolExecutor.

Contrary to multiprocessing.Pool, a Executor does not have a startmap() function. However, its map() implementation supports multiple iterables, which allow us to use repeat(). Another difference is that Executor.map returns a generator, not a list.

Using partial() With a ProcessPoolExecutor (or ThreadPoolExecutor)

By "freezing" the arguments using partial we use the map method from ProcessPoolExecutor like a regular map function. Since they both share the same interface, you can do the same interchangeably with a ThreadPoolExecutor

>>> from concurrent.futures import ProcessPoolExecutor

>>> import functools

>>> def sum_four(a, b, c, d):
                return a + b + c + d

>>> a, b, c = 1, 2, 3

>>> all_d_values = [1, 2, 3, 4]

>>> partial_sum_four = functools.partial(sum_four, a, b, c)

>>> with ProcessPoolExecutor(max_workers=4) as pool:
              res = list(pool.map(partial_sum_four, all_d_values))

>>> res
 [7, 8, 9, 10]

Using repeat()

Again, we can just use itertools.repeat to get the job done like the previous solutions.

>>> from concurrent.futures import ProcessPoolExecutor

>>> from itertools import repeat

>>> def sum_four(a, b, c, d):
                return a + b + c + d

>>> a, b, c = 1, 2, 3

>>> all_d_values = [1, 2, 3, 4]

>>> with ProcessPoolExecutor(max_workers=4) as pool:
              res = list(pool.map(sum_four, repeat(a), repeat(b), repeat(c), all_d_values))

>>> res
 [7, 8, 9, 10]

Conclusion

That’s it for today, folks! I hope you’ve learned something different and useful. The map() function makes Python feel like a functional programming language. map() is available not only as a built-in function but also as methods in the multiprocessing and concurrent.futures module. In this article, I showed what I do to map functions that take several arguments.

Other posts you may like:

• How to Use datetime.timedelta in Python With Examples
• 73 Examples to Help You Master Python's f-strings
• How to Check if an Exception Is Raised (or Not) With pytest
• 3 Ways to Test API Client Applications in Python
• Everything You Need to Know About Python's Namedtuples
• The Best Way to Compare Two Dictionaries in Python
• 5 Hidden Python Features You Probably Never Heard Of

Time is a precious resource so I won't waste yours. In this post, you'll learn how to use a pytest fixture in parametrize using a library or getfixturevalue.

Introduction

In this post, we'll see how we can use pytest.mark.parametrize with fixtures. This is a long-wanted feature that dates back to 2013. Even though pytest doesn't support it yet, you'll see that we can actually make it happen.

Problem

You want to pass a fixture to parametrize.

Suppose that you have a simple function called is_even(n) that returns true if n is divisible by 2. Then you create a simple test for it that receives a fixture named two that returns 2. To make the test more robust, you set up another fixture named four that returns 4. Now you have two individual tests, as illustrated below.

Implementation:

def is_even(n: int) -> bool:
    """Returns True if n is even."""
    return n % 2 == 0

Tests:

@pytest.fixture()
def two():
    return 2

@pytest.fixture()
def four():
    return 4

def test_four_is_even(four):
    """Asserts that four is even"""
    assert is_even(four)

def test_two_is_even(two):
    """Asserts that two is even"""
    assert is_even(two)

If we run these tests, they pass, which is good. Even though you’re quite happy with the outcome, you need to test one more thing. You want to assert that the multiplication of an even number by and odd one produces an even result. To accomplish that, you create two more fixtures, one and three. You plan to use them as arguments in a parameterized test, like so:

@pytest.fixture()
def one():
    return 1

@pytest.fixture()
def three():
    return 3

@pytest.mark.parametrize(
    "a, b",
    [
        (one, four),
        (two, three),
    ],
)
def test_multiply_is_even(a, b):
    """Assert that an odd number times even is even."""
    assert is_even(a * b)

When we run this test, we get the following output:

_______________________ test_multiply_is_even[two-three] _______________________

a = <function two at 0x7f9d862ee790>, b = <function three at 0x7f9d862eedc0>

    @pytest.mark.parametrize(
        "a, b",
        [
            (one, four),
            (two, three),
        ],
    )
    def test_multiply_is_even(a, b):
        """Assert that an odd number times even is even."""
>       assert is_even(a * b)
E       TypeError: unsupported operand type(s) for *: 'function' and 'function'

tests/test_variables.py:71: TypeError
=========================== short test summary info ============================
FAILED tests/test_variables.py::test_multiply_is_even[one-four] - TypeError: ...
FAILED tests/test_variables.py::test_multiply_is_even[two-three] - TypeError:...
============================== 2 failed in 0.05s ===============================

As you can see, passing a fixture as argument in a parameterized test doesn't work.

Solution

To make that possible, we have two alternatives. The first one is using request.getfixturevalue, which is available on pytest. This function dynamically runs a named fixture function.

@pytest.mark.parametrize(
    "a, b",
    [
        ("one", "four"),
        ("two", "three"),
    ],
)
def test_multiply_is_even_request(a, b, request):
    """Assert that an odd number times even is even."""
    a = request.getfixturevalue(a)
    b = request.getfixturevalue(b)
    assert is_even(a * b)

If we run the test again we get the following:

============================= test session starts ==============================
...
collecting ... collected 2 items

tests/test_variables.py::test_multiply_is_even_request[one-four] PASSED  [ 50%]
tests/test_variables.py::test_multiply_is_even_request[two-three] PASSED [100%]

============================== 2 passed in 0.02s ===============================

Process finished with exit code 0

Great! It works like a charm. However, there’s one more alternative, and for that we’ll need a third-party package called pytest-lazy-fixture. Let’s see how the test looks like using this lib.

@pytest.mark.parametrize(
    "a, b",
    [
        (pytest.lazy_fixture(("one", "four"))),
        # same as (pytest.lazy_fixture(("two", "three")))
        (pytest.lazy_fixture("two"), pytest.lazy_fixture("three")), 
    ],
)
def test_multiply(a, b):
    """Assert that an odd number times even is even."""
    assert is_even(a * b)

In this example, we use it by passing a tuple with the fixtures names or passing each one of them as a different argument. When we run this test, we can see it passes!

============================= test session starts ==============================
...
collecting ... collected 2 items

tests/test_variables.py::test_multiply[one-four] PASSED                  [ 50%]
tests/test_variables.py::test_multiply[two-three] PASSED                 [100%]

============================== 2 passed in 0.02s ===============================

Process finished with exit code 0

Conclusion

That’s it for today, folks! I hope you’ve learned something different and useful. Being able to reuse fixtures in parametrized tests is a must when we want to avoid repetition. Unfortunately, pytest doesn’t support that yet. On the other hand, we can make it happen either by using getfixturevalue in pytest or through a third-party library.

Other posts you may like:

• Learn how to unit test REST APIs in Python with Pytest by example.

• 7 pytest Features and Plugins That Will Save You Tons of Time

• How to Use Fixtures as Arguments in pytest.mark.parametrize

• How to Check if an Exception Is Raised (or Not) With pytest

• 7 pytest Plugins You Must Definitely Use

• How to Disable Autouse Fixtures in pytest

• How to Use datetime.timedelta in Python With Examples

• 73 Examples to Help You Master Python's f-strings

• The Best Way to Compare Two Dictionaries in Python

See you next time!

This post was originally published at https://miguendes.me

With timedelta you can add days, minutes, seconds, hours, weeks and more to a datetime.date, or a datetime.datetime object.

You'll also learn how to:

• convert a timedelta to seconds, minutes, hours, or days
• convert a time delta to years
• how to take the difference between two dates
• how to format a time delta as string

Let's go!

Table of Contents

• Adding Seconds, Minutes, Hours, Days, Weeks and Whatnot to a Date
  • How to Use timedelta to Add Days to a date or datetime Object
  • How to Use timedelta to Add Minutes to a datetime Object
  • How to Use timedelta to Add Weeks, Hours, Seconds, Milliseconds to a datetime
  • How to Add Years to a datetime in Python
  • How to Add Months to a datetime in Python
• How to convert a timedelta to seconds, minutes, hours, or days
• How to Take the Difference Between Two Dates
  • How to Calculate the Number of Days Between Two Dates
  • How to Calculate the Number of Minutes Between Two Dates
• How to Format a timedelta as string
• Conclusion

Adding Seconds, Minutes, Hours, Days, Weeks and Whatnot to a Date

A timedelta object denotes a duration, it can also represent the difference between two dates or times.

We can use this object to add to or subtract a duration from a date, and it defines its constructor as datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0). As you can see, all arguments are optional and default to 0. It can take ints or floats, positive or negative.

Even though you can pass weeks, hours, minutes and milliseconds only days, seconds, and microseconds are stored internally.

In this section, we'll see basic arithmetic operations such as adding/subtracting a duration to/from a date.

How to Use timedelta to Add Days to a date or datetime Object

Since timedelta represents a duration, we can use it to add days to a datetime. The number of can be positive or negative, thus allowing us to create a date in the future or in the past. The code snippet below shows an example.

>>> import datetime

>>> now = datetime.datetime.now()

>>> now
datetime.datetime(2020, 11, 3, 22, 5, 21, 979147)

>>> from datetime import timedelta

>>> now + timedelta(days=3)
datetime.datetime(2020, 11, 6, 22, 5, 21, 979147)

>>> now + timedelta(days=-3)
datetime.datetime(2020, 10, 31, 22, 5, 21, 979147)

As you can see, adding a positive number of days yields a future date whereas adding a negative number brings the date to the past.

If you want to add days to a date object, the process is the same.

>>> today = datetime.date.today()

>>> today
datetime.date(2020, 11, 5)

>>> today + timedelta(days=3)
datetime.date(2020, 11, 8)

How to Use timedelta to Add Minutes to a datetime Object

Since timedelta object sets all arguments to 0 by default, we have the option to set only the ones we need. This allows us to add only minutes, for instance.

>>> now
datetime.datetime(2020, 11, 3, 22, 5, 21, 979147)

>>> now + timedelta(minutes=3)
datetime.datetime(2020, 11, 3, 22, 8, 21, 979147)

>>> now + timedelta(minutes=-3)
datetime.datetime(2020, 11, 3, 22, 2, 21, 979147)

How to Use timedelta to Add Weeks, Hours, Seconds, Milliseconds to a datetime

Adding weeks, seconds, milliseconds and even microseconds works in a similar fashion.

>>> now + timedelta(weeks=3)
datetime.datetime(2020, 11, 24, 22, 5, 21, 979147)

>>> now + timedelta(hours=3)
datetime.datetime(2020, 11, 4, 1, 5, 21, 979147)

>>> now + timedelta(microseconds=3)
datetime.datetime(2020, 11, 3, 22, 5, 21, 979150)

>>> now + timedelta(milliseconds=3)
datetime.datetime(2020, 11, 3, 22, 5, 21, 982147)

How to Add Years to a datetime in Python

It's definitely possible to use timedelta to add years to a datetime, but some things can go wrong and it's easy to shoot yourself in the foot. For example, you need to take into account leap years yourself.

IMHO, the best way to add a certain number of years to a datetime is by using the dateutil library.

>>> import datetime

>>> from dateutil.relativedelta import relativedelta

>>> now = datetime.datetime.now()

>>> now
datetime.datetime(2020, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(years=2)
datetime.datetime(2022, 11, 4, 22, 9, 5, 672091)

How to Add Months to a datetime in Python

Adding months to a datetime has the same problem as adding years using timedelta. This feature is not supported by default and requires manual calculation. You can use days, but you’d need to know how many days that month has and so. In a nutshell, it’s too error prone. Again, the best you can do is to use dateutil.relativedelta.

>>> from dateutil.relativedelta import relativedelta
>>> now
datetime.datetime(2020, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(years=2)
datetime.datetime(2022, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(months=12)
datetime.datetime(2021, 11, 4, 22, 9, 5, 672091)

>>> now + relativedelta(months=24)
datetime.datetime(2022, 11, 4, 22, 9, 5, 672091)