Python pathlib Cookbook: 57+ Examples to Master It (2022)
A mega tutorial with dozens of examples on how to use the pathlib module in Python 3
When I started learning Python, there was one thing I always had trouble with: dealing with directories and file paths!
I remember the struggle to manipulate paths as strings using the os
module. I was constantly looking up error messages related to improper path manipulation.
The os
module never felt intuitive and ergonomic to me, but my luck changed when pathlib
landed in Python 3.4. It was a breath of fresh air, much easier to use, and felt more Pythonic to me.
The only problem was: finding examples on how to use it was hard; the documentation only covered a few use cases. And yes, Python's docs are good, but for newcomers, examples are a must.
Even though the docs are much better now, they don't showcase the module in a problem-solving fashion. That’s why I decided to create this cookbook.
This article is a brain dump of everything I know about pathlib
. It's meant to be a reference rather than a linear guide. Feel free to jump around to sections that are more relevant to you.
In this guide, we'll go over dozens of use cases such as:
- how to create (touch) an empty file
- how to convert a path to string
- getting the home directory
- creating new directories, doing it recursively, and dealing with issues when they
- getting the current working directory
- get the file extension from a filename
- get the parent directory of a file or script
- read and write text or binary files
- how to delete files
- how create nested directories
- how to list all files and folders in a directory
- how to list all subdirectories recursively
- how to remove a directory along with its contents
I hope you enjoy!
Table of contents
- What is
pathlib
in Python? - The anatomy of a
pathlib.Path
- How to convert a path to string
- How to join a path by adding parts or other paths
- Working with directories using
pathlib
- How to get the current working directory (cwd) with
pathlib
- How to get the home directory with
pathlib
- How to expand the initial path component with
Path.expanduser()
- How to list all files and directories
- Using
isdir
to list only the directories - Getting a list of all subdirectories in the current directory recursively
- How to recursively iterate through all files
- How to change directories with Python pathlib
- How to delete directories with
pathlib
- How to remove a directory along with its contents with
pathlib
- How to get the current working directory (cwd) with
- Working with files using
pathlib
- How to touch a file and create parent directories
- How to get the filename from path
- How to get the file extension from a filename using
pathlib
- How to open a file for reading with
pathlib
- How to read text files with
pathlib
- How to read JSON files from path with
pathlib
- How to write a text file with
pathlib
- How to copy files with
pathlib
- How to delete a file with
pathlib
- How to delete all files in a directory with
pathlib
- How to rename a file using
pathlib
- How to get the parent directory of a file with
pathlib
- Conclusion
What is pathlib
in Python?
pathlib
is a Python module created to make it easier to work with paths in a file system. This module debuted in Python 3.4 and was proposed by the PEP 428.
Prior to Python 3.4, the os
module from the standard library was the go to module to handle paths. os
provides several functions that manipulate paths represented as plain Python strings. For example, to join two paths using os
, one can use theos.path.join
function.
>>> import os
>>> os.path.join('/home/user', 'projects')
'/home/user/projects'
>>> os.path.expanduser('~')
'C:\\Users\\Miguel'
>>> home = os.path.expanduser('~')
>>> os.path.join(home, 'projects')
'C:\\Users\\Miguel\\projects'
Representing paths as strings encourages inexperienced Python developers to perform common path operations using string method. For example, joining paths with +
instead of using os.path.join()
, which can lead to subtle bugs and make the code hard to reuse across multiple platforms.
Moreover, if you want the path operations to be platform agnostic, you will need multiple calls to various os
functions such as os.path.dirname()
, os.path.basename()
, and others.
In an attempt to fix these issues, Python 3.4 incorporated the pathlib
module. It provides a high-level abstraction that works well under POSIX systems, such as Linux as well as Windows. It abstracts way the path's representation and provides the operations as methods.
The anatomy of a pathlib.Path
To make it easier to understand the basics components of a Path
, in this section we'll their basic components.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/blog/config.tar.gz')
>>> path.drive
'/'
>>> path.root
'/'
>>> path.anchor
'/'
>>> path.parent
PosixPath('/home/miguel/projects/blog')
>>> path.name
'config.tar.gz'
>>> path.stem
'config.tar'
>>> path.suffix
'.gz'
>>> path.suffixes
['.tar', '.gz']
>>> from pathlib import Path
>>> path = Path(r'C:/Users/Miguel/projects/blog/config.tar.gz')
>>> path.drive
'C:'
>>> path.root
'/'
>>> path.anchor
'C:/'
>>> path.parent
WindowsPath('C:/Users/Miguel/projects/blog')
>>> path.name
'config.tar.gz'
>>> path.stem
'config.tar'
>>> path.suffix
'.gz'
>>> path.suffixes
['.tar', '.gz']
How to convert a path to string
pathlib
implements the magic __str__
method, and we can use it convert a path to string. Having this method implemented means you can get its string representation by passing it to the str
constructor, like in the example below.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/tutorial')
>>> str(path)
'/home/miguel/projects/tutorial'
>>> repr(path)
"PosixPath('/home/miguel/projects/blog/config.tar.gz')"
The example above illustrates a PosixPath
, but you can also convert a WindowsPath to string using the same mechanism.
>>> from pathlib import Path
>>> path = Path(r'C:/Users/Miguel/projects/blog/config.tar.gz')
# when we convert a WindowsPath to string, Python adds backslashes
>>> str(path)
'C:\\Users\\Miguel\\projects\\blog\\config.tar.gz'
# whereas repr returns the path with forward slashes as it is represented on Windows
>>> repr(path)
"WindowsPath('C:/Users/Miguel/projects/blog/config.tar.gz')"
How to join a path by adding parts or other paths
One of the things I like the most about pathlib
is how easy it is to join two or more paths, or parts. There are three main ways you can do that:
- you can pass all the individual parts of a path to the constructor
- use the
.joinpath
method - use the
/
operator
>>> from pathlib import Path
# pass all the parts to the constructor
>>> Path('.', 'projects', 'python', 'source')
PosixPath('projects/python/source')
# Using the / operator to join another path object
>>> Path('.', 'projects', 'python') / Path('source')
PosixPath('projects/python/source')
# Using the / operator to join another a string
>>> Path('.', 'projects', 'python') / 'source'
PosixPath('projects/python/source')
# Using the joinpath method
>>> Path('.', 'projects', 'python').joinpath('source')
PosixPath('projects/python/source')
On Windows, Path
returns a WindowsPath
instead, but it works the same way as in Linux.
>>> Path('.', 'projects', 'python', 'source')
WindowsPath('projects/python/source')
>>> Path('.', 'projects', 'python') / Path('source')
WindowsPath('projects/python/source')
>>> Path('.', 'projects', 'python') / 'source'
WindowsPath('projects/python/source')
>>> Path('.', 'projects', 'python').joinpath('source')
WindowsPath('projects/python/source')
Working with directories using pathlib
In this section, we'll see how we can traverse, or walk, through directories with pathlib
. And when it comes to navigating folders, there many things we can do, such as:
- getting the current working directory
- getting the home directory
- expanding the home directory
- creating new directories, doing it recursively, and dealing with issues when they already exist
- how create nested directories
- listing all files and folders in a directory
- listing only folders in a directory
- listing only the files in a directory
- getting the number of files in a directory
- listing all subdirectories recursively
- listing all files in a directory and subdirectories recursively
- recursively listing all files with a given extension or pattern
- changing current working directories
- removing an empty directory
- removing a directory along with its contents
How to get the current working directory (cwd) with pathlib
The pathlib
module provides a classmethod Path.cwd()
to get the current working directory in Python. It returns a PosixPath instance on Linux, or other Unix systems such as macOS or OpenBSD. Under the hood, Path.cwd()
is just a wrapper for the classic os.getcwd()
.
>>> from pathlib import Path
>>> Path.cwd()
PosixPath('/home/miguel/Desktop/pathlib')
On Windows, it returns a WindowsPath.
>>> from pathlib import Path
>>> Path.cwd()
>>> WindowsPath('C:/Users/Miguel/pathlib')
You can also print it by converting it to string using a f-string, for example.
>>> from pathlib import Path
>>> print(f'This is the current directory: {Path.cwd()}')
This is the current directory: /home/miguel/Desktop/pathlib
PS: If you
How to get the home directory with pathlib
When pathlib
arrived in Python 3.4, a Path
had no method for navigating to the home directory. This changed on Python 3.5, with the inclusion of the Path.home()
method.
In Python 3.4, one has to use os.path.expanduser
, which is awkward and unintuitive.
# In python 3.4
>>> import pathlib, os
>>> pathlib.Path(os.path.expanduser("~"))
PosixPath('/home/miguel')
From Python 3.5 onwards, you just call Path.home()
.
# In Python 3.5+
>>> import pathlib
>>> pathlib.Path.home()
PosixPath('/home/miguel')
Path.home()
also works well on Windows.
>>> import pathlib
>>> pathlib.Path.home()
WindowsPath('C:/Users/Miguel')
How to expand the initial path component with Path.expanduser()
In Unix systems, the home directory can be expanded using ~
( tilde symbol). For example, this allows us to represent full paths like this: /home/miguel/Desktop
as just: ~/Desktop/
.
>>> from pathlib import Path
>>> path = Path('~/Desktop/')
>>> path.expanduser()
PosixPath('/home/miguel/Desktop')
Despite being more popular on Unix systems, this representation also works on Windows.
>>> path = Path('~/projects')
>>> path.expanduser()
WindowsPath('C:/Users/Miguel/projects')
>>> path.expanduser().exists()
True
What's the opposite of
os.path.expanduser()
?
Unfortunately, the pathlib
module doesn't have any method to do the inverse operation. If you want to condense the expanded path back to its shorter version, you need to get the path relative to your home directory using Path.relative_to
, and place the ~
in front of it.
>>> from pathlib import Path
>>> path = Path('~/Desktop/')
>>> expanded_path = path.expanduser()
>>> expanded_path
PosixPath('/home/miguel/Desktop')
>>> '~' / expanded_path.relative_to(Path.home())
PosixPath('~/Desktop')
Creating directories with pathlib
A directory is nothing more than a location for storing files and other directories, also called folders. pathlib.Path
comes with a method to create new directories named Path.mkdir()
.
This method takes three arguments:
mode
: Used to determine the file mode and access flagsparents
: Similar to themkdir -p
command in Unix systems. Default toFalse
which means it raises errors if there's the parent is missing, or if the directory is already created. When it'sTrue
,pathlib.mkdir
creates the missing parent directories.exist_ok
: Defaults toFalse
and raisesFileExistsError
if the directory being created already exists. When you set it toTrue
,pathlib
ignores the error if the last part of the path is not an existing non-directory file.
>>> from pathlib import Path
# lists all files and directories in the current folder
>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
PosixPath('/home/miguel/path/reports')]
# create a new path instance
>>> path = Path('new_directory')
# only the path instance has been created, but it doesn't exist on disk yet
>>> path.exists()
False
# create path on disk
>>> path.mkdir()
# now it exsists
>>> path.exists()
True
# indeed, it shows up
>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
PosixPath('/home/miguel/path/reports'),
PosixPath('/home/miguel/path/new_directory')]
Creating a directory that already exists
When you have a directory path and it already exists, Python raises FileExistsError
if you call Path.mkdir()
on it. In the previous section, we briefly mentioned that this happens because by default the exist_ok
argument is set to False
.
>>> from pathlib import Path
>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
PosixPath('/home/miguel/path/reports'),
PosixPath('/home/miguel/path/new_directory')]
>>> path = Path('new_directory')
>>> path.exists()
True
>>> path.mkdir()
---------------------------------------------------------------------------
FileExistsError Traceback (most recent call last)
<ipython-input-25-4b7d1fa6f6eb> in <module>
----> 1 path.mkdir()
~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in mkdir(self, mode, parents, exist_ok)
1311 try:
-> 1312 self._accessor.mkdir(self, mode)
1313 except FileNotFoundError:
1314 if not parents or self.parent == self:
FileExistsError: [Errno 17] File exists: 'new_directory'
To create a folder that already exists, you need to set exist_ok
to True
. This is useful if you don't want to check using if
's or deal with exceptions, for example. Another benefit is that is the directory is not empty, pathlib
won't override it.
>>> path = Path('new_directory')
>>> path.exists()
True
>>> path.mkdir(exist_ok=True)
>>> list(Path.cwd().iterdir())
[PosixPath('/home/miguel/path/not_created_yet'),
PosixPath('/home/miguel/path/reports'),
PosixPath('/home/miguel/path/new_directory')]
>>> (path / 'new_file.txt').touch()
>>> list(path.iterdir())
[PosixPath('new_directory/new_file.txt')]
>>> path.mkdir(exist_ok=True)
# the file is still there, pathlib didn't overwrote it
>>> list(path.iterdir())
[PosixPath('new_directory/new_file.txt')]
How to create parent directories recursively if not exists
Sometimes you might want to create not only a single directory but also a parent and a subdirectory in one go.
The good news is that Path.mkdir()
can handle situations like this well thanks to its parents
argument. When parents
is set to True
, pathlib.mkdir
creates the missing parent directories; this behavior is similar to the mkdir -p
command in Unix systems.
>>> from pathlib import Path
>>> path = Path('new_parent_dir/sub_dir')
>>> path.mkdir()
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-35-4b7d1fa6f6eb> in <module>
----> 1 path.mkdir()
~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in mkdir(self, mode, parents, exist_ok)
1311 try:
-> 1312 self._accessor.mkdir(self, mode)
1313 except FileNotFoundError:
1314 if not parents or self.parent == self:
FileNotFoundError: [Errno 2] No such file or directory: 'new_parent_dir/sub_dir'
>>> path.mkdir(parents=True)
>>> path.exists()
True
>>> path.parent
PosixPath('new_parent_dir')
>>> path
PosixPath('new_parent_dir/sub_dir')
How to list all files and directories
There are many ways you can list files in a directory with Python's pathlib
. We'll see each one in this section.
To list all files in a directory, including other directories, you can use the Path.iterdir()
method. For performance reasons, it returns a generator that you can either use to iterate over it, or just convert to a list for convenience.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> list(path.iterdir())
[PosixPath('/home/miguel/projects/pathlib/script.py'),
PosixPath('/home/miguel/projects/pathlib/README.md'),
PosixPath('/home/miguel/projects/pathlib/tests'),
PosixPath('/home/miguel/projects/pathlib/src')]
Using isdir
to list only the directories
We've seen that iterdir
returns a list of Path
s. To list only the directories in a folder, you can use the Path.is_dir()
method. The example below will get all the folder names inside the directory.
⚠️ WARNING: This example only lists the immediate subdirectories in Python. In the next subsection, we'll see how to list all subdirectories.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> [p for p in path.iterdir() if p.is_dir()]
[PosixPath('/home/miguel/projects/pathlib/tests'),
PosixPath('/home/miguel/projects/pathlib/src')]
Getting a list of all subdirectories in the current directory recursively
In this section, we'll see how to navigate in directory and subdirectories. This time we'll use another method from pathlib.Path
named glob
.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> [p for p in path.glob('**/*') if p.is_dir()]
[PosixPath('/home/miguel/projects/pathlib/tests'),
PosixPath('/home/miguel/projects/pathlib/src'),
PosixPath('/home/miguel/projects/pathlib/src/dir')]
As you see, Path.glob
will also print the subdirectory src/dir
.
Remembering to pass '**/
to glob()
is a bit annoying, but there's a way to simplify this by using Path.rglob()
.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> [p for p in path.rglob('*') if p.is_dir()]
[PosixPath('/home/miguel/projects/pathlib/tests'),
PosixPath('/home/miguel/projects/pathlib/src'),
PosixPath('/home/miguel/projects/pathlib/src/dir')]
How to list only the files with is_file
Just as pathlib
provides a method to check if a path is a directory, it also provides one to check if a path is a file. This method is called Path.is_file()
, and you can use to filter out the directories and print all file names in a folder.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> [p for p in path.iterdir() if p.is_file()]
[PosixPath('/home/miguel/projects/pathlib/script.py'),
PosixPath('/home/miguel/projects/pathlib/README.md')]
⚠️ WARNING: This example only lists the files inside the current directory. In the next subsection, we'll see how to list all files inside the subdirectories as well.
Another nice use case is using Path.iterdir()
to count the number of files inside a folder.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> len([p for p in path.iterdir() if p.is_file()])
2
How to recursively iterate through all files
In previous sections, we used Path.rglob()
to list all directories recursively, we can do the same for files by filtering the paths using the Path.is_file()
method.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> [p for p in path.rglob('*') if p.is_file()]
[PosixPath('/home/miguel/projects/pathlib/script.py'),
PosixPath('/home/miguel/projects/pathlib/README.md'),
PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]
How to recursively list all files with a given extension or pattern
In the previous example, we list all files in a directory, but what if we want to filter by extension? For that, pathlib.Path
has a method named match()
, which returns True
if matching is successful, and False
otherwise.
In the example below, we list all .py
files recursively.
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> [p for p in path.rglob('*') if p.is_file() and p.match('*.py')]
[PosixPath('/home/miguel/projects/pathlib/script.py'),
PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]
We can use the same trick for other kinds of files. For example, we might want to list all images in a directory or subdirectories.
>>> from pathlib import Path
>>> path = Path('/home/miguel/pictures')
>>> [p for p in path.rglob('*')
if p.match('*.jpeg') or p.match('*.jpg') or p.match('*.png')
]
[PosixPath('/home/miguel/pictures/dog.png'),
PosixPath('/home/miguel/pictures/london/sunshine.jpg'),
PosixPath('/home/miguel/pictures/london/building.jpeg')]
We can actually simplify it even further, we can use only Path.glob
and Path.rglob
to matching. (Thanks to u/laundmo
and u/SquareRootsi
for pointing out!)
>>> from pathlib import Path
>>> path = Path('/home/miguel/projects/pathlib')
>>> list(path.rglob('*.py'))
[PosixPath('/home/miguel/projects/pathlib/script.py'),
PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]
>>> list(path.glob('*.py'))
[PosixPath('/home/miguel/projects/pathlib/script.py')]
>>> list(path.glob('**/*.py'))
[PosixPath('/home/miguel/projects/pathlib/script.py'),
PosixPath('/home/miguel/projects/pathlib/tests/test_script.py'),
PosixPath('/home/miguel/projects/pathlib/src/dir/walk.py')]
How to change directories with Python pathlib
Unfortunately, pathlib
has no built-in method to change directories. However, it is possible to combine it with the os.chdir()
function, and use it to change the current directory to a different one.
⚠️ WARNING: For versions prior to 3.6,
os.chdir
only accepts paths as string.
>>> import pathlib
>>> pathlib.Path.cwd()
PosixPath('/home/miguel')
>>> target_dir = '/home'
>>> os.chdir(target_dir)
>>> pathlib.Path.cwd()
PosixPath('/home')
How to delete directories with pathlib
Deleting directories using pathlib
depends on if the folder is empty or not. To delete an empty directory, we can use the Path.rmdir()
method.
>>> from pathlib import Path
>>> path = Path('new_empty_dir')
>>> path.mkdir()
>>> path.exists()
True
>>> path.rmdir()
>>> path.exists()
False
If we put some file or other directory inside and try to delete, Path.rmdir()
raises an error.
>>> from pathlib import Path
>>> path = Path('non_empty_dir')
>>> path.mkdir()
>>> (path / 'file.txt').touch()
>>> path
PosixPath('non_empty_dir')
>>> path.exists()
True
>>> path.rmdir()
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-64-00bf20b27a59> in <module>
----> 1 path.rmdir()
~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in rmdir(self)
1350 Remove this directory. The directory must be empty.
...
-> 1352 self._accessor.rmdir(self)
1353
1354 def lstat(self):
OSError: [Errno 39] Directory not empty: 'non_empty_dir'
Now, the question is: how to delete non-empty directories with pathlib
?
This is what we'll see next.
How to remove a directory along with its contents with pathlib
To delete a non-empty directory, we need to remove its contents, everything.
To do that with pathlib
, we need to create a function that uses Path.iterdir()
to walk or traverse the directory and:
- if the path is a file, we call
Path.unlink()
- otherwise, we call the function recursively. When there are no more files, that is, when the folder is empty, just call
Path.rmdir()
Let's use the following example of a non empty directory with nested folder and files in it.
$ tree /home/miguel/Desktop/blog/pathlib/sandbox/
/home/miguel/Desktop/blog/pathlib/sandbox/
├── article.txt
└── reports
├── another_nested
│ └── some_file.png
└── article.txt
2 directories, 3 files
To remove it we can use the following recursive function.
>>> from pathlib import Path
>>> def remove_all(root: Path):
for path in root.iterdir():
if path.is_file():
print(f'Deleting the file: {path}')
path.unlink()
else:
remove_all(path)
print(f'Deleting the empty dir: {root}')
root.rmdir()
Then, we invoke it for the root directory, inclusive.
>>> from pathlib import Path
>>> root = Path('/home/miguel/Desktop/blog/pathlib/sandbox')
>>> root
PosixPath('/home/miguel/Desktop/blog/pathlib/sandbox')
>>> root.exists()
True
>>> remove_all(root)
Deleting the file: /home/miguel/Desktop/blog/pathlib/sandbox/reports/another_nested/some_file.png
Deleting the empty dir: /home/miguel/Desktop/blog/pathlib/sandbox/reports/another_nested
Deleting the file: /home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt
Deleting the empty dir: /home/miguel/Desktop/blog/pathlib/sandbox/reports
Deleting the file: /home/miguel/Desktop/blog/pathlib/sandbox/article.txt
Deleting the empty dir: /home/miguel/Desktop/blog/pathlib/sandbox
>>> root
PosixPath('/home/miguel/Desktop/blog/pathlib/sandbox')
>>> root.exists()
False
I need to be honest, this solution works fine but it's not the most appropriate one. pathlib
is not suitable for these kind of operations.
As suggested by u/Rawing7
from reddit, a better approach is to use shutil.rmtree
.
>>> from pathlib import Path
>>> import shutil
>>> root = Path('/home/miguel/Desktop/blog/pathlib/sandbox')
>>> root.exists()
True
>>> shutil.rmtree(root)
>>> root.exists()
False
Working with files
In this section, we'll use pathlib
to perform operations on a file, for example, we'll see how we can:
- create new files
- copy existing files
- delete files with
pathlib
- read and write files with
pathlib
Specifically, we'll learn how to:
- create (touch) an empty file
- touch a file with timestamp
- touch a new file and create the parent directories if they don't exist
- get the file name
- get the file extension from a filename
- open a file for reading
- read a text file
- read a JSON file
- read a binary file
- opening all the files in a folder
- write a text file
- write a JSON file
- write bytes data file
- copy an existing file to another directory
- delete a single file
- delete all files in a directory
- rename a file by changing its name, or by adding a new extension
- get the parent directory of a file or script
How to touch (create an empty) a file
pathlib
provides a method to create an empty file named Path.touch()
. This method is very handy when you need to create a placeholder file if it does not exist.
>>> from pathlib import Path
>>> Path('empty.txt').exists()
False
>>> Path('empty.txt').touch()
>>> Path('empty.txt').exists()
True
Touch a file with timestamp
To create a timestamped empty file, we first need to determine the timestamp format.
One way to do that is to use the time
and datetime
. First we define a date format, then we use the datetime
module to create the datetime object. Then, we use the time.mktime
to get back the timestamp.
Once we have the timestamp, we can just use f-strings to build the filename.
>>> import time, datetime
>>> s = '02/03/2021'
>>> d = datetime.datetime.strptime(s, "%d/%m/%Y")
>>> d
datetime.datetime(2021, 3, 2, 0, 0)
>>> d.timetuple()
time.struct_time(tm_year=2021, tm_mon=3, tm_mday=2, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=1, tm_yday=61, tm_isdst=-1)
>>> time.mktime(d.timetuple())
1614643200.0
>>> int(time.mktime(d.timetuple()))
1614643200
>>> from pathlib import Path
>>> Path(f'empty_{int(time.mktime(d.timetuple()))}.txt').exists()
False
>>> Path(f'empty_{int(time.mktime(d.timetuple()))}.txt').touch()
>>> Path(f'empty_{int(time.mktime(d.timetuple()))}.txt').exists()
True
>>> str(Path(f'empty_{int(time.mktime(d.timetuple()))}.txt'))
'empty_1614643200.txt'
How to touch a file and create parent directories
Another common problem when creating empty files is to place them in a directory that doesn't exist yet. The reason is that path.touch()
only works if the directory exists. To illustrate that, let's see an example.
>>> from pathlib import Path
>>> Path('path/not_created_yet/empty.txt')
PosixPath('path/not_created_yet/empty.txt')
>>> Path('path/not_created_yet/empty.txt').exists()
False
>>> Path('path/not_created_yet/empty.txt').touch()
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-24-177d43b041e9> in <module>
----> 1 Path('path/not_created_yet/empty.txt').touch()
~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in touch(self, mode, exist_ok)
1302 if not exist_ok:
1303 flags |= os.O_EXCL
-> 1304 fd = self._raw_open(flags, mode)
1305 os.close(fd)
1306
~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in _raw_open(self, flags, mode)
1114 as os.open() does.
...
-> 1116 return self._accessor.open(self, flags, mode)
1117
1118 # Public API
FileNotFoundError: [Errno 2] No such file or directory: 'path/not_created_yet/empty.txt'
If the target directory does not exist, pathlib
raises FileNotFoundError
. To fix that we need to create the directory first, the simplest way, as described in the "creating directories" section, is to use the Path.mkdir(parents=True, exist_ok=True)
. This method creates an empty directory including all parent directories.
>>> from pathlib import Path
>>> Path('path/not_created_yet/empty.txt').exists()
False
# let's create the empty folder first
>>> folder = Path('path/not_created_yet/')
# it doesn't exist yet
>>> folder.exists()
False
# create it
>>> folder.mkdir(parents=True, exist_ok=True)
>>> folder.exists()
True
# the folder exists, but we still need to create the empty file
>>> Path('path/not_created_yet/empty.txt').exists()
False
# create it as usual using pathlib touch
>>> Path('path/not_created_yet/empty.txt').touch()
# verify it exists
>>> Path('path/not_created_yet/empty.txt').exists()
True
How to get the filename from path
A Path
comes with not only method but also properties. One of them is the Path.name
, which as the name implies, returns the filename of the path. This property ignores the parent directories, and return only the file name including the extension.
>>> from pathlib import Path
>>> picture = Path('/home/miguel/Desktop/profile.png')
>>> picture.name
'profile.png'
How to get the filename without the extension
Sometimes, you might need to retrieve the file name without the extension. A natural way of doing this would be splitting the string on the dot. However, pathlib.Path
comes with another helper property named Path.stem
, which returns the final component of the path, without the extension.
>>> from pathlib import Path
>>> picture = Path('/home/miguel/Desktop/profile.png')
>>> picture.stem
'profile'
How to get the file extension from a filename using pathlib
If the Path.stem
property returns the filename excluding the extension, how can we do the opposite? How to retrieve only the extension?
We can do that using the Path.suffix
property.
>>> from pathlib import Path
>>> picture = Path('/home/miguel/Desktop/profile.png')
>>> picture.suffix
'.png'
Some files, such as .tar.gz
has two parts as extension, and Path.suffix
will return only the last part. To get the whole extension, you need the property Path.suffixes
.
This property returns a list of all suffixes for that path. We can then use it to join the list into a single string.
>>> backup = Path('/home/miguel/Desktop/photos.tar.gz')
>>> backup.suffix
'.gz'
>>> backup.suffixes
['.tar', '.gz']
>>> ''.join(backup.suffixes)
'.tar.gz'
How to open a file for reading with pathlib
Another great feature from pathlib
is the ability to open a file pointed to by the path. The behavior is similar to the built-in open()
function. In fact, it accepts pretty much the same parameters.
>>> from pathlib import Path
>>> p = Path('/home/miguel/Desktop/blog/pathlib/recipe.txt')
# open the file
>>> f = p.open()
# read it
>>> lines = f.readlines()
>>> print(lines)
['1. Boil water. \n', '2. Warm up teapot. ...\n', '3. Put tea into teapot and add hot water.\n', '4. Cover teapot and steep tea for 5 minutes.\n', '5. Strain tea solids and pour hot tea into tea cups.\n']
# then make sure to close the file descriptor
>>> f.close()
# or use a context manager, and read the file in one go
>>> with p.open() as f:
lines = f.readlines()
>>> print(lines)
['1. Boil water. \n', '2. Warm up teapot. ...\n', '3. Put tea into teapot and add hot water.\n', '4. Cover teapot and steep tea for 5 minutes.\n', '5. Strain tea solids and pour hot tea into tea cups.\n']
# you can also read the whole content as string
>>> with p.open() as f:
content = f.read()
>>> print(content)
1. Boil water.
2. Warm up teapot. ...
3. Put tea into teapot and add hot water.
4. Cover teapot and steep tea for 5 minutes.
5. Strain tea solids and pour hot tea into tea cups.
How to read text files with pathlib
In the previous section, we used the Path.open()
method and file.read()
function to read the contents of the text file as a string. Even though it works just fine, you still need to close the file or using the with
keyword to close it automatically.
pathlib
comes with a .read_text()
method that does that for you, which is much more convenient.
>>> from pathlib import Path
# just call '.read_text()', no need to close the file
>>> content = p.read_text()
>>> print(content)
1. Boil water.
2. Warm up teapot. ...
3. Put tea into teapot and add hot water.
4. Cover teapot and steep tea for 5 minutes.
5. Strain tea solids and pour hot tea into tea cups.
The file is opened and then closed. The optional parameters have the same meaning as in open(). pathlib docs
How to read JSON files from path with pathlib
A JSON file a nothing more than a text file structured according to the JSON specification. To read a JSON, we can open the path for reading—as we do for text files—and use json.loads()
function from the the json
module.
>>> import json
>>> from pathlib import Path
>>> response = Path('./jsons/response.json')
>>> with response.open() as f:
resp = json.load(f)
>>> resp
{'name': 'remi', 'age': 28}
How to read binary files with pathlib
At this point, if you know how to read a text file, then you reading binary files will be easy. We can do this two ways:
- with the
Path.open()
method passing the flagsrb
- with the
Path.read_bytes()
method
Let's start with the first method.
>>> from pathlib import Path
>>> picture = Path('/home/miguel/Desktop/profile.png')
# open the file
>>> f = picture.open()
# read it
>>> image_bytes = f.read()
>>> print(image_bytes)
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01R\x00\x00\x01p\x08\x02\x00\x00\x00e\xd3d\x85\x00\x00\x00\x03sBIT\x08\x08\x08\xdb\xe1O\xe0\x00\x00\x00\x10tEXtSoftware\x00Shutterc\x82\xd0\t\x00\x00 \x00IDATx\xda\xd4\xbdkw\x1cY\x92\x1ch\xe6~#2\x13\xe0\xa3\xaa\xbbg
... [OMITTED] ....
0e\xe5\x88\xfc\x7fa\x1a\xc2p\x17\xf0N\xad\x00\x00\x00\x00IEND\xaeB`\x82'
# then make sure to close the file descriptor
>>> f.close()
# or use a context manager, and read the file in one go
>>> with p.open('rb') as f:
image_bytes = f.read()
>>> print(image_bytes)
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01R\x00\x00\x01p\x08\x02\x00\x00\x00e\xd3d\x85\x00\x00\x00\x03sBIT\x08\x08\x08\xdb\xe1O\xe0\x00\x00\x00\x10tEXtSoftware\x00Shutterc\x82\xd0\t\x00\x00 \x00IDATx\xda\xd4\xbdkw\x1cY\x92\x1ch\xe6~#2\x13\xe0\xa3\xaa\xbbg
... [OMITTED] ....
0e\xe5\x88\xfc\x7fa\x1a\xc2p\x17\xf0N\xad\x00\x00\x00\x00IEND\xaeB`\x82'
And just like Path.read_text()
, pathlib
comes with a .read_bytes()
method that can open and close the file for you.
>>> from pathlib import Path
# just call '.read_bytes()', no need to close the file
>>> picture = Path('/home/miguel/Desktop/profile.png')
>>> picture.read_bytes()
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01R\x00\x00\x01p\x08\x02\x00\x00\x00e\xd3d\x85\x00\x00\x00\x03sBIT\x08\x08\x08\xdb\xe1O\xe0\x00\x00\x00\x10tEXtSoftware\x00Shutterc\x82\xd0\t\x00\x00 \x00IDATx\xda\xd4\xbdkw\x1cY\x92\x1ch\xe6~#2\x13\xe0\xa3\xaa\xbbg
... [OMITTED] ....
0e\xe5\x88\xfc\x7fa\x1a\xc2p\x17\xf0N\xad\x00\x00\x00\x00IEND\xaeB`\x82'
How to open all files in a directory in Python
Let's image you need a Python script to search all files in a directory and open them all. Maybe you want to filter by extension, or you want to do it recursively. If you've been following this guide from the beginning, you now know how to use the Path.iterdir()
method.
To open all files in a directory, we can combine Path.iterdir()
with Path.is_file()
.
>>> import pathlib
>>> for i in range(2):
print(i)
# we can use iterdir to traverse all paths in a directory
>>> for path in pathlib.Path("my_images").iterdir():
# if the path is a file, then we open it
if path.is_file():
with path.open(path, "rb") as f:
image_bytes = f.read()
load_image_from_bytes(image_bytes)
If you need to do it recursively, we can use Path.rglob()
instead of Path.iterdir()
.
>>> import pathlib
# we can use rglob to walk nested directories
>>> for path in pathlib.Path("my_images").rglob('*'):
# if the path is a file, then we open it
if path.is_file():
with path.open(path, "rb") as f:
image_bytes = f.read()
load_image_from_bytes(image_bytes)
How to write a text file with pathlib
In previous sections, we saw how to read text files using Path.read_text()
.
To write a text file to disk, pathlib
comes with a Path.write_text()
. The benefits of using this method is that it writes the data and close the file for you, and the optional parameters have the same meaning as in open().
⚠️ WARNING: If you open an existing file,
Path.write_text()
will overwrite it.
>>> import pathlib
>>> file_path = pathlib.Path('/home/miguel/Desktop/blog/recipe.txt')
>>> recipe_txt = '''
1. Boil water.
2. Warm up teapot. ...
3. Put tea into teapot and add hot water.
4. Cover teapot and steep tea for 5 minutes.
5. Strain tea solids and pour hot tea into tea cups.
'''
>>> file_path.exists()
False
>>> file_path.write_text(recipe_txt)
180
>>> content = file_path.read_text()
>>> print(content)
1. Boil water.
2. Warm up teapot. ...
3. Put tea into teapot and add hot water.
4. Cover teapot and steep tea for 5 minutes.
5. Strain tea solids and pour hot tea into tea cups.
How to write JSON files to path with pathlib
Python represents JSON objects as plain dictionaries, to write them to a file as JSON using pathlib
, we need to combine the json.dump
function and Path.open()
, the same way we did to read a JSON from disk.
>>> import json
>>> import pathlib
>>> resp = {'name': 'remi', 'age': 28}
>>> response = pathlib.Path('./response.json')
>>> response.exists()
False
>>> with response.open('w') as f:
json.dump(resp, f)
>>> response.read_text()
'{"name": "remi", "age": 28}'
How to write bytes data to a file
To write bytes to a file, we can use either Path.open()
method passing the flags wb
or Path.write_bytes()
method.
>>> from pathlib import Path
>>> image_path_1 = Path('./profile.png')
>>> image_bytes = b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00 [OMITTED] \x00I
END\xaeB`\x82'
>>> with image_path_1.open('wb') as f:
f.write(image_bytes)
>>> image_path_1.read_bytes()
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00 [OMITTED] \x00IEND\xaeB`\x82'
>>> image_path_2 = Path('./profile_2.png')
>>> image_path_2.exists()
False
>>> image_path_2.write_bytes(image_bytes)
37
>>> image_path_2.read_bytes()
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00 [OMITTED] \x00IEND\xaeB`\x82'
How to copy files with pathlib
pathlib
cannot copy files. However, if we have a file represented by a path that doesn't mean we can't copy it. There are two different ways of doing that:
- using the
shutil
module - using the
Path.read_bytes()
andPath.write_bytes()
methods
For the first alternative, we use the shutil.copyfile(src, dst)
function and pass the source and destination path.
>>> import pathlib, shutil
>>> src = Path('/home/miguel/Desktop/blog/pathlib/sandbox/article.txt')
>>> src.exists()
True
>>> dst = Path('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')
>>> dst.exists()
>>> False
>>> shutil.copyfile(src, dst)
PosixPath('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')
>>> dst.exists()
True
>>> dst.read_text()
'This is \n\nan \n\ninteresting article.\n'
>>> dst.read_text() == src.read_text()
True
⚠️ WARNING:
shutil
prior to Python 3.6 cannot handlePath
instances. You need to convert the path to string first.
The second method involves copying the whole file, then writing it to another destination.
>>> import pathlib, shutil
>>> src = Path('/home/miguel/Desktop/blog/pathlib/sandbox/article.txt')
>>> src.exists()
True
>>> dst = Path('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')
>>> dst.exists()
False
>>> dst.write_bytes(src.read_bytes())
36
>>> dst.exists()
True
>>> dst.read_text()
'This is \n\nan \n\ninteresting article.\n'
>>> dst.read_text() == src.read_text()
True
⚠️ WARNING: This method will overwrite the destination path. If that's a concern, it's advisable either to check if the file exists first, or to open the file in writing mode using the
x
flag. This flag will open the file exclusive creation, thus failing withFileExistsError
if the file already exists.
Another downside of this approach is that it loads the file to memory. If the file is big, prefer shutil.copyfileobj
. It supports buffering and can read the file in chunks, thus avoiding uncontrolled memory consumption.
>>> import pathlib, shutil
>>> src = Path('/home/miguel/Desktop/blog/pathlib/sandbox/article.txt')
>>> dst = Path('/home/miguel/Desktop/blog/pathlib/sandbox/reports/article.txt')
>>> if not dst.exists():
dst.write_bytes(src.read_bytes())
else:
print('File already exists, aborting...')
File already exists, aborting...
>>> with dst.open('xb') as f:
f.write(src.read_bytes())
---------------------------------------------------------------------------
FileExistsError Traceback (most recent call last)
<ipython-input-25-1974c5808b1a> in <module>
----> 1 with dst.open('xb') as f:
2 f.write(src.read_bytes())
3
How to delete a file with pathlib
You can remove a file or symbolic link with the Path.unlink()
method.
>>> from pathlib import Path
>>> Path('path/reports/report.csv').touch()
>>> path = Path('path/reports/report.csv')
>>> path.exists()
True
>>> path.unlink()
>>> path.exists()
False
As of Python 3.8, this method takes one argument named missing_ok
. By default, missing_ok
is set to False
, which means it will raise an FileNotFoundError
error if the file doesn't exist.
>>> path = Path('path/reports/report.csv')
>>> path.exists()
False
>>> path.unlink()
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-6-8eea53121d7f> in <module>
----> 1 path.unlink()
~/.pyenv/versions/3.9.4/lib/python3.9/pathlib.py in unlink(self, missing_ok)
1342 try:
-> 1343 self._accessor.unlink(self)
1344 except FileNotFoundError:
1345 if not missing_ok:
FileNotFoundError: [Errno 2] No such file or directory: 'path/reports/report.csv'
# when missing_ok is True, no error is raised
>>> path.unlink(missing_ok=True)
How to delete all files in a directory with pathlib
To remove all files in a folder, we need to traverse it and check if the path is a file, and if so, call Path.unlink()
on it as we saw in the previous section.
To walk over the contents of a directory, we can use Path.iterdir()
. Let's consider the following directory.
$ tree /home/miguel/path/
/home/miguel/path/
├── jsons
│ └── response.json
├── new_parent_dir
│ └── sub_dir
├── non_empty_dir
│ └── file.txt
├── not_created_yet
│ └── empty.txt
├── number.csv
├── photo_1.png
├── report.md
└── reports
This method only deletes the immediate files under the current directory, so it is not recursive.
>>> import pathlib
>>> path = pathlib.Path('/home/miguel/path')
>>> list(path.iterdir())
Out[5]:
[PosixPath('/home/miguel/path/jsons'),
PosixPath('/home/miguel/path/non_empty_dir'),
PosixPath('/home/miguel/path/not_created_yet'),
PosixPath('/home/miguel/path/reports'),
PosixPath('/home/miguel/path/photo_1.png'),
PosixPath('/home/miguel/path/number.csv'),
PosixPath('/home/miguel/path/new_parent_dir'),
PosixPath('/home/miguel/path/report.md')]
>>> for p in path.iterdir():
if p.is_file():
p.unlink()
>>> list(path.iterdir())
[PosixPath('/home/miguel/path/jsons'),
PosixPath('/home/miguel/path/non_empty_dir'),
PosixPath('/home/miguel/path/not_created_yet'),
PosixPath('/home/miguel/path/reports'),
PosixPath('/home/miguel/path/new_parent_dir')]
How to rename a file using pathlib
pathlib
also comes with a method to rename files called Path.rename(target)
. It takes a target file path and renames the source to the target. As of Python 3.8, Path.rename()
returns the new Path instance.
>>> from pathlib import Path
>>> src_file = Path('recipe.txt')
>>> src_file.open('w').write('An delicious recipe')
19
>>> src_file.read_text()
'An delicious recipe'
>>> target = Path('new_recipe.txt')
>>> src_file.rename(target)
PosixPath('new_recipe.txt')
>>> src_file
PosixPath('recipe.txt')
>>> src_file.exists()
False
>>> target.read_text()
'An delicious recipe'
Renaming only file extension
If all you want is to change the file extension to something else, for example, change from .txt
to .md
, you can use Path.rename(target)
in conjunction with Path.with_suffix(suffix)
method, which does the following:
- appends a new suffix, if the original path doesn’t have one
- removes the suffix, if the supplied suffix is an empty string
Let's see an example where we change our recipe file from plain text .txt
to markdown .md
.
>>> from pathlib import Path
>>> src_file = Path('recipe.txt')
>>> src_file.open('w').write('An delicious recipe')
19
>>> new_src_file = src_file.rename(src_file.with_suffix('.md'))
>>> new_src_file
PosixPath('recipe.md')
>>> src_file.exists()
False
>>> new_src_file.exists()
True
>>> new_src_file.read_text()
'An delicious recipe'
>>> removed_extension_file = new_src_file.rename(src_file.with_suffix(''))
>>> removed_extension_file
PosixPath('recipe')
>>> removed_extension_file.read_text()
'An delicious recipe'
How to get the parent directory of a file with pathlib
Sometimes we want to get the name of the directory a file belongs to. You can get that through a Path
property named parent
. This property represents the logical parent of the path, which means it returns the parent of a file or directory.
>>> from pathlib import Path
>>> path = Path('path/reports/report.csv')
>>> path.exists()
False
>>> parent_dir = path.parent
>>> parent_dir
PosixPath('path/reports')
>>> parent_dir.parent
PosixPath('path')
Conclusion
That was a lot to learn, and I hope you enjoyed it just as I enjoyed writing it.
pathlib
has been part of the standard library since Python 3.4 and it's a great solution when it comes to handling paths.
In this guide, we covered the most important use cases in which pathlib
shines through tons of examples.
I hope this cookbook is useful to you, and see you next time.
Other posts you may like:
See you next time!
This article was originally published at https://miguendes.me