Search for a command to run...
Learn how python works internally by going deep inside the python virtual machine.
AI Software Engineer based in London, UK with 6+ years of professional experience developing and releasing software in different programming languages - Hobbyist Technical Writer - Interested in Software Testing, Best Practices, Scalability, Machine Learning/AI, and Python.
Great! These are excellent resources for beginners to learn Python! ✨✌
For a newbie like me, these are excellent resources. It's been 6 months since I started learning Python. As someone from a non-technical background, it was daunting at first, but over time I enjoyed it. For beginners learning Python, Chris Hawkes' YouTube channel is a great resource in addition to the resources you have shared. Thank you so much for this awesome guide.
Best Regards Jason Carl SEO Analyst
Thanks Patrick! I really appreciate it.
Check if two lists are equal, which elements match, get the difference between two lists, compare lists of dictionaries, list of strings and more!
A guide on how to check if two strings are equal, or similar. Learn how to find the difference between two strings. Make complex comparisons and more!
Learn how to fix this new Pylint rule that raises: 'formatting a regular string which could be a f-string (consider-using-f-string)' error
A mega tutorial with dozens of examples on how to use the pathlib module in Python 3
Learn how to ignore autouse fixtures for one test or more tests
"How does Python work internally?"
I have been asking myself that question for the past few months and now it seems that I'm starting to grasp, slowly...
During this time, I have grown a strong interest in learning more about the internal working of python. I find the CPython implementation so fascinating that I even started contributing to the language.
The CPython runtime is the most popular one but there are a few others like pypy. Unlike pypy, the core language is written in C whereas the standard library is a blend of Python and C.
For newcomers, navigating through the code can be a daunting task. Fortunately, there are some nice resources out there that can help to smooth the learning curve.
In this post, I'll show you my favorite resources to start learning more about the inner workings of Python, a.k.a CPython Internals.
By the end of this tutorial, you should be able to:
Python has grown immensely during the past years and the more people learning it, the more the demand for learning materials discussing advanced topics.
A few years ago, I bet just a few curious developers would be interested in learning more about CPython.
These days it’s not unusual to find comprehensive blog posts, videos and books going over the inner guts of Python. Speaking of books, I can surely vouch for two:
This one is the most newly published book about CPython. Amongst all the things it covers, you will find information about:
You can find the full table of contents in the Real Python website, it comes in digital format and paperback. The eBook versions are DRM-Free and available in three different formats: epub, mobi and PDF. The paperback, on the other hand, can be found on amazon .
The nicest thing to me is that Anthony demonstrates how to add a new operator to the language: the almost-equal. This operator is represented by “~=” and can determine if two numbers are close enough to each other but not exactly equal. It walks through all the fundamental changes required to make this happen, including extending the grammar.
Thanks to this book, I started contributing to CPython and have already a handful PRs merged, including enhancements, documentation and bug fixes.
Lastly, the code samples are available on github.
Another good book covering Python's internals, and if I'm not mistaken it's the first one to explore CPython in detail. Inside The Python Virtual Machine is much shorter than CPython Internals but covers some parts of the language in more detail, such as Python objects, Code and Frame objects. It's available for free as PDF, ePub and Kindle (mobi) on leanpub but I definitely encourage you to buy it.
What I loved the most is the in-depth examination of Python objects. It does a brilliant job dissecting the types, goes over the internals of objects and their attributes and concludes with an explanation of the Method Resolution Order (MRO). I haven’t read it in full yet but I like it so far.
When it comes to video contents, there isn’t much structured content out there. The first one I learned about was P. Guo’s series covering Python 2.7. Unfortunately, he’s taken out the series from his website but you can still find it via an unlisted playlist.
Check it out the first video of the series.
The full playlist can be found here .
In this talk, Pablo Galindo, who is a Python’s core developer, looks at the former Python’s grammar and its limitations. This presentation is fantastic for those who want to understand the general structure of the compiler. By the end, Pablo shows how you can add a new operator to Python, the 'arrow operator' ->.
In this presentation, Eric Snow talks about Python’s GIL (Global Interpreter Lock) and the future developments to circumvent its impact on performance and unlock multi-core capability in Python. It won’t dive into the source code but it’s a good intro to one of the most controversial topics in Python.
In this talk, Pablo presents the "magic" behind Python’s memory management by detailing the inner works of the garbage collector and why it matters. He illustrates some gotchas such as the reason you cannot rely on __del__ method and describes in detail the reference counter.
I’ve recently found this resource, and it looks very neat. The author examines the implementation of NoneType and also demonstrates how to include a __len__ method to ints.
What caught my eye was that the author uses gbd to debug the C portion of the code. I find it nice because it’s not so easy for find videos demonstrating how to debug the CPython code.
My favorite blog post series on this topic is from Ten thousand meters by Victor. I believe it’s the most complete and up-to-date coverage into the internals of the Python interpreter in the form blog posts.
Another exceptional series, albeit slightly outdated, is the Python internals series from Eli Bendersky. There are lots of interesting stuff including adding new keywords and a great coverage on symbol tables.
This article / mini book is a great resource to understand Python’s bytecode. In only 500 lines of code Allison implements Byterun, a Python interpreter written in pure Python.
As impressive as it may sound, Byterun can actually run a variety of simple Python programs. After reading it you’ll have a much better understanding of the Python interpreter.
The booklet can be found one https://www.aosabook.org.
The official documentation is also an excellent place to go, the text can be particularly dry but that's what you usually expect from a reference. A nice section in particular is the exploring guide. It goes through the source code’s structure and also links to other resources. The official website also shows the C-API in considerable detail.
Learning about CPython can be disheartening, but thanks to the increasing demand, more and more materials have been developed which softens the learning curve considerably. In this article I presented my preferred resources to learn about the internals of Python ranging from books, to videos and blog posts. I hope this can be useful to you as it is to me.
Other posts you may like:
See you next time!
