# Python Gotchas: 5 Worst Mistakes You Should Avoid Have you ever had a bug that took ages to fix and made no sense at all? If the answer is yes, then keep reading. Chances are that if you program in Python, you will probably fall into one of these silly behaviors. In this post, I’m going to show you 5 things that have a great potential drive you mad in Python. Some of them are very subtle and others are not obvious at all. By learning about them in advanced, you can save hours of debugging time. Here's what we're gonna cover: - [How implicit string concatenation can be dangerous](#implicit-string-concatenation) - [Be careful when using the walrus operator like this](#walrus-wat) - [Don't use the += operator on lists](#be-careful-when-using-with-lists) - [Why mutable default arguments in functions is the most common bug in Python](#mutable-default-arguments) - [When comparing values, don't be clever!](#chained-operations-gone-wrong) Without further ado, brace yourself and let’s go! ## Implicit String Concatenation ![A dog falling into a pit](https://media.giphy.com/media/L7Nnb9j5NU48g/giphy.gif) I confess that this one has costed me several hours of my life. When used correctly it is very nice but when you don’t, it’s a headache. In Python you can concatenate strings using not only using the `+` operator but also implicitly. The following snippet illustrates a very common bug in Python. ```python In [2]: "french " + "bulldog" Out[2]: 'french bulldog' ``` That's fine, ` + ` generates a new ``. But what you may not know is that you can leave `+` out and Python will still concatenate the string. ```python In [3]: "french " "bulldog" Out[3]: 'french bulldog' ``` When is this a problem, then? It'll be an issue when you want a list of strings and forget a comma. ```python In [4]: dogs = ["poodle" "french bulldog", "pit bull", "american bully"] In [6]: dogs Out[6]: ['poodlefrench bulldog', 'pit bull', 'american bully'] ``` Oh, it can get worse. Imagine you have a function that accepts two strings, but the second one is optional. ```python In [11]: def print_pair(a: str, b: Optional[str] = None): ...: print("a: ", a, "b: ", b) In [13]: print_pair( ...: "First string" ...: "Second string" ...: ) a: First stringSecond string b: None ``` You see? The function runs just fine, so that’s not great! Situations like these can hide nasty bugs. The lesson here is clear: be careful when passing strings to functions or using them as list of items. ## Walrus WAT!? ![A dog playing goes #fail](https://media.giphy.com/media/cv4aHA3NS4YxO/giphy.gif) In 2019, Python 3.8 introduced the walrus operator. This new feature generated a lot of controversies. Some people loved whereas other actually hated it. The goal of this post is not to debate that, so I’ll dive right into what makes walrus confusing. Before Python 3.8, you could not assign a value to a variable and test if it was “truthy” in the same statement. For example, see the following example, of reading data from a socket until an empty string is read. This examples is inspired by [one](https://www.python.org/dev/peps/pep-0572/#capturing-condition-values) described in the PEP. Without `walrus`: ```python data = sock.recv(4096) while data: clean_data = clean(data) print("Received data:", data) data = sock.recv(4096) ``` With `walrus`: ```python while data := sock.recv(4096): clean_data = clean(data) print("Received data:", data) ``` That is, you do both the assignment and the checking in the same line by using `:=`. In Python we can use tuples to assign values to more than one variable in the same line. ```python In [1]: a, b = 2, 3 In [2]: a Out[2]: 2 In [3]: b Out[3]: 3 ``` > Hum... we probably can do the same using walrus, right? ```python In [4]: (a, b := 16, 19) Out[4]: (2, 16, 19) ``` > WAT! Yeah, a 3-tuple is returned! ̶T̶h̶e̶ ̶r̶e̶a̶s̶o̶n̶ ̶f̶o̶r̶ ̶t̶h̶a̶t̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶t̶h̶e̶ ̶`̶b̶`̶ ̶t̶a̶k̶e̶s̶ ̶p̶r̶e̶c̶e̶d̶e̶n̶c̶e̶ ̶a̶n̶d̶ ̶g̶e̶t̶s̶ ̶a̶s̶s̶i̶g̶n̶e̶d̶ ̶t̶o̶ ̶i̶t̶ ̶t̶h̶e̶ ̶t̶u̶p̶l̶e̶ ̶`̶1̶6̶,̶ ̶1̶9̶`̶.̶ ̶I̶n̶ ̶o̶t̶h̶e̶r̶ ̶w̶o̶r̶d̶s̶,̶ ̶i̶t̶’̶s̶ ̶t̶h̶e̶ ̶s̶a̶m̶e̶ ̶a̶s̶ ̶`̶(̶a̶,̶ ̶(̶b̶ ̶:̶=̶ ̶1̶6̶,̶ ̶1̶9̶)̶)̶`̶.̶ ̶A̶n̶d̶ ̶s̶i̶n̶c̶e̶ ̶`̶a̶`̶ ̶h̶a̶d̶ ̶a̶l̶r̶e̶a̶d̶y̶ ̶b̶e̶e̶n̶ ̶b̶o̶u̶n̶d̶ ̶t̶o̶ ̶`̶2̶`̶,̶ ̶s̶o̶ ̶t̶h̶e̶ ̶r̶e̶t̶u̶r̶n̶ ̶i̶s̶ ̶t̶h̶e̶ ̶t̶u̶p̶l̶e̶ ̶`̶(̶a̶,̶ ̶b̶ ̶:̶=̶ ̶1̶6̶,̶ ̶1̶9̶)̶ ̶=̶>̶ ̶(̶2̶,̶ ̶(̶1̶6̶,̶ ̶1̶9̶)̶)̶`̶.̶ Thanks to [@ForceBru](https://www.reddit.com/user/ForceBru/) who kindly [corrected me](https://www.reddit.com/r/Python/comments/jh66bh/how_to_shoot_yourself_in_the_foot_with_python/g9wfoi9/?utm_source=reddit&utm_medium=web2x&context=3), what gets assigned to `b` is not a tuple, but only the first element after `:=`. As a result, `(a, b := 16, 19)` is the same as `(a, (b := 16), 19)`. And that explains why a 3-tuple is returned. You can verify that by [printing the AST](https://tio.run/##K6gsycjPM7YoKPr/PzO3IL@oRCGxuISLq6AoM69EA8jUSynNLQAzChKLilM1lDQSdRSSrGwVDM10FAwtNZU0NTX//wcA) . ```python import ast print(ast.dump(ast.parse("(a, b:= 16, 19)"))) ``` Which produces the following output: ```python Module( body=[ Expr( value=Tuple( elts=[ Name(id="a", ctx=Load()), NamedExpr( target=Name(id="b", ctx=Store()), value=Constant(value=16, kind=None), ), Constant(value=19, kind=None), ], ctx=Load(), ) ) ], type_ignores=[], ) ``` As you can see, the thing is confusing! > What if `a` is not defined? ```python In [14]: (a, b := 16, 19) --------------------------------------------------------------------------- NameError Traceback (most recent call last) in ----> 1 (a, b := 16, 19) NameError: name 'a' is not defined ``` When the first variable is not defined, a `NameError` will be raised. Now you know how to avoid that! ## Be Careful When Using `+=` With Lists ![A dog falling down the stairs](https://media.giphy.com/media/GEObzG7vNeRUs/giphy.gif) Lists in Python are incredibly nice. You can perform all sorts of stuff like: - concatenating multiple lists using `+` operator - generating a repeated list by using the `*` operator - concatenate and assign lists using `+=` Let’s look at an example on how `+` operator works with the `list` object. I know you may be tired of such toy examples, but please, bear with me. ```python In [20]: lst = [3, 4, 5, 6, 7] In [21]: lst_copy = lst In [22]: lst = lst + [8, 9] In [23]: lst Out[23]: [3, 4, 5, 6, 7, 8, 9] In [24]: lst_copy Out[24]: [3, 4, 5, 6, 7] ``` Cool, we created a list called `lst` then we built a new one named `lst_copy` by pointing it to `lst`. Then we changed `lst` by appending `[8, 9]` to it. As expected, the `+` operator expanded `lst` whereas `lst_copy` remained the same. In Python one can shorten expressions like `a = a + 1` as `a += 1`. As I mentioned in the beginning, you can also use the `+=` operator with lists. So, let’s give it a shot and re-write our example. ```python In [25]: lst = [3, 4, 5, 6, 7] In [26]: lst_copy = lst In [27]: lst += [8, 9] In [28]: lst Out[28]: [3, 4, 5, 6, 7, 8, 9] In [29]: lst_copy Out[29]: [3, 4, 5, 6, 7, 8, 9] ``` > WAAATTT!? What happened here? The reason for this behavior is that, like other Python operators, the implementation of `+=` is defined by the class that implements it. That is, to define `+=` the `list` class has defined a `object.__iadd__(self, other)` magic method. And the way it works is the same as `list.extends`. So why has `lst_copy` been modified? Because it is not an actual copy of `lst` but it points to the in memory. ![Screenshot_2020-10-18_08-11-01.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1603005116647/dEuGgiAqj.png) ```python In [28]: lst Out[28]: [3, 4, 5, 6, 7, 8, 9] In [29]: lst_copy Out[29]: [3, 4, 5, 6, 7, 8, 9] In [30]: lst = [3, 4, 5, 6, 7] In [31]: lst_copy = lst In [32]: lst.extend([8, 9]) In [33]: lst Out[33]: [3, 4, 5, 6, 7, 8, 9] In [34]: lst_copy Out[34]: [3, 4, 5, 6, 7, 8, 9] ``` The key takeaway is, don't blindly assume operators will have the same semantics across different classes. ## Mutable Default Arguments ![A dog jumping sofas #fail](https://media.giphy.com/media/13cbZDgXa23UeQ/giphy.gif) I understand that this one might not be new to you. However, it’s unquestionably one of the most dangerous. The case I’m talking about is the usage of mutable default arguments on functions. If you don’t know what this is all about, take a look at the following example. ```python In [35]: def add_fruit(fruit: str, basket: list = []) -> list: ...: basket.append(fruit) ...: return basket ...: In [36]: b = add_fruit("banana") In [37]: b Out[37]: ['banana'] In [38]: c = add_fruit("apple") In [39]: c Out[39]: ['banana', 'apple'] ``` > WAAATT!? As you can see, we call the function twice without passing a list to it. The ultimate result is a list with two items, how did that happen? The reason for this behavior is that when the interpreter defines the function, it also creates the default argument. Then, it binds the object created to the function argument. In our problem, Python allocated an empty list and bound it to the argument `basket`. To make things simpler to follow, let’s look at a visual example made with [python tutor](http://www.pythontutor.com). ![mutable-1.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1603048050068/v39uSktN5.png) ![mutable-2.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1603048076920/D_qYippaU.png) As you can see, the argument `basket` is created once and the function points to it during its entire lifetime. The only exception is when you pass another list to it but that won’t change the default. Whenever you call the function again without passing a list to it, it will use the one created when the function was defined. > How can we avoid this, then? To avoid this, you must set the argument to `None` and create a list if none is passed. ```python In [44]: def add_fruit(fruit: str, basket: Optional[list] = None): ...: if basket is None: ...: basket = [] ...: basket.append(fruit) ...: return basket In [45]: b = add_fruit("banana") In [46]: b Out[46]: ['banana'] In [47]: c = add_fruit("apple") In [48]: c Out[48]: ['apple'] ``` Great! Now we create a list whenever no argument is passed to the function, which fixes the bug. ## Chained Operations Gone Wrong ![A dog playing on grass and failing](https://media.giphy.com/media/3oEduU8I0dHP38CdLW/giphy.gif) Chained operations are an exceptional feature. It makes the code terse without sacrificing readability. I discuss it in more detail in another blog [post](https://miguendes.me/5-hidden-python-features-you-probably-never-heard-of) but to provide you a bit of context let’s see it in action. ```python >>> x = 10 >>> 20 == x == 0 False >>> 25 > x <= 15 True ``` Let's pay close attention to the first example. If Python didn't have this feature, that statement could be re-written as: ```python In [53]: 20 == x and x == 0 Out[53]: False ``` Now, what happens if we add parentheses to enforce some kind of precedence? ```python In [51]: 20 == x == 0 Out[51]: False In [52]: (20 == x) == 0 Out[52]: True ``` > Wait? What on earth has just happened? When we added the parentheses, `(20 == x)` was evaluated to `False`. However, the problem is that `False` is then compared to `0`. ̶S̶i̶n̶c̶e̶ ̶`̶0̶`̶ ̶i̶s̶ ̶c̶o̶n̶s̶i̶d̶e̶r̶e̶d̶ ̶a̶ ̶"̶F̶a̶l̶s̶y̶"̶ ̶v̶a̶l̶u̶e̶,̶ ̶t̶h̶e̶n̶ ̶t̶h̶e̶ ̶c̶o̶m̶p̶a̶r̶i̶s̶o̶n̶ ̶r̶e̶t̶u̶r̶n̶s̶ ̶'̶T̶r̶u̶e̶`̶.̶ As pointed out by @alexmojaki, `False == 0` is `True` because `bool` is a subclass of `int`. For instance, other "Falsy" values such as `[]` or `""` are not equal to `False`. ```python In [54]: False == 0 Out[54]: True In [55]: bool(0) Out[55]: False In [56]: False == "" Out[56]: False In [57]: False == [] Out[57]: False ``` The lesson here is, be careful when using parentheses in chained operations. ## Conclusion That’s it for today, folks! I hope you’ve learned something new and useful. Python has amazing features, but we must use some of them with caution. If we’re not mindful, we may lose tons of time debugging our code. By learning the common pitfall we are much better prepared and not only can prevent these bugs but also avoid them. Other posts you may like: - [The Best Way to Compare Two Dictionaries in Python](https://miguendes.me/the-best-way-to-compare-two-dictionaries-in-python) - [How I Set up Multiple Python Versions with pyenv on Linux](https://miguendes.me/how-i-set-up-my-python-workspace) - [How to Check if an Exception Is Raised - or Not - With pytest](https://miguendes.me/how-to-check-if-an-exception-is-raised-or-not-with-pytest) - [How to Use datetime.timedelta in Python With Examples](https://miguendes.me/how-to-use-datetimetimedelta-in-python-with-examples) - [73 Examples to Help You Master Python's f-strings](https://miguendes.me/73-examples-to-help-you-master-pythons-f-strings) - [11 Useful Resources To Learn Python's Internals From Scratch](https://miguendes.me/useful-resources-to-learn-pythons-internals-from-scratch) - [3 Ways to Unit Test REST APIs in Python](https://miguendes.me/3-ways-to-test-api-client-applications-in-python) See you next time!