Search for a command to run...
AI Software Engineer based in London, UK with 6+ years of professional experience developing and releasing software in different programming languages - Hobbyist Technical Writer - Interested in Software Testing, Best Practices, Scalability, Machine Learning/AI, and Python.
This is well detailed and very well explained.
Thanks for writing and publishing this article Miguel Brito.
Thanks for your kind the words, Edidiong Asikpo!
Check if two lists are equal, which elements match, get the difference between two lists, compare lists of dictionaries, list of strings and more!
A guide on how to check if two strings are equal, or similar. Learn how to find the difference between two strings. Make complex comparisons and more!
Learn how to fix this new Pylint rule that raises: 'formatting a regular string which could be a f-string (consider-using-f-string)' error
A mega tutorial with dozens of examples on how to use the pathlib module in Python 3
Learn how to ignore autouse fixtures for one test or more tests
In this tutorial, we'll learn how to test complex data structures in Python.
Examples of such data include images, nested dictionaries, dictionaries of numpy arrays and even file contents. Learning how to test general data is a very helpful skill, especially if you work in Data Science, Machine Learning or other kinds of scientific programming.
For this tutorial we'll use Python 3.6+, pytest, poetry, pandas, numpy, PIL and pytest-regressions.
To create a new project, we’ll use poetry. If you don’t know what poetry is and how to use it, please refer to the amazing official docs.
The first step is to create a new directory and initialize the project.
mkdir -p complex_data/app
mkdir -p complex_data/tests
cd complex_data
Then, you can install all the dependencies required from the pyproject.toml file.
$ cat <<EOF > pyproject.toml
[tool.poetry]
name = "complex_data"
version = "0.1.0"
description = "Sample project to learn how to test complex data in Python"
authors = ["miguendes"]
license = "MIT"
[tool.poetry.dependencies]
python = "^3.7"
pytest = "^6.0.1"
pytest-regressions = "^2.0.1"
numpy = "^1.19.1"
pandas = "^1.1.2"
pillow = "^7.2.0"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0a5"]
build-backend = "poetry.core.masonry.api"
EOF
Now that we have all setup, let's install the packages. Poetry will take care of the virtualenv creation for you and will install everything there.
poetry install
Now, let’s create a demo app so we can build tests for it. This demo will be very basic and enough to demonstrate how to test complex data. The concepts can be applied to any domain that deals with a similar type of data.
To start with, we’ll create two files. The first one will be the actual demo implementation. The next one will hold all the test case we are going to build. Alternatively, you can also clone the project from github .
touch app/__init__.py
touch app/complex_app.py
touch tests/__init__.py
touch tests/test_complex_app.py
A public REST API is a popular way to get data from third party entities. They usually return the data in a json format. And sometimes the data is organized in a deep nested structure. In Python this data will most likely be represented as a dictionary. To exemplify a function that returns a deep dictionary, consider the following example.
# app/complex_app.py
def config_data(verbose_desc: str, simple_desc: str) -> dict:
return {
"agg_results": {
"t1": [True, False, True, False],
"nums": [1.3, 3.4, 23.456, 21.3456],
"count": 120,
},
"sample": {"desc": {"verbose": verbose_desc, "simple": simple_desc}},
}
This function takes two parameters, verbose_desc and simple_desc, and set it under sample -> desc.
Now, how can we test this function? One way is to test it by inspecting each key->value pair.
# test_complex_app.py
def test_config_data():
actual = config_data(verbose_desc="first experiment", simple_desc="first")
assert actual["agg_results"]["t1"] == [True, False, True, False]
assert actual["agg_results"]["nums"] == [1.3, 3.4, 23.456, 21.3456]
assert actual["agg_results"]["count"] == 120
assert actual["sample"]["desc"]["verbose"] == "first experiment"
assert actual["sample"]["desc"]["simple"] == "first"
If we run the test, we can see that it works, but... You know, it’s too verbose and error-prone. For example, if we add new keys we will need to update the test to add more assert statements. Also, it doesn’t scale at all if the dictionary contains large data. So what is the alternative?
It turns out there’s a very interesting pytest plugin that makes this kind of test very easy and simple. With just a few lines of code, you can test this config data.
# tests/test_complex_app.py
def test_config_data_with_pytest_regressions(data_regression):
actual = config_data(verbose_desc="first experiment", simple_desc="first")
data_regression.check(actual)
That's it! The only step needed is using the data_regression fixture and check the actual data returned. pytest-regressions will create a yaml file with the expected data the first time you run the test. Then it will use this file when you call the test again. As a result, if a regression is introduced, the plugin will detect the diff and the test will fail.
When you run the test for the first time, the test will fail and pytest-regressions will store the output in a directory with the name of the test file.
test_complex_app.py::test_config_data_with_pytest_regressions FAILED [100%]
test_complex_app.py:23 (test_config_data_with_pytest_regressions)
data_regression = <pytest_regressions.data_regression.DataRegressionFixture object at 0x7ff6c4b4f130>
def test_config_data_with_pytest_regressions(data_regression):
actual = config_data(verbose_desc="first experiment", simple_desc="first")
> data_regression.check(actual)
E Failed: File not found in data directory, created:
E - /home/miguel/projects/tutorials/complex_data/tests/test_complex_app/test_config_data_with_pytest_regressions.yml
If you run it again, the test pass.
test_complex_app.py::test_config_data_with_pytest_regressions PASSED [100%]
Image manipulation is a very common task in AI. In order to train Machine Learning algorithms, images must be pre-processed. For instance, some algorithms require the image to be in gray-scale. Sometimes, images must be scaled down, or up. In other occasions you might want to do image augmentation, which usually consists in rotating or skewing the image.
Now, let’s say that we need to resize down an image to 100x100 and convert it to gray as a pre-processing step. We can write a very simple function called convert_image_to_gray. To test this function, we can rely on the great fixture called image_regression provided by pytest-regressions. The following example illustrates that.
# app/complex_app.py
def convert_image_to_gray(
input_path: str, output_path: str, size: Tuple[int, int] = (200, 200)
):
image = Image.open(input_path)
gray_image = image.convert("L")
gray_image.thumbnail(size, Image.ANTIALIAS)
gray_image.save(output_path, "PNG")
The test for it is just like this:
# tests/test_complex_app.py
def test_convert_image_to_gray(image_regression):
output_file = Path("tests/resources/python_logo_gray.png")
convert_image_to_gray(
"tests/resources/python_logo.png", str(output_file), size=(100, 100)
)
image_regression.check(output_file.read_bytes(), diff_threshold=1.0)
And that's it again! No extra setup steps, no complex asserts, just one line.
Another common practice in Data Science is to manipulate numpy arrays. It’s very usual to have dictionaries composed of string keys and numpy arrays as values. Testing that is just as annoying as testing nested dictionaries. It tends to get worse when the arrays are made up of float numbers.
Again, as our previous examples, let's consider a function that returns a dictionary of numpy arrays.
# app/complex_app.py
def element_wise_mult(a: np.array, b: np.array) -> Dict[str, np.array]:
res = a * b
return {"res": res}
This function just multiplies two 1D numpy arrays and stores the result in a dictionary. To test that we can use num_regression fixture. As you may have guessed, the interface is the same as the other fixtures using the check method. A full test definition goes like this:
# tests/test_complex_app.py
def test_elemwise_multi_calculation(num_regression):
np.random.seed(0)
a = np.random.randn(6)
b = np.random.randn(6)
result = element_wise_mult(a, b)
num_regression.check(result)
Python can represent a file object in three categories: raw binary files, buffered binary files and text files.
A binary file can be used to store any kind of binary data such as images, mp3s, videos and so on. A text file, on the other hand, can be a simple .txt or a .html, .md or just your gold old .py script.
The simplest example of binary file would be a function that takes a path string and loads a binary file from disk and returns its bytes. pytest-regression comes with a fixture file_regression that helps a lot when testing file contents.
# app/complex_app.py
def read_from_file(path: str) -> bytes:
with open(path, "rb") as f:
return f.read()
The test goes like this:
# app/complex_app.py
def test_read_from_file(file_regression, datadir):
contents = read_from_file("tests/resources/data.bin")
file_regression.check(contents, binary=True, extension=".bin")
Like I said before, an .html file is just a text file. In this example we'll consider a function that converts an dictionary to a HTML table.
# app/complex_app.py
def dict_to_html(data: dict) -> str:
html = "<table><tr><th>" + "</th><th>".join(data.keys()) + "</th></tr>"
for row in zip(*data.values()):
html += "<tr><td>" + "</td><td>".join(row) + "</td></tr>"
html += "</table>"
return html
And the test...
# tests/test_complex_app.py
def test_dict_to_html(file_regression):
data = {
"Heights": ["30", "12", "12"],
"IDs": ["XXDDUBAB", "LOPSKSKSS", "KIOTGGAGA"],
"Download Count": ["123", "34", "2"],
}
html = dict_to_html(data)
file_regression.check(html, extension=".html")
Well... We could just assert the string returned, right?
You might be asking. We could just assert the string returned. Yes, that’s a fair point, however if the string is too big, the test setup will become polluted and overly verbose. As an alternative, we could store it as a file and then load it. Well, that’s actually what pytest-regressions does, with the benefit of hiding that complexity.
Testing complex data can be tricky and cumbersome. Fortunately, pytest allows great extensibility as plugins. pytest-regression is a non popular but very useful plugin that makes testing complex data easier. I hope you learned something useful and see you next time!