How to Pass Multiple Arguments to a map Function in Python
Solution: You need to transform your function so it can only accept one argument. We can achieve that with
import functools from multiprocessing import Pool from concurrent.futures import ProcessPoolExecutor # Defining a multiple arguments function def sum_four(a, b, c, d): return a + b + c + d # Making it accept only one argument by using partial partial_sum_four = functools.partial(sum_four, a, b, c) partial_sum_four(3) 9 # Using python map function with multiple arguments list(map(partial_sum_four, ds)) [7, 8, 9, 10] # Using pool map for multiple arguments with Pool(processes=4) as pool: res = pool.map(partial_sum_four, ds) res [7, 8, 9, 10] # Using multiprocessing pool multiple arguments with ProcessPoolExecutor(max_workers=4) as pool: res = list(pool.map(partial_sum_four, ds)) res [7, 8, 9, 10]
In a nutshell, that's it folks! If you want to know more details and other methods, please follow along!
In this post, I’m going to show what I do to map a function that expects multiple arguments. The solutions work not only for the regular
map function, you can also use the trick to pass multiple parameters to
Let’s imagine that you have a function called
sum_four that takes 4 arguments and returns their sum.
def sum_four(a, b, c, d): return a + b + c + d
Let’s also suppose that you are solving a very specific problem that requires the first 3 arguments to be fixed. In this problem, you want to compare how the function behaves when you vary only the last parameter.
1, 2, 3 sum_four(a=a, b=b, c=c, d=1) 7 sum_four(a=a, b=b, c=c, d=2) 8 sum_four(a=a, b=b, c=c, d=3) 9 sum_four(a=a, b=b, c=c, d=4) 10a, b, c =
Now, say that you want to use
map, because you like functional programming, or maybe because you come from a language that encourages this paradigm. Since only
d varies, we could store all potential values we want to test in a list
ds = [1, 2, 3, 4]. The issue is, given a function and a list of single elements, if you want to pass that list to a
map function and it takes only one element, what can you do?
The first solution is to not adopt the
map function but use
itertools.starmap instead. This function will take a function as arguments and an iterable of tuples. Then,
starmap will iterate over each tuple
t and call the function by unpacking the arguments, like this
for t in tuples: function(*t).
To make things more clear, consider the following example.
import itertools ds = [1, 2, 3, 4] items = ((a, b, c, d) for d in ds) list(items) [(1, 2, 3, 1), (1, 2, 3, 2), (1, 2, 3, 3), (1, 2, 3, 4)] list(itertools.starmap(sum_four, items)) [7, 8, 9, 10]
As you can see, there’s a lot of repetition, which may inevitably consume a lot of memory if the list is big. To improve that I made
items as a generator, this way we only hold in memory the element we’ll be processing.
The second solution is to use currying and create a new partial function. According to the docs,
partial() will "freeze" some portion of a function’s arguments and/or keywords resulting in a new function with a simplified signature.
In : import functools In : partial_sum_four = functools.partial(sum_four, a, b, c) In : partial_sum_four(3) Out: 9 In : list(map(partial_sum_four, ds)) Out: [7, 8, 9, 10]
The third alternative is to use the
itertools.repeat(). This function produces an iterator that returns object over and over again. It will run indefinitely if you don’t specify the times argument. If we take a closer look at
map()'s signature, it accepts a function and multiple iterables,
map(function, iterable, ...).
According to its description,
If additional iterable arguments are passed, function must take that many arguments and is applied to the items from all iterables in parallel. With multiple iterables, the iterator stops when the shortest iterable is exhausted.
Bingo! We can make
c infitnite iterables by using
itertools.repeat(). As soon as
ds is exhausted, which is the shortest iterable,
map() will stop.
import itertools list(map(sum_four, itertools.repeat(a), itertools.repeat(b), itertools.repeat(c), ds)) [7, 8, 9, 10]
To put it another way, using
repeat() is roughly equivalent to:
1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3], ds)) [7, 8, 9, 10]list(map(sum_four, [
You don't need to worry too much about memory as
repeat produces the elements on the go. In fact, it returns a
list [ref] .
Problem 2: How to Pass Multiple Parameters to
This is very similar to using the regular
map() with multiple parameters. Suppose that we want to speed up our code and run
sum_four in parallel using processes. The good news is, you can use the solutions above, with one exception:
Pool.map only accepts one iterable. This means we cannot use
repeat() here. Let's see the alternatives.
from multiprocessing import Pool import itertools def sum_four(a, b, c, d): return a + b + c + d a, b, c = 1, 2, 3 ds = [1, 2, 3, 4] items = [(a, b, c, d) for d in ds] items [(1, 2, 3, 1), (1, 2, 3, 2), (1, 2, 3, 3), (1, 2, 3, 4)] with Pool(processes=4) as pool: res = pool.starmap(sum_four, items) res [7, 8, 9, 10]
import functools partial_sum_four = functools.partial(sum_four, a, b, c) with Pool(processes=4) as pool: res = pool.map(partial_sum_four, ds) res [7, 8, 9, 10]
Problem 3: How to Pass Multiple Arguments to
concurrent.futures module provides a high-level interface called
Executor to run callables asynchronously.
There are two different implementations available, a
ThreadPoolExecutor and a
ProcessPoolExecutor. Contrary to
Executor does not have a
startmap() function. However, its
map() implementation supports multiple iterables, which allow us to use
repeat(). Another difference is that
Executor.map returns a generator, not a list.
from concurrent.futures import ProcessPoolExecutor import functools def sum_four(a, b, c, d): return a + b + c + d a, b, c = 1, 2, 3 ds = [1, 2, 3, 4] partial_sum_four = functools.partial(sum_four, a, b, c) with ProcessPoolExecutor(max_workers=4) as pool: res = list(pool.map(partial_sum_four, ds)) res [7, 8, 9, 10]
from concurrent.futures import ProcessPoolExecutor from itertools import repeat def sum_four(a, b, c, d): return a + b + c + d a, b, c = 1, 2, 3 ds = [1, 2, 3, 4] with ProcessPoolExecutor(max_workers=4) as pool: res = list(pool.map(sum_four, repeat(a), repeat(b), repeat(c), ds)) res [7, 8, 9, 10]
That’s it for today, folks! I hope you’ve learned something different and useful. The
map() function makes Python feel like a functional programming language.
map() is available not only as a built-in function but also as methods in the
concurrent.futures module. In this article, I showed what I do to map functions that take several arguments.
Other posts you may like:
- How to Use datetime.timedelta in Python With Examples
- 73 Examples to Help You Master Python's f-strings
- How to Check if an Exception Is Raised (or Not) With pytest
- 3 Ways to Test API Client Applications in Python
- Everything You Need to Know About Python's Namedtuples
- The Best Way to Compare Two Dictionaries in Python
- 5 Hidden Python Features You Probably Never Heard Of